Here is an example with DNS where the 1st request is done with dnsmasq that set the DF bit and the second request is done via dig that does not set the DF bit. Both requests are sent to Quad9 DNS to resolve the `lesterpig.com` domain:
```
# tcpdump -vv -i wwan0 'port 53 and (host 9.9.9.10 or host 149.112.112.10)'
tcpdump: listening on wwan0, link-type RAW (Raw IP), capture size 262144 bytes
17:30:39.832839 IP (tos 0x0, ttl 64, id 40453, offset 0, flags [DF], proto UDP (17), length 70)
While we successfully validated the OpenVPN solution, we do not want to add this complexity in our daily setup.
Moreover, as only the egress traffic (TX) is impacted and not the ingress (RX), we can rewrite the packets locally.
By being fast and secure, we selected XDP to rewrite the packets.
In the journey, we mainly learnt/discovered 3 main things:
- XDP is INGRESS only, an EGRESS patch has been proposed but is not yet merged. It means we can not rewrite packets that are sent over an interface, only packets that are received on the interface. Hopefully, when my computer sends an UDP packet (to a DNS server for example), it is sent to its ethernet interface, received on the `br-lan` interface of the router then forwarded on its `wwan` interface. So, the only place we can do the rewrite is on `br-lan`. It has two drawbacks: local packets are rewritten while its useless and the router can not benefit from the XDP module for its own traffic. Still, this situation seemed acceptable to us.
- At first, we thought it would be simple as we only want to flip a bit! But IPv4 has checksums (while IPv6 not), so we need to recompute it. bpf-helpers has some tools for that but they often requires a `sk_buff`, the internal structure of the kernel. But we are such low level here that this structure has not been created yet. It stll might be possible to use some functions but without any example and due to the quite strange interface and documentation, none of our tests we successful. We finally reimplemented in BPF the original checksuming from [RFC 1071](https://tools.ietf.org/html/rfc1071), and it works! We know that as we are only modifying part of the packet, we could do a more efficient incremental update as decribed in [RFC 1624](https://tools.ietf.org/html/rfc1624). We did not put the effort to understand and implement this optimization as the simple solution is easier to understand/debug/implement for us while providing satisfying performances.
- Finally, it appears that OpenWRT (in its upstream version) supports XDP out of the box, but you need a loader. Because we do not need BPF maps, we can use the loader embedded in iproute2. However, by default on OpenWRT, iproute2 is not compiled with the support for XDP/eBPF. To embed the loader in iproute2, you need to install/select the `ip-full` from OpenWRT (in the `Network/Routing and Redirection` section). You can find [these details in the Makefile](https://github.com/openwrt/openwrt/blob/master/package/network/utils/iproute2/Makefile) of the package.
Linked resources:
- [The code + instructions to compile and load](../xdp)
