Read on Twitter
A week ago, Ron and other people complained about horrible download speeds from my home server over IPv6. This runs on a 6in4 tunnel with @henet over a fast FTTH IPv4-only connection, but in fact IPv6 uploads speeds have been bad since a while ago <thread>... https://twitter.com/drmpeg/status/1346970080609857537
After some careful debugging, I've been able to locate and fix the problem: generic receive offload (GRO) gives problems with the 6in4 tunnel at my router, which is the tunnel endpoint. Let's see this in more detail with Wireshark captures, which show why debugging can be tricky.
To test this, I'm downloading from an IPv6 VPS a file over HTTPS hosted at my home server, just as Ron did. This is the traffic between the server and router. Packets at the router are tagged with VLAN 3. Note the packet marked in black and white. It's 2946 bytes because of GRO.
GRO means that somewhere in the kernel adjacent packets in the same TCP are merged so that the network stack has to process less frames. tcpdump already sees the merged packets. These packets need to be split back before being sent over the 6in4 tunnel.
And in fact that's what it appears to happen. This is the traffic over the FTTH, which uses a PPPoE link over VLAN 6. The packet has been split in the 3 packets shown in black and white below...
...or that's what it would seem if we run tcpdump on the router. If we tap the FTTH line between the router and the ONT we see the following. No trace of the black & white packets. In fact, the first packet that appears has seq=5273. This is marked in blue in all the captures.
Go check yourself the captures and see how all packets affected by GRO are not split back but rather lost in limbo. This causes severe packet loss, which impacts TCP speeds. I stress that this is only seen by tapping outside of the router. tcpdump in the router lies to us.
The fix is simple. Just run:
ethtool -K eth0 gro off
at the router to disable GRO. After doing this, upload speeds are good. </thread>
ethtool -K eth0 gro off
at the router to disable GRO. After doing this, upload speeds are good. </thread>
