smalllife // packet & bench

Chasing a silent Path MTU blackhole on a 1492-byte PPPoE uplink

2026-08-19 · routing, ipv6, nftables

After swapping the ISP ONT into pure bridge mode and terminating PPPoE directly on gw01, interactive SSH and small HTTPS fetches worked fine, while a handful of TLS 1.3 handshakes with large certificate chains stalled right after ClientHello. IPv4 recovered after a one-second hiccup; IPv6 connections to two specific CDNs hung until timeout.

Reproducing without a browser

An Ethernet interface at MTU 1500 carrying 8 bytes of PPPoE encapsulation leaves a ppp0 MTU of 1492. Any LAN host with a 1500-byte route expects an ICMP Packet Too Big (IPv6 type 2, or IPv4 type 3 code 4) before stepping down from 1500.

$ ip -br link show dev ppp0
ppp0             UNKNOWN        00:00:00:00:00:00 <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP>

$ ping -6 -M do -s 1444 2606:4700:4700::1111
PING 2606:4700:4700::1111(2606:4700:4700::1111) 1444 data bytes
1452 bytes from 2606:4700:4700::1111: icmp_seq=1 ttl=58 time=14.2 ms

$ ping -6 -M do -s 1445 2606:4700:4700::1111
From 240e:xxxx::1 icmp_seq=1 Packet too big: mtu=1492

Outbound PMTUD from the LAN worked because gw01 generated Packet Too Big locally. Inbound was the problem: an upstream load balancer sending 1500-byte TCP segments toward the PPPoE BRAS dropped ICMPv6 type 2 at its edge firewall, so the server kept retransmitting full-sized ServerHello + certificate flights that the BRAS silently discarded.

Option A: RFC 4638 baby jumbo frames (cleanest when the ONT cooperates)

Raising the underlying Ethernet device to 1508 lets pppd negotiate a full 1500-byte PPP payload via PPP-Max-Payload:

# /etc/network/interfaces.d/wan or systemd.link
ip link set dev enp1s0 mtu 1508
# in /etc/ppp/peers/isp:
mtu 1500
mru 1500

My GPON ONT forwarded 1508-byte frames, but the upstream access concentrator capped MRU negotiation at 1492 anyway, rejecting PPP-Max-Payload: 1500 in LCP Configure-Ack.

Option B: Bidirectional MSS clamping to route MTU

When true 1500-byte PPPoE is off the table, clamping TCP MSS on both ingress and egress SYN packets keeps negotiated segment sizes at 1452 (IPv4) and 1432 (IPv6) without hardcoding magic numbers:

table inet filter {
  chain forward {
    type filter hook forward priority filter; policy drop;

    tcp flags syn / syn,rst tcp option maxseg size set rt mtu
    # ... established/related and LAN-to-WAN rules follow
  }
}

Note the subtle caveat with rt mtu: it clamps against the egress route MTU of each packet. That fixes SYN packets leaving ppp0 (lowering the MSS advertised to the remote server, which fixes the inbound blackhole), and if the remote server sends SYN+ACK back into the 1500-byte LAN, rt mtu evaluates to 1500 unless you also clamp ingress from ppp0 explicitly when needed:

iifname "ppp0" ip6 nexthdr tcp tcp flags syn / syn,rst tcp option maxseg size set 1432
iifname "ppp0" ip protocol tcp tcp flags syn / syn,rst tcp option maxseg size set 1452

Verified with tcpdump -ni enp2s0 'tcp[tcpflags] & tcp-syn != 0': both SYN and SYN+ACK on the LAN side now advertise mss 1432 over IPv6, and the stalled TLS handshakes disappeared immediately.