@jelv - indeed, that’s the kind of thing that I’m already doing now with my failover to 3G; I’m already limiting IPv6 IP MTU to 1408 bytes all the time in order to deal with the limited MTU of the 3G USB dongle NIC that I have for failover. The IP MTU of my DSL lines is 1500, PPP MTU 1508. I’ve left the IPv4 MTU at 1500 bytes so there will probably be some fragmentation during failover and that may or may not be a really bad thing. Depends on how well fragments are handled.
I don’t know how many systems misbehave when using IPv4 if the far end of some path they’re accessing can’t get an IPv4 MTU of 1500. I hope there won’t be too many, especially given that so many links are restricted to an IP MTU of 1492.
I always made it a priority to get an IPv4 MTU of 1500, but I don’t know how important that is in practice, if at all. Being over-cautious, maybe.
I just wish that all links and NICs had an actual IP MTU and L2 MTU of something more like 1600 bytes, then all machines could use an IP MTU of 1500 bytes with no problem, even if there’s tunnel overhead, say also PPPoE overhead, and possible other underlying L2 overhead. BT seems to do the right thing, with a large L2 MTU; they can of course handle things such as PPPoE 1508 byte PDUs. I presume the answer is in a BT SIN somewhere.
I had a weird issue on my line that uses baby jumbo frames to hit a 1500 byte MTU.
I noticed on two endpoints I was getting stalling issues. One of them was FreeBSD distribution servers, a simple http fetch would fail at a very high rate by just sitting there for ages with a very high timeout. Occasionally (sub 5%) it would establish the connection and download.
Another was a server I manage myself, this one because I controlled both endpoints I fiddled with lots of thinks like the new TCP based MSS negotiation, and orphan timeout. On this one the issue was downloads would always start normally, proceed at normal speed, but when the file was almost complete it would stall until the ftp client times out and then resume with no problems and carry on, I found raising the orphan timeout made it recover by itself after a few seconds. So would then stall, but then carry on after 5-10 seconds and finish with no timeout in the client.
I eventually discovered dropping my MTU to 1492 PPPoE standard fixed both endpoints, yet on most endpoints I could see 1500 byte MTU was working fine, and also if I downloaded from FreeBSD on a native 1500 byte MTU connection was also fine so a very strange issue.
I ended up for now adding my own custom patch to pfsense so when it generates the rules, it sets a lower outgoing MSS using a scrub mss clamp rule for end points I define in a variable, of which I added all the FreeBSD dist servers and my problematic server. I of course would rather just ditch PPP and have native 1500 bytes. I did the patch because when I dropped it to 1492 globally I was noticing longer waits on web browsing, confirmed via dev tools, I assume maybe due to MSS negotiation.