Kitz Forum

Broadband Related => Broadband Technology => Topic started by: Weaver on June 30, 2020, 02:51:08 PM

Title: Change of MTU on the fly with TCP (recap)
Post by: Weaver on June 30, 2020, 02:51:08 PM
[I’m sure we have discussed this before; I’d like a recap relating to TCP especially though]

Q: In either IPv4 or IPv6, if you experience a change of path MTU on the fly, do protocols and applications normally just pick up and adjust ok ? Say you’re using TCP, what happens then - does TCP adjust an existing connection ok ?



(Background: discussed some while ago -
When I’m using my 3G USB emergency failover NIC (‘dongle’) that is connected to my router and which kicks in automatically in seconds if all the DSL lines go down, then there is unfortunately a reduced MTU/MRU of 1440 bytes max due to limitations of the stupid NIC for some reason. Also the 3G service doesn’t support IPv6 so I have to use tunnelling in IPv4, which is automatically handled by my router and ISP with proto 41 (what is it called? - 6in4 ? I keep getting 6in4/6over4 et al mixed up - ‘over’ is the more complex system which is not just using static configuration - is that right?) That means that there’s an additional 20 byte header imposed on IPv6, so a max IPv6 MTU of no more than 1420. I chose however to use IPv6 MTU of 1408 as that gives a magic number for for full ATM cell payloads to maximise protocol efficiency, assuming 32 bytes of one-off headers in my case for a total which is 32 + 1408 = 1440 = 30 * 48, so an integer multiple of 48 bytes for full ATM cell payloads. I therefore have chosen to use IPv6 MTU 1408 all the time, not just during 3G failover, but IPv4 MRU/MTU is 1500 bytes at all times. So I’m relying on hope and the power of mindless optimism, to take care of IPv4. If you’re using TCP with IPv4, is it possible that TCP might drop the link and reestablish it because of ICMP error messages regarding ‘too big’ errors or whatever it’s called in IPv4?

This means that IPv6 failover is going to work seamlessly guaranteed, but as for IPv4 - I don’t know what might happen to IPv4 flows using TCP or not when failover happens.

I don’t somehow want to permanently reduce the size of TCP packets for IPv4, but am I doing the right thing here by leaving it as is? Failover is such a rare event that I don’t want to optimise for a very rare case and disadvantage the common case. Using 1408 is in one sense better than MTU 1500 because the latter is not a full cells 48 bytes multiple (in fact 1500 + 32 bytes overhead = ADSL2 total in my case), however there is also a downside because smaller packets mean more IP and TCP headers sent to achieve a total transfer of some n bytes of payload. I should work out the effect of the latter really, which works in the opposite direction.)



Im wondering if an IPv4 flow might get permanently fragmented for the rest of the duration of the connection. If using TCP, then I suppose a new TCP connection would work out a sane lower TCP MSS ? But not sure what would happen in practice until then, especially given that some systems really hate delivering IPv4 fragments anyway.
Title: Re: Change of MTU on the fly
Post by: j0hn on June 30, 2020, 03:00:21 PM
I've read that post twice now and I'm still not sure what the question is.
I'm dizzy though.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on June 30, 2020, 03:07:10 PM
Question is : What does TCP do if an ICMP too big message (or the IPv4 equivalent) is received ?

Apologies for the verbosity of posts - feeling as sick as a dog and full of anti nausea drugs, plus writing prob helps take my mind off it.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Alex Atkin UK on June 30, 2020, 08:53:39 PM
I hope you react better to anti nausea drugs than I did, made me feel like I was dying and I had horrific vivid dreams.  The nausea was almost better than the drug side effects.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on June 30, 2020, 11:25:52 PM
I’m on ondansetron, cinnarizine and prochlorperazine and nothing has worked today. Very horrible day indeed.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on July 01, 2020, 04:13:52 PM
Feeling a million times better today apart from fever which is still very bad.

Coming back to the point, does TCP ever use the IPv4 DF flag ? Was that what I was thinking about, getting very confused? If not then I suppose an IPv4 TCP connection in progress and already established will just get fragmented completely, no?



I read https://en.wikipedia.org/wiki/IP_fragmentation which was helpful as a recap and very good on the differences between IPv4 and IPv6, but I didn’t see any discussion of dynamic change of L3 MTU on the fly.

I suppose I could do an experiment with IPv4 TCP: start capturing traffic, do a transfer and then trigger failover to 3G in order to see what happens. Would require some manual dexterity to get the timing right. My ISP, AA, can do a traffic capture at their end which provides me with an easy way of getting a snapshot.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Alex Atkin UK on July 02, 2020, 10:37:42 PM
All I know is when I load-balanced between DSL at 1500 and 4G at whatever it was (1440 maybe), I would randomly get hangs loading web pages.

I switched to a better 4G router and discovered my local tower seems to handle 1500 just fine and load balancing now is flawless,  apart from the huge speed and latency fluctuations 4G can have.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on July 03, 2020, 10:29:17 AM
Thanks for letting me know, Alex.

The thing is, you would have established your TCP connections based on the 1500 or 1492 byte MTU at your LAN i/f or router‘s combined WAN i/f, so that would have been too high for 4G then? Anyway, it’s a permanent mismatch nit a change in the fly. Would have been interesting to see if reducing your LAN i/f MTU to 1400 say would have fixed the problem.

I’m interested in the case where you start out ok but then the path MTU changes while a TCP connection say is already set up and still in progress.



I did a traffic capture of an upload but it is massive and I drowned. I need some way of making the mass if information more readable and digesting it down into some overall relevant dynamic info and overall stats. But I don’t know how to define that.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: niemand on July 14, 2020, 11:57:36 AM
Question is : What does TCP do if an ICMP too big message (or the IPv4 equivalent) is received ?

Packets start getting fragmented and reassembled automagically. Only get that ICMP message if the DF bit is set in the IP header. TCP cannot modify MSS in the middle of a session, during 3-way handshake only then it's up to the IP layer to fragment and reassemble as it needs to.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on July 14, 2020, 08:18:01 PM
I’m suddenly realising. Do TCP implementations ever/sometimes/always set the DF bit in data packets?
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on November 09, 2021, 10:30:35 PM
Niemand woke me up; I was being stupid, forgetting that the DF flag has to be set in order to get the ICMP message returned up the reverse path. So it probably won’t be, and IPv4 TCP will never have a problem, because it won’t know that things are getting fragmented, and IPv4 will simply fragment and reassemble the IP packets with no problem, unless there’s some evil firewall problem or something. So in the case of IPv4 I would get away with it. In the case of IPv6 I don’t think things would work though if I change MTU on the fly, not unless IPv6 TCP reacts to ICMP too big messages. But I dont change IPv6 MTU on the fly; it’s pre-reduced to an adequate low level, so there should be no problem at all.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Chrysalis on November 10, 2021, 08:31:02 AM
MSS is adjusted not MTU with detection.

The problem is far too many people block ICMP, which has led to alternative ways been developed.  In linux there is MSS detection via TCP instead, but like many new network things, it is of course not fully enabled by default.

Quote
tcp_mtu_probing - INTEGER
   Controls TCP Packetization-Layer Path MTU Discovery.  Takes three
   values:
     0 - Disabled
     1 - Disabled by default, enabled when an ICMP black hole detected
     2 - Always enabled, use initial MSS of tcp_base_mss.
Title: Re: Change of MTU on the fly with TCP (recap)
Post by: Weaver on November 10, 2021, 11:20:16 PM
Sorry, I should have said ‘max MTU’ of links, not of o/s’s protocol stacks.

The Linux technology you mentioned, I need to read up some more about this. Another good thing that we all should be enjoying, as changes of PMTU on the fly can be quite a thing in failover modes, and when users switch to their 4G NIC, or switch their s/w VPN on in a portable device, or roam to alien WLAN locations and need to use tunnelled IPv6, want to use encrypted tunnels, or whatever.