The Mikrotik routers do something similar which is a functional defect. If my router was doing NAT and the route changed, it would retain the old NAT until it closed or timed out. That's not a problem with TCP, or infrequent UDP where it times out anyway. But it was a problem with SIP where the UDP by design has to be frequent enough to keep NAT and firewall entries in place. I did not resolve that, although I had some sort of plan for a script that would clear sessions when a gateway changed state.
In the current design the 4011 doesn't do NAT for either LTE or ADSL, so that specific issue doesn't apply any more. There may be some other hang up that's I've not yet found.
One thing to explore is redirect. I don't at the moment have a L3 transit to the ADSL router, so main router is default gateway and .1 on the subnet, ADSL is .2 on the same subnet. This may redirect, or it may route I have not yet checked or tested.