I believe it is perfectly possible to maintain a connection with a BER that is worse than the 0dB BER . But I suspect the modem might be supposed to retrain at 0dB, on the assumption that such a connection was undesirable.
Please note careful, politician-style, choice of wording in above.. 'I suspect', 'I believe', etc. Deduce that I'm not 100% sure of my facts... and not volunteering to download ITU specs and spend the rest of the week researching to confirm.
I
think that it is more to do with the number of errors recorded rather than the actual value of the SNRm.
The modem should go into an alarm state when 'x' no of errors per second is recorded. If you look at
LOS you can see the type of errors and alarm states which cause the modem to resync.
So yes you are correct in that it is possible to go into negative SNRm - as some of the old netgears used to. The E/Sec rate would get pretty high though and the connection would get really laggy. This was back in the days before DLM monitored Errored seconds, today's DLM would kick in pretty quick, and bump up things like interleaving - not just the target SNRm