Interesting questions, Weaver. I don't know of any way of reading a modem serial number, but if the modem is on a local IPv4 network you could use ARP to recover its MAC address, and use that as an identifier. At least in principle. The details might be tricky.
As you no doubt know, the Hlog spectrum (among other things) changes when retraining. Otherwise, quoting the G992.3 standard, "The [Hlog] measurement shall not be updated during showtime". So the change is an unambiguous flag that something has happened.
As for recognising anomalies, others know much more than I do. In my limited experience, SNR is the most reliable indicator that performance is becoming marginal. SNR from time to time drops towards a level that would cause the modem to retrain, but then recovers within a few seconds. In those cases, it might not be productive to sound an alarm.
For counters, I use "Total" counts, rather than the latest 15 minutes or latest hour, and log the difference between the current total and the previous total. If the total ever falls (to zero), I reset things. During training, by the way, counts are not reported (on my particular modem).
You asked about actual numbers. I am looking now at a 15 hour log with:
downstream SNR 6.3 dB,
downstream date rate 15.3 Mbps
FEC 120 (corrected errors) per second
13 downstream CRC errors (i.e. one per hour)
A few days earlier, before it retrained to an SNR of 6 dB, a 24 hour record of the same modem showed it running consistently at:
downstream SNR 3.1 dB
downstream data rate 17.6 Mbps
FEC 3000 per second
1256 downstream CRC errors (i.e. just under one a minute)
Although the error numbers differ greatly, I regard both as acceptable. It shows the tradeoff between error rate and data rate.
On Regexes, can I suggest that you write a benchmark first, as some expressions can take a lot of CPU to evaluate. I found that out the hard way.
I look forward to you finding a useful warning signal. Until then, I will fall back on my personal alarm of complaints that the internet's not working.