Thanks John, perhaps the yellow low level latency is just a VM thing.
Back on pfSense now.
I've always run pfSense with the whole world pretty much blocked, everywhere except the UK is blocked, unless I'm out of the country then I allow the country I'm visiting as well.
Now pfBlockerNG states:
"It's also not recommended to block the 'world', instead consider rules to 'Permit' traffic from selected Countries only. Also consider protecting just the specific open WAN ports and it's just as important to protect the outbound LAN traffic."
The above makes total sense, but I could never work out how to just permit the UK rather than block everything else, so have always blocked the world and temporarily allowed any countries I'm visiting, which is not often these days.
I've been using the above approach ever since I've been using pfSense, so I doubt its that causing the issue, and the system certainly doesn't seem stressed with CPU usage around 2-5%