It's a good suggestion but I'm pretty sure that's not it, I've already got a decent WPA2 key and where I live is very rural which makes it unlikely.
The results from last night are narrowing things down, I got lucky with the 3 clients I disabled, 2 Chromecasts and an amazon firestick, you can see overnight the latency was fine, the high latency in the evening was caused by watching BBC iplayer, I'll investigate further tonight.
I thought it would be a good idea to monitor hourly data usage as well to see what correlation there was with latency and if any of my devices are using silly amounts of data and I'm going to use dslstats to do this. There are two interfaces I can monitor and I'm not sure which I should be looking at, ptm0 shows RX : 3.1 GB / TX : 1.3 GB and ptm0.1 shows RX : 3.4 GB / TX 1.1 GB, I'm using a Zyxel VMG1312-B10A?