You're right, it "should not be affected". However, some times PN do not light up new segments fast enough to cope with demand; some times their switches go out of balance. I recall that particular gateways almost always dropped more packets than others.
It is true that the http/s packets should not be affected, however there have been times when they were badly throttled on more than one gateways. There was even a thread a few months ago (I can't find it now but I think it was opened by BarryZ) where he asked for people to provide relevant info like gateway, speedtest results, router stats, etc. in the hope that PN identified some pattern in the problem.
Notwithstanding the above, if as you say the gateways look quiet and other users are not reporting problems, then the slowdown could be due to contention at the local exchange: GOM, what's your neighbours' ADSL performance like?