A quick way to tell (well not super quick as you have to wait for the times of the day), is to test at off peak and peak, if performance is consistently good off peak but drops at peak then its an isp issue, if its consistent then the problem is likely the client side.