It sounds as though the faceplate plug isn't making good contact with the test socket when the faceplate is fixed in place, but it could just be something making bad contact inside the faceplate. Either way, replacing it would be the first thing I would do. It was fitted by BT Openreach, and you can't officially contact them directly, so you need to badger Sky to insist on a replacement.