Kitz Forum

Broadband Related => ADSL Issues => Topic started by: siofjofj on May 02, 2018, 06:30:26 PM

Title: Failing MSAN Ports
Post by: siofjofj on May 02, 2018, 06:30:26 PM
I have a TalkTalk ADSL service and in November 2017 I had a fault where I would lose sync both randomly and when the phone was picked up or rung. There was also data noise audible on the phone when the router was connected. I assumed this was a high resistance fault on the line, and indeed the remote fast tests did sometimes (but not always) state the presence of a CIDT fault. However after 1 PSTN and 2 SFI Openreach engineer visits the fault was found to be with the port on TalkTalk’s MSAN in the exchange. I was changed to a different port and all was well.

However in April 2018, after my router had been in sync for 160 days, the same symptoms reoccurred. A fault was reported and I gave the attending PSTN engineer the name of the SFI engineer who ultimately found the fault last time. The SFI engineer told the PSTN engineer what to test for, and it was once again found to be the MSAN port. Once again this was changed and all is now well.

The engineer said it is very rare for a port to go bad, so two going bad suggests something else is causing it, perhaps some other undetectable fault on the line. The engineer left and returned to his van, and I reconnected my internal wiring and router (I had been using the TalkTalk provided HG633 router and a basic corded phone plugged into the test socket via a dangly filter for diagnostic purposes, but my normal set-up has a data extension from a MK3 SSFP and a TP-Link VR600 router).

From the van the engineer initiated a line test (presumably this is standard procedure when closing a job) and got back a CIDT fault. He tried this 5 times and got the same result, so returned to the house and asked me to remove everything again. On initiating a final line test, no fault was found so he closed the job, but commented that perhaps the house wiring and/or the VR600 is bad and is causing the MSAN ports to go bad.

The line seems perfectly stable now, and I did try swapping to a different pair on the data extension which made no difference to anything. But this has got me wondering, can bad wiring / the VR600 in the house cause MSAN ports to go bad over time (excluding things like shoving 240V down the phone line which would presumably blow up the MSAN port in an instant)?
Title: Re: Failing MSAN Ports
Post by: burakkucat on May 03, 2018, 12:10:06 AM
That is an interesting situation.

As you have checked your data extension wiring and found no obvious fault, then there are just two other items to be ruled out. The SSFP which, as far as the circuit is concerned, is a passive device and the TP-Link VR600.

All modems will connect via a high-pass filter which, at the very basic level, can be regarded as a capacitor in each leg of the pair. I am at a loss as to what sort of fault a modem might possess that results in the associated MSAN port to be zapped.

If I was in your situation, I would test by having the VR600 powered up but disconnected from the circuit and then checking for any spurious voltages across the pair, also between each leg of the pair & a good earth. (Prerequisites: a DMM and a good earthing point.)
Title: Re: Failing MSAN Ports
Post by: Weaver on May 03, 2018, 02:34:00 AM
Has there been any thundery weather, dark looming clouds?

How long is the phone line?
Title: Re: Failing MSAN Ports
Post by: siofjofj on May 03, 2018, 07:47:01 AM
Before the fault occurred in November I was actually using a MK2 SSFP. I changed this to the MK3 just after the fault was fixed, so I think this rules out it being this. Good thought regarding the check for spurious voltages appearing from the DSL input to the VR600. I'll check tonight.

The phone line is about 1.5km long and has a FTTC cabinet just outside the exchange and an AIO FTTC cabinet around 900m from the exchange. The BT wholesale ADSL checker predicts an ADSL sync speed of 14mbps. In reality I'm syncing at 17.7mbps, and DSLstats now shows an arrow straight and flat SNRM, so I'm inclined to think the whole metallic path from the exchange to the router is in perfect condition.

There weren't any notable external conditions in November when the port went bad, but in April the day the problems started we had around 3 short duration (say 2 seconds) power cuts.
Title: Re: Failing MSAN Ports
Post by: siofjofj on May 03, 2018, 06:16:12 PM
I can now confirm there are no significant* voltages, AC or DC, present across the pair or from each side of the pair to earth.

* 1V RMS @ 50Hz was detected between each side of the pair to earth, but my DVM detects this if one probe is grounded and the other is left disconnected, so this is almost certainly not real.
Title: Re: Failing MSAN Ports
Post by: burakkucat on May 03, 2018, 07:05:19 PM
I can now confirm there are no significant* voltages, AC or DC, present across the pair or from each side of the pair to earth.

So that is now ruled out.

Quote
* 1V RMS @ 50Hz was detected between each side of the pair to earth, but my DVM detects this if one probe is grounded and the other is left disconnected, so this is almost certainly not real.

Yes, I agree. That reading is just a phantom artefact.
Title: Re: Failing MSAN Ports
Post by: siofjofj on May 04, 2018, 08:09:39 AM
Well, I guess I don't need to be concerned that I'm causing this then. I'm probably just unlucky / something outside my control is causing it.

Thanks for the advice!
Title: Re: Failing MSAN Ports
Post by: burakkucat on May 04, 2018, 05:38:00 PM
Thanks for the advice!

You are welcome.
Title: Re: Failing MSAN Ports
Post by: siofjofj on August 31, 2018, 06:42:44 PM
So 4 months down the line and I'm back with the same issue. Given the identical symptoms I'm guessing that I now need to be moved to yet another MSAN port (the 4th)! I now find it very hard to believe that I was just unlucky with getting two bad ports in a row, so the theory has to be that something is causing this.

So before I get Openreach on the case (via TT), would the knowledgeable people on this forum mind taking a look at my connection statistics and let me know if you have any theories.

SNRM:
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi1380.photobucket.com%2Falbums%2Fah161%2Faieucy%2Fdslstats%252031-08-2018%2FSNRMargin-2018-08-31-18.03.43_zpsm8pvzvrl.png&hash=0189403a10b9459babc1b6316c8486b7c4e2f39e)

CRC:
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi1380.photobucket.com%2Falbums%2Fah161%2Faieucy%2Fdslstats%252031-08-2018%2FCRC-2018-08-31-18.03.43_zps1bvdbugd.png&hash=5706eb3ba5d914823553de08dee0241feef18073)

To avoid creating a huge post, the rest of the DSLStats snapshots can be seen here:

[Link removed at author's request]

Cheers


Title: Re: Failing MSAN Ports
Post by: burakkucat on August 31, 2018, 09:02:47 PM
Very mysterious.  ???

I would be tempted to check, once again, for the presence of alien voltages (both DC & AC) whilst all your equipment is connected as normal. B-wire to Earth, A-wire to Earth and B-wire to A-wire. Then disconnect everything by removing the lower front face-plate and the SSFP and repeat the measurements on the incoming pair. Finally, repeat the measurements at the plug where your own wiring connects to the NTE5.

Looking at the various plots, I am not sure what to make of the situation. They clearly show a sickly circuit.
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 01, 2018, 10:20:48 AM
I now have the results of these measurements, and they are quite interesting.

Everything connected / V
  A-B    A-E    B-E 
DC    -49  -50  -1
AC    *  *  1

Disconnected - Internal / V
  A-B    A-E    B-E 
DC    0  0  0
AC    0  10  10

Disconnected - External / V
  A-B    A-E    B-E 
DC    -49  -50  -1
AC    *  *  1

*: Gives a wildly fluctuating reading in the range 20-200V on my Amprobe AM520-EUR DVM at a frequency the DVM cannot determine. Repeating this with a generic cheapo M-830B DVM gives a steady 110V AC, the mean of the range given by the Amprobe. This value is achieved on both the 500V and 200V ranges of the M-830B.

Likely the 10V to earth on the internal wiring is just inductive pickup. The measurements indicated by * are odd though. Unfortunately I do not possess a scope, a true RMS DVM or a DVM with a low impedance mode; all of which may well reveal some more information about this.

I'd have thought any stray voltages on the line would be detected during a pair quality test on the JDSU as part of the test for a battery contact fault. This part of the PQ always passes, but I wonder if the JDSU checks for AC. It may only be DC which would explain why this is not being picked up. Perhaps Black Sheep might know. The alternative is that these measurements are just an artefact of what the TT MSAN puts on the line as measured by a DVM. If someone with a TT ADSL service is able to repeat these measurements this may confirm/disprove this.
Title: Re: Failing MSAN Ports
Post by: burakkucat on September 01, 2018, 04:39:53 PM
More than interesting, I'd say very interesting . . . and surprising.  :o

A quick comment. What you have "labelled" A and B are actually swapped over. There is always a 50% chance that the pair has been inverted as the polarity is no longer considered to be relevant. With the DVM +ve probe connect to a good earth, the B-wire is the one which shows the greatest voltage. So your first table is actually --

Everything connected / V
  B-A    B-E    A-E 
DC    49  50  1
AC    *  *  1

-- and likewise for the others.  :)

So I think you have found the reason why the MSAN ports are going faulty, there is a high alien AC voltage present on the pair.

As a user of a TT (LLU) service, I can say that your results are definitely abnormal. I see --

Everything connected / V
  B-A    B-E    A-E 
DC    49.7  50.9  1.2
AC     0.0   0.5  0.9
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 01, 2018, 05:24:18 PM
Indeed. It's rather satisfying to have found something that could be causing this problem. Thanks for the suggested measurements, I would never have thought to check for stray voltages on the incoming line.

Yes, I thought the DC voltages seemed to be the wrong way round . As far as I remember, when I originally installed the data extension off the SSFP the IDC labelled 'B' was the one with -50V on it, but now it's the IDC labelled 'A'. I guess they must have got swapped somewhere during one of the previous Openreach visits. Presumably this doesn't matter for anything though.

The only mystery remaining now is why this has never been picked up before. I found a video (https://www.youtube.com/watch?v=L0Q0Q7HDZ_s) showing a JDSU PQT taking place, and it does check for AC voltages. This part of the PQT has always passed however. I wonder what the input impedance of the JDSU is when measuring this. I should be able to borrow a true RMS DVM with a low Z mode from work, so should be able to get a little more info later on.

One theory I've got is perhaps this voltage is not there when the 'test conditions' are applied prior to the PQT being done. I don't have much of an idea what exactly these conditions are, but Openreach engineers have mentioned the line being connected to the test head. Presumably this means it is also disconnected from the MSAN, 50V supply and ringing generator. If, for example, the ringing generator has gone bad and is putting this voltage here, then under test conditions the voltage won't be there.

In any case, the plan now has to be to get Openreach here again and show them this, then let them investigate.
Title: Re: Failing MSAN Ports
Post by: burakkucat on September 01, 2018, 06:21:15 PM
Perhaps we should wait for any input from Black Sheep, as he will have experienced all sorts of abnormal line/circuit conditions . . . possibly with alien AC voltages present.
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 01, 2018, 06:38:56 PM
Sounds like a good plan  :)
Title: Re: Failing MSAN Ports
Post by: kitz on September 02, 2018, 02:03:31 AM
Quote
would the knowledgeable people on this forum mind taking a look at my connection statistics and let me know if you have any theories.

I'll leave all the voltage & wiring talk to others.   But just based on what I see from the graph..  it looks indicative of EMI/RFI or faulty line card.
Years ago there were a whole batch of faulty BTw line cards.  I can't recall which MSAN type they were on now, but Azzaka would recall - and BTw ended up swapping them all out.  TT use Huawei MSANs ..  but I don't think it was Huawei that BTw had the problem with.

However the symptoms when graphed, showed something like what yours is doing.   It took them a while to track it down because at first the natural assumption was EMI type REIN.  It was Zen who put forward a suggestion that it may be line card due to them having had several more techie uses who were graphing using DSLstats and routerstats...  and Azzaka who worked for Zen at that time put 2+2 together which is why he knew so much about it.

The reason I mention it...  if it is a faulty line card then swapping you to another port may not be the cure as depending which line card is in use by TT, from memory they could have up to 96 ports :/   
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 02, 2018, 10:30:24 AM
This is an interesting thought. I think we can rule out EMI/RFI here too as this would not result in the symptoms disappearing when the port is changed, nor is this compatible with the DSL dropping when the phone is picked up.

The symptoms disappearing for a few months after a port change is also problematic for the theory regarding a bad line card, though perhaps one could have a situation where all the ports on a card are in some way 'weak' and will fail after a few months use. Ideally to check this I'd record stats against some other line on this line card, naturally though I'm not going to be able to find out who else is connected to it. I do have a relative who has a TT ADSL service provisioned from the same exchange, though their connection is fine and has been for years.
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 04, 2018, 08:24:39 PM
Today I borrowed a Keysight U1232A True RMS multimeter and did some more investigation.

Firstly, the steady 110V reading from the M-830B should be disregarded. Measuring a 12V battery with this multimeter on the AC ranges results in a reading of 27V, so the 110V reading is likely just an artefact of the 50V DC on the line.

The Keysight multimeter reports much lower AC voltages: initially about 30V falling to 0.1V within a couple of seconds. On the low impedance setting a steady 0V is seen.

The Amprobe multimeter still gives the wildly fluctuating reading from 20 to 200V on auto range mode. However when set to any range manually, only 0.1V is seen.

So I'm not too sure what to make of this now.
Title: Re: Failing MSAN Ports
Post by: burakkucat on September 04, 2018, 09:40:45 PM
I think it might be best to ignore the AC readings as just artefacts of the mode of measurement and the equipment so used.

Other than steadily working through all the available ports, I can't think of any obvious way to test for a faulty line-card.
Title: Re: Failing MSAN Ports
Post by: kitz on September 04, 2018, 11:46:27 PM
Other than steadily working through all the available ports, I can't think of any obvious way to test for a faulty line-card.

From what I recall, they don't all fail at the same time which is what made it harder to find.   
I don't have all the details as it was Azzaka who was speaking to Openreach about it...  and its several years ago now.    All I do know is that when the fault presents it looked remarkably REIN like on Routerstats/DSLstats.   Unfortunately Azzaka no longer works at Zen so isnt around to ask.

It is something to bear in mind that if a line is displaying unattributable REIN like symptoms, then the port could be faulty.
Please bear in mind I may be giving you a red herring by mentioning a faulty line card, but with you having had port swaps which have also gone bad it does sound a bit suspect. 

I wonder if you can get a Lift and Shift to another Line card (not port) on the MSAN.   I wouldn't even know how to approach TT with that, as I think port swaps are usually initiated by the Openreach engineer suspecting a bad port.    Problem is with you being with TT, unlike with BTw then the request goes back to the SP. 

I would think the best course of action would be explaining to the Openreach engineer that you are now on your 4th port which fails after a certain period...  and if he thinks it could be the line card rather than port.
Title: Re: Failing MSAN Ports
Post by: siofjofj on September 05, 2018, 08:46:19 AM
I think it might be best to ignore the AC readings as just artefacts of the mode of measurement and the equipment so used.

Agreed, I'd say now these measurements are not conclusive of anything.

From what I recall, they don't all fail at the same time which is what made it harder to find.   
I don't have all the details as it was Azzaka who was speaking to Openreach about it...  and its several years ago now.    All I do know is that when the fault presents it looked remarkably REIN like on Routerstats/DSLstats.   Unfortunately Azzaka no longer works at Zen so isnt around to ask.

Ah, understood. This makes it more likely that this could be the issue here too. Sorry for doubting this earlier, I was somewhat blinded by the supposed discovery of the alien AC voltage!

I wonder if you can get a Lift and Shift to another Line card (not port) on the MSAN.   I wouldn't even know how to approach TT with that, as I think port swaps are usually initiated by the Openreach engineer suspecting a bad port.    Problem is with you being with TT, unlike with BTw then the request goes back to the SP. 

I would think the best course of action would be explaining to the Openreach engineer that you are now on your 4th port which fails after a certain period...  and if he thinks it could be the line card rather than port.

Unfortunately my experience of TT's customer service is that it is impossible to have a technical discussion of that nature. The previous port swaps have indeed been initiated by the Openreach engineers. I'll definitely suggest a Lift and Shift to a different line card to the engineer and see where we go from there.

In any case, I will make the engineer aware of the previous port failures and provide the name of the SFI engineer that identified this as the problem. Not doing this leads to a high probability of a 'Right When Tested' visit since the PQ usually passes.

Thanks very much burakkucat and kitz for your invaluable advice. I'll update this thread as I go for the benefit of anyone else that might suffer something similar, though I won't be able to do anything for the next few weeks as work commitments mean I can't have a day off for an engineer visit.
Title: Re: Failing MSAN Ports
Post by: siofjofj on October 09, 2018, 07:38:23 PM
Hi all,

Well I'm back with a somewhat late update and I'm more confused than ever.

Since the 24th September the fault has disappeared. No more disconnections, high numbers of CRCs, SNRM fluctuations or noise on the phone. The odd bit: I did nothing! I have had no Openreach visits. I did not even tell TalkTalk. Simply the last loss of sync was at 08:43 on 24/09/2018, after which the errors dropped to a far more reasonable level and the SNRM became arrow straight. After about a week TalkTalk's DLM removed the high target SNRM and set this back to 6dB and all was fine. I've now reapplied an SNRM tweak to 3db and my connection has been in sync for 5 days with only 1 or 2 errored seconds per day. As far as I can see, everything is back to normal.

However I am completely at a loss as to what the problem was. The ending of the SNRM fluctuations could suggest a REIN source that no longer exists, however this explanation is not consistent (unless someone more knowledgeable knows otherwise) with the problem being fixed previously by changing MSAN port. Nor is it consistent with the loss of sync on picking up the phone. The only thing I can think of is perhaps the line card got changed anyway due to someone else's problem / some other reason.

Any thoughts welcome  :)

Cheers
Title: Re: Failing MSAN Ports
Post by: burakkucat on October 09, 2018, 08:00:48 PM
That is good news. Thank you for reporting back.  :)

In three words: I am mystified.
Title: Re: Failing MSAN Ports
Post by: Black Sheep on October 09, 2018, 08:08:05 PM
Perhaps we should wait for any input from Black Sheep, as he will have experienced all sorts of abnormal line/circuit conditions . . . possibly with alien AC voltages present.

My sincere apologies, gents ..... I've only just seen this ??

Regarding the AC Voltage that our hand-held testers will display during a PQT test ...... less than 1.0v is quite common, and even at 4.0v or under the PQT will still pass ..... and as mooted by you both, it is purely inductive from nearby power sources.

Title: Re: Failing MSAN Ports
Post by: burakkucat on October 09, 2018, 10:04:13 PM
My sincere apologies, gents ..... I've only just seen this ??

No problem and no harm done. We got there in the end.  :)