Kitz Forum

Broadband Related => FTTC and FTTP Issues => Topic started by: Chrysalis on June 25, 2021, 08:19:47 AM

Title: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 08:19:47 AM
Line went down briefly just after midnight I assumed was an AAISP outage at time.

Noticed just now netflix is taking noticeably longer to buffer, and can see sync speed dropped from 74.5meg capped by myself all the way down to 27576kbit, and line is interleaved, in addition the line is failing to reach max upload speed.

Very bizarre.

The DLM action is not odd as I had high errors since around 5pm according to my graph, so that is normal but the drop in sync speed is massive.  I actually think this DLM behaviour is better than the old one where my line would be running with errors for 24 hours before action.

Sadly didnt get my wish of hassle free until FTTP.

The drop in sync speed happened when the errors started I only just noticed things been slower.

There is currently a faulty streetlamp right outside my house, it flashes on off all night, not sure if that has any relation though.  But notice the error rates were higher before 8am as well, so tonight will check times lamp starts flashing and try and see when it stops tomorrow.

Attenuation changed a fair bit.

Old stats

Max:    Upstream rate = 28242 Kbps, Downstream rate = 77244 Kbps
Bearer: 0, Upstream rate = 20000 Kbps, Downstream rate = 74481 Kbps

Link Power State:       L0
Mode:                   VDSL2 Annex B
VDSL2 Profile:          Profile 17a
TPS-TC:                 PTM Mode(0x0)
Trellis:                U:ON /D:ON
Line Status:            No Defect
Training Status:        Showtime
                Down            Up
SNR (dB):        6.8             10.9
Attn(dB):        15.2            0.0
Pwr(dBm):        5.0             5.0

New stats

Max:    Upstream rate = 19097 Kbps, Downstream rate = 25825 Kbps
Bearer: 0, Upstream rate = 18786 Kbps, Downstream rate = 26239 Kbps

Link Power State:       L0
Mode:                   VDSL2 Annex B
VDSL2 Profile:          Profile 17a
TPS-TC:                 PTM Mode(0x0)
Trellis:                U:ON /D:ON
Line Status:            No Defect
Training Status:        Showtime
                Down            Up
SNR (dB):        5.3             6.2
Attn(dB):        22.6            0.0
Pwr(dBm):        6.8             6.8
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 09:03:34 AM
Old and new bitloading data.

Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 09:46:21 AM
Also attaching hlog graphs, as some find these useful.

The line I can see from aaisp's logs is passing all tests, I am surprised GEA no longer auto fails on a large speed drop.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: RealAleMadrid on June 25, 2021, 09:50:44 AM
There has been a significant change in the line attenuation, the conclusion I would draw is that you have a copper line fault which needs to be fixed.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 10:27:02 AM
So far I have gave it 10 mins, and offered aaisp to use test socket which its in now, just to rule this stuff out, I expect they will raise the fault. :(

The test socket feels quite loose though, its not a satisfying click followed by a snug fit.  If I wiggle it trying different positions there is no change though.

Thanks RealAle for the reply and advice.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 25, 2021, 11:00:51 AM
Quote
Line went down briefly just after midnight I assumed was an AAISP outage at time.

Noticed just now netflix is taking noticeably longer to buffer, and can see sync speed dropped from 74.5meg capped by myself all the way down to 27576kbit, and line is interleaved, in addition the line is failing to reach max upload speed.

Very bizarre.

The DLM action is not odd as I had high errors since around 5pm according to my graph, so that is normal but the drop in sync speed is massive.  I actually think this DLM behaviour is better than the old one where my line would be running with errors for 24 hours before action.

Sadly didnt get my wish of hassle free until FTTP.

The drop in sync speed happened when the errors started I only just noticed things been slower.




The MTBE/MTBR figures quoted on the main site were very recently confirmed as still being applicable. 
The only changes that I'm aware of is the trial Temporal DLM system which Ive mentioned elsewhere  and that some ISPs may have some customers using the new GEA Stable profile, which is quite harsh.


From the linestat data you provided, the line speed dropped at about 7:30ish to below 27Mbps which looks to me like a result of a fault rather than anything to do with DLM because both your downstream & upstream SNRM also dropped drastically at this same time. 

After 7:30ish there was insufficient SNR left in your line to allow you to sync higher than 27Mbps, rather than being capped.   The drop in SNR can obviously be attributed to the increase in attenuation... suggesting a cooper fault.   
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 25, 2021, 11:09:37 AM
btw your hlog is slightly wavy.  I wonder if running a GEA will show that as sufficiently noticeable enough to trigger 'bridge tap' on their system.   

Would be interesting if AAISP could run a GEA test for you.    Whatever way you need to get the decrease in [real]SNR and attenuation reported as a line fault asap.


---
PS
Just remembered that the AAISP CP allows you to run your own GEA test... in which case something you said indicating you may have run one now makes sense.   So apols if you did run one.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 11:55:49 AM
They ran it here is the results I see kitz.

Test Description = GEA service test completed and no fault found .
Diagnostic Code = GTC_FTTC_SERVICE_0000, Sync Status = In Sync, NTE Power Status = PowerOn
Down Sync = 27Mbps, Up Sync = 19Mbps
REIN = Not Detected,
Test Outcome = Pass, Main Fault Location = OK

Also was a WLR test which passed.

I am still waiting for them to phone back, but I really wanted to be sure so I have now even swapped the modem.  Still no change of sync speed.  Thats why I got no live stats at the moment, the billion modem is plugged in.  I can see in its UI though its just under 30mbit sync.

Also yeah initially it was just a drop in line speed, but when it went down error rates went up as well, and DLM kicked in just after midnight which was the outage I noticed. (DLM only lost a further 2mbit or so of sync, its just an interleaving profile not banding I think).  I said in first post DLM did its job, it isnt the issue at hand. :)

Thank you.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: RealAleMadrid on June 25, 2021, 12:15:49 PM
An 80/20 service that was syncing at 74Mbps is now at 27Mbps and the GEA test thinks that is OK. I would disagree with that test result, the downstream speed is ridiculously low.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: meritez on June 25, 2021, 12:59:48 PM
An 80/20 service that was syncing at 74Mbps is now at 27Mbps and the GEA test thinks that is OK. I would disagree with that test result, the downstream speed is ridiculously low.

I agree, my minimum threshold sync is 58Mbps before I can start shouting for an OR engineer.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 25, 2021, 01:18:53 PM
Well apparently there was a big power cut in another part of my city around the same time the line problem started, seems a bit of a coincidence?

AAISP have raised the fault now.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on June 25, 2021, 11:12:03 PM
The downstream to suddenly be lower than what the upstream estimate was before but the system thinks thats okay?   :lol:
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 26, 2021, 12:55:28 AM
I have plugged back in the zyxel now and front plate back on to be left alone until Openreach turn up whenever that may be.  So stats will be visible to people again.

I also noticed is barely any water now extremely low pressure, I kind of think something has broke somewhere with a utility company and there is some chain reactions going on.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 26, 2021, 11:00:44 AM
I have plugged back in the zyxel now and front plate back on to be left alone until Openreach turn up whenever that may be.  So stats will be visible to people again.

Had another look at your hlog, there definitely is something going on there.  I can see a slight but distinct repetitive wave in the plotted line. Your SNR is also reflecting this although harder to spot.  A JDSU would probably pick it up and do the relevant calculations as there is a formula used based on the number of tones between dips as to whether its something serious or not.

 I am quite surprised that your GEA test passed because the sync is so low.  I was also a wee bit surprised that your GEA test didn't show any mention of Bridge Tap results - bythat I mean there wasnt any mention of it all... as the ones I've seen run by Plusnet usually say something like

Code: [Select]
Bridge Tap Not Detected
Radio Frequency Ingress Not Detected
Repetitive Impulse Noise Not Detected
Cross Talk Not Detected
Profile Name 40M-80M Downstream, Error Protection Off - 10M-20M Upstream, Error Protection Off

It may however perhaps just be the way AAISP do the formatting and chose not to display some of the other tests run in their CP  such as

Code: [Select]
Upstream Rate Assessment Very Good
Downstream Rate Assessment Very Good
Interference Pattern Not Detected
Service Impact No Impact Observed
Home Wiring Problem Not Detected

Anyhow the main thing is that an engineer visit has been arranged.   I've no doubt he should be able to sort it for you.  Good luck :)




Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 26, 2021, 11:26:04 AM
Yeah the aaisp GEA data is not showing all data I reckon, I also remember the results from PlusNet e.g. show your DLM profile, whilst the one provided by AAISP doesnt show it.

Thanks for your comments, and everyone else also.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: underzone on June 26, 2021, 12:57:04 PM
Just be glad you are with AAISP,  this is where they will earn their keep!

Their higher prices are suddenly well worth it when problems like this arise. Good luck!
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 26, 2021, 02:27:13 PM
Quote
I also remember the results from PlusNet e.g. show your DLM profile, whilst the one provided by AAISP doesnt show it.

I seem to remember something about there are different tests that can be run depending upon permissions and which section of the CP is used.   I haven't actually seen this myself, its just something Ive seen mentioned in conversation with someone who does have access.  The info certainly looks cut down when compared to the full version.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 26, 2021, 03:25:45 PM
Yeah that may be the case.

Anyway engineer is now booked for Monday afternoon, a lot sooner then I expected given covid etc.

I have a 3 week wait to get water supply back.  To put into perspective of waiting times.

I will be very curious if I find out what happened here, having water fail, a power cut in another part of city and this happen all in close proximity seems like at least 2 out the 3 might be related.  The power cut I was told was between 7 and 8 not an exact time given.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 26, 2021, 03:50:53 PM
Here is a scenario that might fit the facts . . .

A large diameter, high pressure, high flow rate water main develops a leak / bursts. The escaping water washes out significant quantity of sub-soil. A road or pavement subsides. The subsidence damages other underground plant -- sewage, electricity, gas, telecommunications. To re-establish telecommunications, Openreach lay a temporary diversion cable around the hole. That diversion cable is crimped onto the existing cable at appropriate access points before the damage at hole. There are now two shortish-length bridge taps on each and every pair in the cable. Hence the abnormality in the Hlog plot for your circuit.

----- existing cable ----- damage ----- existing cable -----
                               |                   |
                               \                   /
                                 \                /
                                  |              |
                                   ------------
                                Diversion Cable
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 27, 2021, 04:02:07 AM
The first part sounds viable, not sure about laying a temp diversion though as would that not involve a short period of downtime or which there didnt appear to be any.  I'm with you on water could cause u/g duct to collapse and a diversion cable causing bridge taps though and that part of the theory is probable. 
 
I think it was wwwombat who used to be quite good at having a stab of location from home based on hlog first wave dip and distance between the dips..  which is something the JDSU does automatically based on hlog and (real)SNR.  As far as I can make out, those wavedips run quite evenly throughout almost all of the tones, but may possibly start at around tone 450ish. 

Below tone 300 is rather messy on the QLN - this also shows up in the Bit loading and real SNR.  Look how many tones are knocked out at around 120-300.   However that alone isnt sufficient to account for the total amount of speed loss.   Something is causing a load of noise in those lower tones...  and its then causing a reflection type wave on the hlog in the higher tones.   

I wouldn't like to guess what could be causing that, but its certainly not impossible for water to cause duct collapse and damage to other cables.  Not going to spend any more time theorising as I'm sure something like this would be picked up by a JDSU ,  but it would be nice to know what the engineer says what the cause was.


However, I have almost convinced myself that the GEA run by AAISP doesnt include the full suite of tests because (1) We can see what looks like signs of a bridge tap in hlog with the naked eye*  and (2)  It should have failed the downstream rate assessment test.



*Of all the tests run, the bridge tap test is usually quite sensitive; There's many a time I've seen FAIL reported when (a) there's hardly anything to see on hlog or (b) it gives a false positive.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 27, 2021, 09:36:04 AM
. . . not sure about laying a temp diversion though as would that not involve a short period of downtime or which there didnt appear to be any.

There are three-way crimps which have a slot and, thus, can be placed over a contiguous length of wire. Such crimps can be used in those cases where a disconnection is to be avoided.  :)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: boozy on June 27, 2021, 12:21:28 PM
There are two taps, both between 20m and 30m long, going from the Hlog.  BCat seems to be the winner. 

I don't think you can figure out how far away they are - but if someone knows I'd love to find out.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 27, 2021, 08:54:59 PM
I did look at existing works for severn trent near me, incase its already planned.

There is a lot, seems they have a lot of faults, but none seem to be on the route I think my cable takes and none were mentioned for my street specifically, which is why I reported a new fault to severn trent on saturday.

I will of course let you guys know if I get anything useful from the engineer.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 27, 2021, 11:45:26 PM
There are two taps, both between 20m and 30m long, going from the Hlog.  BCat seems to be the winner. 

b*cat waves a paw.  ;)  I took a look at the second of the Hlog plots shown, above, and it was very much a case of "tingles in the whiskers" telling me that it was not a single tap.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 28, 2021, 12:00:49 AM
There are three-way crimps which have a slot and, thus, can be placed over a contiguous length of wire. Such crimps can be used in those cases where a disconnection is to be avoided.  :)

Ah .. are you saying that they may have in effect cut into the routing that Chrys's line takes to make a repair to another site?   I think I discounted that because I could see how that would then make Chrys's routing longer.   I was working of the assumption that because Chry's line didn't actually go down, then his routing was unaffected.  I couldnt see how that if a mains collapsed elsewhere and unrelated to him... couldnt see why his line length would suddenly get longer.
   
I did try then envisage various routings in my head but my fingers werent keeping up at this point and I thought not worth bothering about because there were too many things we didnt know such as where the power cut was could be right over the other side of the exchange and nowhere near him anyhow.   If Openreach have spliced into Chrys's routing I will imagine it will just come back up right again when Openreach get around to sorting whatever is wrong elsewhere in his city.   I hope it isnt that, because it could take ages to be able to fix...  and they may not even be able to do anything about it tomorrow.     
It took best part of a year to fully fix the collapsed duct in my daughters street and even after it was supposedly fixed, the line never ran as good again as it did previously..  and it was a partial reason for them ditching g.fast.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 28, 2021, 12:17:38 AM
There are two taps, both between 20m and 30m long, going from the Hlog.  BCat seems to be the winner.

Sorry, but unless I'm missing something, I really can't see another tap and can only see the one continuos wave. It's the regular up and down of a wave that indicates the presence of a bridge tap.  I can only see one distinct regular wave. :(

I'm not sure what is going on at apprx tone 150 but since that narrower wave bump does not recur across the whole band-plans plus it also coincides with a load of noise in the same tones then that to me would indicate something else.... 

... and whilst there is something else happening at <tone 300 but its not just solely visible on the hlog.   At this point we need to switch to QLN, bit load & SNR where we can see something noise related going on here.  Whatever it is, it is showing in the real SNR and has caused zero bit loading up to around tone 300 which will amount to a chunk of his speed loss.  It's also showing in the QLN so its a sure assumption that its not a tap but something more like a HR fault is causing an issue at those lower tones up to around 300-350ish.

The first (repeating) dip that I can make out occurs at around tone 500ish / 2.16MHz, which I think equates to a tap length of about 22m.

Quote
I don't think you can figure out how far away they are - but if someone knows I'd love to find out.


I think wwwombat worked out something.  I recall him being quite accurately being able to diagnose one particular fault to within 1m of how far the fault was located from the NTE.  He managed to do this on more than one occasion. He did say he was going to write more about it, but I think it was one of hose things he never got around to. :(

b*cat may have more recollection because they were iirc talking about the formula in PMs too.  About the only other thing I can recall is that he didn't place too much importance on the number of dips,  but more to do with the frequency at which the first repeating dip occured.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 28, 2021, 01:16:09 AM
I think wwwombat worked out something.  I recall him being quite accurately being able to diagnose one particular fault to within 1m of how far the fault was located from the NTE.  He managed to do this on more than one occasion. He did say he was going to write more about it, but I think it was one of hose things he never got around to. :(

b*cat may have more recollection because they were iirc talking about the formula in PMs too.  About the only other thing I can recall is that he didn't place too much importance on the number of dips,  but more to do with the frequency at which the first repeating dip occured.

We, wwwombat & b*cat, used a table, shown in an JDSU Application Note, that allowed an estimation of the length of a bridge tap to be calculated. If Chrys still has the raw data, from which the above Hlog plot was generated, it would be interesting to have a play with the data. I also know, from prior PMs, that boozy would also be interested in seeing the raw data.  ;)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 28, 2021, 01:26:15 AM
Ah .. are you saying that they may have in effect cut into the routing that Chrys's line takes to make a repair to another site?

I have worked on the assumption that the Openreach ducts were left hanging over the hole and the cable(s) therein were not severed. To allow a proper reinstatement of all the services and to allow the hole to be filled in, an appropriately sized diversion cable was connected either side of the hole. Using three-way crimps, the live services could be transferred to the diversion cable. Then the unsupported duct(s) could be cut away, the cable(s) therein severed and thrown to either sides of the hole. Those two ends of the original cable(s) would then show up as two bridge taps on each and every pair.

However all of the above is just pure speculation based upon what Chrys has reported.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 02:31:20 AM
I do have high power cut back on adsl1 tones, so those tones have always had bitswapping and low bit count.

Also as you said kitz I do fear there will be no fix tomorrow (later today, really need to go bed), as I expect this could be damage caused by another utility company, just a wait and hope.

ps. my water is back to normal now.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 02:35:17 AM
We, wwwombat & b*cat, used a table, shown in an JDSU Application Note, that allowed an estimation of the length of a bridge tap to be calculated. If Chrys still has the raw data, from which the above Hlog plot was generated, it would be interesting to have a play with the data. I also know, from prior PMs, that boozy would also be interested in seeing the raw data.  ;)

I have historical files including a hlog file full of numbers, is that what you after?

The 00-51 file is from start of june so before fault, 00-17 is today.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: boozy on June 28, 2021, 12:11:26 PM
Sorry, but unless I'm missing something, I really can't see another tap and can only see the one continuos wave. It's the regular up and down of a wave that indicates the presence of a bridge tap.  I can only see one distinct regular wave. :(

You can see the first two individual troughs for each of the taps (~150 then ~300 and ~200 then 450) after that the effects of two taps generally bleed into each other.  This depresses the whole Hlog much more than a single tap.
Chrysalis has posted the Hlog data so I'll try to pass it through both a low pass and a high pass filter to separate the two waves - it may not work because of the Hlog depression (you can see a tap much better at the low frequencies as the signal is stronger an the reflection causes more of an effect).

For the first dip in each wave the tap length can be calculated as:
Code: [Select]
        const double speedOfLight  = 299792458.0;
        const double frequencyStep = 4312.5;
        const double propogationVelocity = 0.666666666666; // can be changed for different Wire thicknesses
        const double baseDistance = speedOfLight * propogationVelocity / (frequencyStep * 2); // 2 is for there and back to get the distance
        static double baseLn = Math.Log(baseDistance);

//Above is pretty much a constant

// First Dip:
        answerInM = Math.Exp(baseLn - Math.Log(firstTone*2));

//Multiple Dips:
        answerInM = Math.Exp(baseLn - Math.Log(tonesBetweenDips  / (noOfDdips - 1)));

We're on our staycation (or visiting the relatives we haven't seen in two years to be accurate), but I'll try to find the Butterworth Filter code this evening and separate the tap curves.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 28, 2021, 01:30:58 PM
I do have high power cut back on adsl1 tones, so those tones have always had bitswapping and low bit count.

Yeah, I can recall your PSD spectral mask has always been quite harsh*, but in theory the DSM (Dynamic spectral management) should see your lower attenuation and compensate accordingly.  From your stats in first post it appears to have done so and it's also increased power output in an attempt to increase your bitload.   It's the spikeyness in your QLN which indicates that the low bit load is a result of noise rather than power management.

However its worth bearing in mind that because Openreach uses DSM, then a line which is in fault and has (false) increased atten will be allocated a different mask than if that same line was clear. The DSLAM sees the increased attenuation and assumes you are further from the DSLAM, so will automatically allocate a mask with less PCB in an attempt to increase bit load in the lower band plans (U0,U1 & D1) if applicable. 

If you look closely you will see that your U1 bandplan actually now has a better bitload than before the fault.   Your previous mask allowed max 4 bits and then dropped to 2 bits (typical of some masks within reasonable/mid distance to cab) whereas now it starts max of 5 and drops to 4.  It's the change in your U1 that confirms to me that you are now on a different spectral mask and DSM is attempting to give you more power over those lower band plans in compensation for the increased attenuation.


*It's sometimes possible to take a rough guess at someones distance from the cab and distance from the exchange based on the shaped of their PSD mask. 
I can usually tell the difference in those lines which are near to exchange but some distance from the cab -v- lines which are at a distance from the exchange but close to the cab, based entirely on their spectral mask.

Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 28, 2021, 01:42:18 PM
You can see the first two individual troughs for each of the taps (~150 then ~300 and ~200 then 450) after that the effects of two taps generally bleed into each other.  This depresses the whole Hlog much more than a single tap.

Thank you for posting further.   Yes I did notice the 150/300 but as I saw no repeat of the pattern..  AND because of the mess going on in QLN/Bitload/SNR over those very same tones discounted it as related to the noise generated in the same tones.  There definitely is something going on in those lower tones indicating noise.

The tap beginning at around 500 was more regular and clearer to see and I would have thought that because those earlier waves only had a width of less than 100 tones it would have made the overall plot look rather bumpy... or at least bumpier than I see with the naked eye.   The curve of 450ish isn't that great and could easily be overlooked as it is, so I didnt do any counting between dips to see how exact the wave was.   I would be interested if you are able to separate the 2 waves please as I don't have anything program wise which takes the raw data to plot specific wave graphs.
 
I accept that its possible they may bleed but without specifically analysing each individual tone, I still cant see that early bump reoccur on the graph alone.  Bearing in mind its causing at least 2 waves every graph line (300 tone divisor), there are several parts of the graph showing a smooth 300 tones eg 3000-3300 showing absolutely no sign of a narrower reflection wave, yet continuing with the wider wave that is around 450/500 tones. 

Is is possible that there could be some sort of damage to a joint or cable which would cause all that HR type noise, but then cause a bridge type effect of a terminal that is not terminated causing the reflection.  Damaged cable?


Quote
We're on our staycation (or visiting the relatives we haven't seen in two years to be accurate), but I'll try to find the Butterworth Filter code this evening and separate the tap curves.

No worries.

TBH I only mentioned the bridge tap because I spotted it, but in Chrys's case with the massive loss of speed it should be a relatively easy fault to find and hopefully clear the same day.
I don't usually bother analysing hlogs in great detail unless in a case which is proving difficult to clear,  as I personally think there's no point speculating if we dont know area/circumstances when its something that has more obvious signs of a fault and that should be easy to fix for an engineer who is on site.   

However as the topic has been brought up at a time in-between flares when there are windows in the day when I'm not in a morphine haze... and because wombat never did get around to publishing his thoughts, then it is interesting to revisit.   Thanks :) 

   
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 03:21:59 PM
Thanks for the explanation kitz I get what you are saying now.

I am starting to remember the days gone by when BT would not turn up for afternoon appointments, still no one here yet today.

Thanks to Boozy and everyone else as well.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 28, 2021, 04:35:42 PM
I have historical files including a hlog file full of numbers, is that what you after?

The 00-51 file is from start of june so before fault, 00-17 is today.

It is the raw data that produced the Hlog plot, the second image attached to your post (Reply #2) dated June 25, 2021, 09:46:21, that would be the most interesting to see. I'll take copies of both files and have a look at the data. The first of the two (Hlog-00-17.zip) is probably the one of most interest.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 05:28:53 PM
Just be glad you are with AAISP,  this is where they will earn their keep!

Their higher prices are suddenly well worth it when problems like this arise. Good luck!

We will see, they refused to chase up today until the slot is missed and its too late, I have just read focused openreach appointments exist so when they have to rebook this will they do a focused appointment for me or just the cheaper 5 hour slot and of course I cannot even attempt to rebook until tomorrow as they close when the afternoon slot ends.  Also the question of if I will get any compensation.

Might they still turn up? as I lied down they rang once and and I mean just one ring like a test call, wonder if engineer to say he tried to ring to get on his call log.  Or if they will turn up.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 05:31:55 PM
It is the raw data that produced the Hlog plot, the second image attached to your post (Reply #2) dated June 25, 2021, 09:46:21, that would be the most interesting to see. I'll take copies of both files and have a look at the data. The first of the two (Hlog-00-17.zip) is probably the one of most interest.

Thanks, going for a lie down I will maybe check back this evening to see if you have come back with anything. :) 
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 28, 2021, 06:48:14 PM
Openreach call centre just rang, usual excuses about booking too much work for too few engineers, I could have had wed slot but was afternoon so is now thurs morning.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: boozy on June 28, 2021, 11:25:50 PM
I checked my figures again and had the lengths embarassingly wrong (one at 60m and one at 80m as opposed to 20 and 30) because I'd put upper and lower limits in the UI. :-[  The maths earlier is correct. That would also mean less depth to the troughs as the signal is degraded more after travelling that for, causing less interference.

Using Butterworth high and low pass filter didn't work brilliantly but it's entirely possibly it's correct (I'm thinking through it with the help of a beer, but may need another).  High Pass is essentially whats left after the low pass is applied (which equates to error with no band pass in the middle).  Its says 40 and 45m, but I think I might need to use a high number of decimal places due to the frequencies being close.

Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 29, 2021, 12:13:54 AM
My manual method seemed to show the first four local minima to be --

M1        168
M2        204
M3        380
M4        504

Hence deltas of --

D1        212
D2        300

By lookup in Table 2 (attached to Reply #26, above) and interpolation in the case of D1, I make the two tap lengths to be --

T1        109.3 metres
T2         77.3 metres

Meaningful? Sensible?  :shrug2:
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 29, 2021, 04:11:52 AM
I am trying to figure out what you guys are doing and I will admit I dont know but it is great, I expect you will turn out to be right if I get told whats happened by the engineer.

I did find this.  It is a bit odd two taps would suddenly appear with no intermediate outage?

https://www.youtube.com/watch?v=JDCc4pkdTLE
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 29, 2021, 05:33:20 AM
Thanks guys for the further graphs and explanations - appreciated, but some further questions


I've never seen a bridge tap wave distinctly change width shape before. It is my understanding that a bridge tap produces a fairly even wave that will continue across the whole spectrum.   It is the frequency at which the first dip occurs which is used to calculate the length of the bridge tap. 

Even though we may primarily use hlog for detection of bridge tap, it does also show up capacitive problems that can cause noise and increased attenuation. 
I have seen certain types of severe noise make an appearance in hlog as one off narrow dips on many occasions before now.  iirc even some of the sample graphs in other learning documentation can show slight noise on hlog of perfect lines or outside of the fault area. 

I'm afraid you've lost me a bit with the more complex calculations and what you are trying to do with the table.  It was my assumption that wombat was doing further investigations to be able to calculate the appx location of the bridge tap.  I may have not gotten what he was trying to achieve, but I have at least once seen him be able to estimate the distance of the fault from the NTE.  I distinctly remember BS congratulating him upon diagnosis of being within 1m.   I never really got much involved in those conversations primarily because I don't have tools to analyse the raw data and was content with wombat saying he would later write up his findings, I just waited for that to appear and only read what was said in public.

I'm not quite sure what the additional table was meant to bring other than I know wombat was doing further investigations into it, but conversely I also pretty sure I do remember him concluding that the first dip was the most important. Since b*cat was actively involved in what wombat was doing with that table, then he will know far more about it than me.

The formula I use is the standard set out by the broadband forum TR-197 namely
Quote
160 ÷ Frequency in MHz = Tap Length in feet using the frequency in MHz at the point of the first null, or center frequency of the dip.




IMHO because of all the noise in tones <500, I'm really not sure if we can draw anything conclusive about those first early dips.   I really cannot see anything where they repeat outside of that noise zone.   The first regular and continual wave I see starts at around tone 500. I deliberately didn't mention a distance in my first post because I was unsure about those early tones and to the naked eye the first visible continual wave only started after that.    However when pushed I think I said about 22m... which uses the TR-197 formula as such

160 ÷ (Tone 500 = 2.16MHz) = 74 ft = 22.5m

If we start theorising, based on we know chrys has had no water in his direct area and there was no electricity in a totally different part of the city:   
What if a burst mains caused the collapsed duct and damaged the telephone cable or joint.  The cable and joints are underground and under water. Moisture in the cable is known to cause resistance changes and capacitance faults, which can and do show on the hlog.  What I'm not certain about is if this sort of damage can then also trigger off a bridge tap effect.  It doesn't seem impossible that damage could cause something elsewhere not to terminate properly. 
Other theories seem to ignore everything that is happening in those tones up to 500 - which are abundantly clear in all the other graphs. 

I am interested in what you are trying to do, but it's just that I cannot understand what is going on with the 2x50 tone wide waves that appear to vanish after the noise stops.  If there wasn't all that noise in several other graphs telling us something else was going on, then perhaps I could.   I don't like making assumptions without knowing full facts but with the mention of water problems, its water ingress and damaged cable/joint more making my whiskers tingle - especially from what all 4 of the graphs (QLN,bit-load,SNR & hlog)  are showing us for tones 0-500.  I'm not trying to dismiss anything you say, its just that those early tones are telling me something different is going on.



Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 29, 2021, 05:35:52 AM
The city location isnt that far from me, it where my sister lives, about 5 mins drive.  Not really local to the FTTC cabling but its possible the power grid to her area runs through my area?

Whilst looking at my bitloading to see what kitz told me about my power cutback been removed from upstream, I noticed U2 seems less affected and is achieving higher tones but I guess thats down to less noise.

I will speculate that assuming my line gets fixed, there may be other pairs still damaged, and since there is now different power masks been used across different lines, some weird crosstalk effect may happen.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 29, 2021, 06:30:27 AM
Openreach call centre just rang, usual excuses about booking too much work for too few engineers, I could have had wed slot but was afternoon so is now thurs morning.

Damn.   Thats annoying that you have to wait until Thurs.   I'm hopeful though with it being such an obvious loss in speed that the fault can easily be found.

Whilst looking at my bitloading to see what kitz told me about my power cutback been removed from upstream, I noticed U2 seems less affected and is achieving higher tones but I guess thats down to less noise.

I will speculate that assuming my line gets fixed, there may be other pairs still damaged, and since there is now different power masks been used across different lines, some weird crosstalk effect may happen.

DSM has very obviously given you a different power mask (PSD mask).  The DSLAM has seen the increase in attenuation and thinks that you are further from the cab than you really are.   

atm its working in your favour on the upstream and allowing more bits to load in your lower tones than they would normally.  Because most of the noise is <500 its only really of the most advantageous to your U0.   The DSLAM is trying to give you more power to keep your speed up.  If you notice your power figure has also changed in your stats.   
Unfortunately your downstream cant benefit because of all the noise that is happening in D1, but when I looked and compared old and new the other day, even the D1 shape had slightly changed.   So yeah the DSLAM has noticed the noise and trying to keep you connected by increasing your SNR > bitload.   If you were still on your old PSD mask, then your sync speed would be a bit lower than it is now.

Once your line gets fixed then the DSLAM will notice and DSM will immediately put you back on your old mask. 
 
Shouldn't really make much difference to crosstalk.  The whole idea of the DSM is to apply masks based on individual line conditions.   Borked lines are under performing so even though they may be on a 'better' mask, they are too sick to take full advantage of it, so wont be able to give you any more crosstalk issues than they would under normal circumstances. 

Thats the beauty of Dynamic Spectral Management compared to the old system;- the DSLAM only allows longer lines more power...  but if a line is under performing it can also ramp up the power for those lines which need it. In days gone by there were only a handful of masks and you stuck with what you were initially allocated.  Now there's many that can change each and every time you sync up depending upon line conditions.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: boozy on June 29, 2021, 08:06:20 PM
I wasn't going to post this particular reply because the bit in bold is confusing me, but I can't think of anything better than 2, or 4 (2 on each leg), taps - so I'll throw it out and hopefully someone will be able to explain it.

The two figures I was using are 150 and 200 (rough estimates of the two first dips equating to 58m and 77m) BCat went more accurately to 168 and 204.  How confident would I be about any of those...  well I wouldn't bet on any numbers (as I've thought too much about it). The biggest head melter is that the Hlog from the impaired service looks like it would be from a much shorter line than the original even without being fixed :)

The first dips are likely to be best as you get into interference in the higher bins. Additionally if the tap is long then there are no "dips" just a general depression (https://www.analog.com/-/media/images/analog-dialogue/en/volume-34/number-1/articles/vdsl-technology-issues-an-overview/vdsl-fig-03b.gif?la=en&vs=1).  What happen for two taps... not sure but I'll come to that.

A VDSL modem has in effect 4096 56K modems each with a different frequency (bin/tone). 
Taking just one 56K modem:  data is transmitted on the particular frequency it "owns".  The data goes down the conductor until it reaches the tap.  At that point the waveform goes down the tap and simultaneously on to the receiver.  The waveform travels down the tap, slightly degrading as it goes, and reflects off the end.  After reflection, where the resulting waveform will fall somewhere into the range of having destroyed itself to reflecting almost perfectly, what remains will travel back up the tap and crash back into waveform at the tap start.  The longer the tap, the less the effect of the crash (degredation) and the same for the worse reflection.

With two taps I think something like two source interference will also be happening (http://www.physics.louisville.edu/cldavis/phys299/notes/lo_interference.html), light also being a transverse wave. I can't picture what will happen, but messy would be my guess.

I think that was a long way of saying I don't trust any of the period measurements, but would go for 168 and 204 as the best bets for first dips(I used 150 and 200 as mine originally).  I have no clue why the attenuation is less with the degraged service - that makes no sense at all.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 29, 2021, 10:35:26 PM
It is a bit odd two taps would suddenly appear with no intermediate outage?

My two posts (Reply #18 (https://forum.kitz.co.uk/index.php/topic,26103.msg438005.html#msg438005) and Reply #20 (https://forum.kitz.co.uk/index.php/topic,26103.msg438041.html#msg438041)), above, show one example of how that could occur. I won't be surprised if someone comes up with other variants.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 29, 2021, 10:53:28 PM
The formula I use is the standard set out by the broadband forum TR-197 namely
Quote
160 ÷ Frequency in MHz = Tap Length in feet using the frequency in MHz at the point of the first null, or center frequency of the dip.

Having no recollection of that formula, I went and downloaded both Issue 1 and Issue 2 of TR-197. (I confess to going to sleep after reading about two thirds of Issue 1.)

Carefully searching through both issues of the report, I could not find anything that resembles the formula . . .
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on June 30, 2021, 01:26:00 AM
In days gone by there were only a handful of masks and you stuck with what you were initially allocated.  Now there's many that can change each and every time you sync up depending upon line conditions.

So is that why in the early days they limited what speed you could order so strictly, even though your line technically could easily handle it?

For example initially I was only allowed to order 500Kbit, later they allowed 1Mbit and I even tried for 2Mbit which was refused - but somehow they accidentally processed the order anyway and I did indeed get 2Mbit.

Then once I moved to ADSL2+ with a different company (forgot the name off the top of my head) which had their own DSLAM in the exchange, it was allowed to sync at whatever the line could handle.  When I moved back to Openreach hardware, DSLMax had become a thing and I was forced only a much slower service as I could no longer use 3dB SNR downstream.

Its easy for us to think "oh but it SHOULD handle it", forgetting that the telco have to ensure our lines are blasting out too much interference to other customers.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 30, 2021, 02:21:11 AM
Apols for incorrectly remembering the source as the formula has been referenced several times on this forum particularly by wombat over the years.
Source is actually the official JDSU documentation which I'm sure you will have a copy of already. 

In fact if you google the formula, it actually pulls up wombats thread (https://forum.kitz.co.uk/index.php/topic,16536.msg306163.html#msg306163) which contains a link to the JDSU documentation in case you dont, and with the lapse of time his post shows why I may have easily thought that the info originally came from TR-197.   I'm glad you made me search so I can re-read the thread again properly and check if he did make any further posts about the topic.

Quote
Bridged Tap
Impact HLog data can indicate the presence of certain impairments.

The HLog graph in Figure 2 shows a dip at Tone #1325 for the first null (approx. 5.7 MHz [1325 x 4.3125 kHz]) of an impairment caused by a bridged tap. The dip Figure 2 that looks like a valley with sloping sides is the tap signature.

The tap length can be estimated with this formula: 160 ÷ Frequency in MHz = Tap Length in feet using the frequency in MHz at the point of the first null, or center frequency of the dip.




Here is a link to *another* document which mentions bridge taps, but this one has a couple of decent examples:
https://www.broadband-forum.org/technical/download/TR-197.pdf

This document is TR-197 from the Broadband Forum, discussing DSL quality factors. The relevant section is on "dual-ended line tests" that are performed in some DSL modems.

Figure 10, page 51, shows the Hlog for a line in 3 states:
- Perfectly normal
- With a 30m bridge tap, first dip somewhere around tone 520. We get to see one further dip.
- With a 200m bridge tap, first dip somewhere around tone 55. We get to see 13 further dips.

Sanity check of the previous formula:
- Tone 520 = 2.25MHz. Tap length = (160 ÷ 2.25) ft = 71 ft = 22m.
- Tone 55 = 0.24MHz. Tap length = (160 ÷ 0.24) ft = 667 ft = 203m

The sanity check isn't far off, telling us the formula is a reasonable estimate. In that case, we just measure the first dip, and ignore all the rest.

With gazaai's first dip occuring at tone 164, I calculate:
- Tone 164 = 0.71MHz. Tap length = (160 ÷ 0.71) ft = 225 ft = 69m

I think that gives us the answer for the total tap length.

That backs up (nearly) just reading data from table 2 (in the original JDSU document) as the only step needed. I don't think you need to perform averages over the extra dips (except for the fact it might give you a better accuracy for where the first dip really was), nor to multiply by the number of visible dips (after all, in reality, those dips go on ad-infinitum).

I also think I know how to read the example in that 1st document better now ... but I'll have to do that in a separate post, likely tomorrow.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on June 30, 2021, 04:19:10 AM
@boozy

Thanks for the further explanation and link which I shall have to read later. :/  It was not my intention to be up at this ungodly hour but circumstance dictates that I am, but it also means Im slow and not the best time to attempt to digest reading anything technical, didn't even get far with wombats thread.

>>  I have no clue why the attenuation is less with the degraged service

It's not uncommon. Noise can and does affect the attenuation.  I've seen faulty/incorrectly fitted filters can add up to 10dB, there's also plenty of examples where oxidised joints have also affected the attenuation, even on my own line many years ago.  RFI, EMI, electrical currents and damaged cables all have the potential to increase the attenuation.   

Whilst the measurement provided by the modem is used to estimate line length, the reality is the modem calculates the attenuation figure based on the difference between the power transmitted and the power received.  The difference between these 2 figures can be affected by a multitude of things, not just line length.  *linky (https://itel.com/what-is-attenuation/)


@Chrys

Are neighbours also experiencing similar severe speed losses or it just you? 
With the fault having other effects such as increasing the attenuation, does this not mean the problem is pretty local - between the cab & home. iirc youre <300m from the cab?

One of my concerns is ignoring what the other graphs and figures are telling us and focusing solely on the bridge tap. 

*was about to link to my own page written many years ago, but then deliberately chose a different source. 
Got a heavy few days coming up with appts,  so bowing out of this convo now as regretfully dont have any more time to speculate or justify my reasoning.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 30, 2021, 09:24:21 AM
I got no idea on my neighbours sadly, the one I am still friendly with uses virgin media.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: j0hn on June 30, 2021, 01:20:09 PM
It would be a monumental noise source if it was external to OpenReach infrastructure and hit your sync so badly.

Given the Hlog it's almost certainly a defect in the copper pair.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 30, 2021, 02:22:32 PM
In fact if you google the formula, it actually pulls up wombats thread (https://forum.kitz.co.uk/index.php/topic,16536.msg306163.html#msg306163) which contains a link to the JDSU documentation in case you dont, and with the lapse of time his post shows why I may have easily thought that the info originally came from TR-197.   I'm glad you made me search so I can re-read the thread again properly and check if he did make any further posts about the topic.

Thank you for the link back to gazaai (https://forum.kitz.co.uk/index.php?action=profile;u=8593)'s "About my possible bridge tap issue (https://forum.kitz.co.uk/index.php/topic,16536.0.html)" forum topic which was begun in December 2015. I see that a certain grumpy, old, black cat was also a contributor . . . but after five and a half years all details had been completely forgotten.  :-[
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 30, 2021, 03:28:25 PM
I have historical files including a hlog file full of numbers, is that what you after?

The 00-51 file is from start of june so before fault, 00-17 is today.

It is the raw data that produced the Hlog plot, the second image attached to your post (Reply #2) dated June 25, 2021, 09:46:21, that would be the most interesting to see. I'll take copies of both files and have a look at the data. The first of the two (Hlog-00-17.zip) is probably the one of most interest.

Chrys -- Do you have the raw data, from your modem, in tabular form which should show the Hlog values to four decimal places, please? Something like this (https://pastebin.com/RLJ7tri1).  :-\

Both boozy's software and my manual method would benefit by using the more precise data. (One decimal place versus four decimal places.)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on June 30, 2021, 03:55:17 PM
I dont think I can find that in my dslstats files, so I just ran the command pasted it into a txt file for you.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on June 30, 2021, 05:26:34 PM
Safely downloaded. Thank you.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 01, 2021, 09:48:12 AM
I initially wasnt hopeful for getting the reason for the fault as engineer wasnt very talkative but he just rang me with an update and told me the location, so I will see when he comes back if I can get any information on what went wrong.

-----


There was 4 faults he said he fixed, he said two of them the cable was disintegrating when he touched, the other two were joints, all located close to my property he swapped out the cable close to my house.

Looks like he did a DLM reset as well as I am back on fast path.

2 ES in an hour O_o

Since Link time = 59 min 33 sec
FEC:            0               0
CRC:            2               0
ES:             2               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
HostInitRetr:   0
FailedRetr:     0

Title: Re: Crazy fault lost 2/3 sync speed.
Post by: underzone on July 01, 2021, 10:35:17 PM
Your line stats are much nicer now than pre-fault condition. Much higher sync and also attainable speed now  :)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 02, 2021, 09:31:51 AM
Its about 5-6 mbit higher now yeah, the socket now has the Master Socket 5C as well.

When I asked the engineer about external damage, he said there was no fault logged for the cabinet affecting multiple customers, but a duct was collapsed he noticed on repair work.  So that theory was not confirmed and it could have all been coincidental.

It is nice on this new socket there is clear icons next to each socket to indicate which is which.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 02, 2021, 06:05:30 PM
Careful saying nice things about the 5C, that sort of thing is forbidden round here.  :P
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: burakkucat on July 02, 2021, 06:37:43 PM
 :shoot:  NTE5C (& SSFP)  :whip:
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 02, 2021, 10:09:41 PM
some people dont like then? :)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 02, 2021, 11:34:46 PM
A lot of the MK4 faceplates are a loose fit (mine isn't) and some people find their line performs worse (mine also doesn't but I have no phones on that line).

Personally I find the RJ11 connector fits better than the MK3 and I much prefer them being side by side rather than vertical.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 09, 2021, 06:51:03 PM
For the curious, I didnt get any compensation for the missed appointment even though it was mentioned in an email to AAISP.

AAISP are monitoring the line for a period of time as well.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: underzone on July 10, 2021, 01:22:41 PM
Nice clean BQM graph there Chris. Are you using any kind of traffic shaper or limiter in pfsense?
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 10, 2021, 01:28:04 PM
I am using dummynet limiters.

I switched from fq_codel to round robin on my main limiter which is currently only a few mbit below vs my max throughput.

There is also a second limiter fq_codel based for my consoles to restrict their speeds.

I use fq_codel on the upstream I think its set to around 18 and half mbit.

There is some tuning of the variables as well but I dont have that info to hand.

I have actually eased of the limiting, it used to be much more aggressive on the downstream but I realised it was giving me no more benefit than just having the extra bandwidth available, so now the firewall will let streams etc. get very close to line speed.

Steam downloads go through the main pipe just below line speed pipe, but are throttled within steam itself to 5 mega bytes/sec.

The opnsense setup I have been working on actually has traffic been passed through a linux device so I can use the superior cake shaper.

I also have set aaisp to not rate limit (110%), so its done within my network.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 10, 2021, 03:45:31 PM
Its generally advised to rate limit downstream at the ISP and upstream at your end though.  Because rate limiting is of limited value if your pipe is already saturated from the ISP.

I disabled limiters as I found they weren't working correctly on dual-WAN.  I think this was a pfSense bug that crept in recently.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 10, 2021, 07:35:05 PM
That's what I did when I first joined AAISP, I have been tinkering with things over time, and what I posted is just how things are at the moment, it is always subject to change.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: underzone on July 10, 2021, 07:45:31 PM
Thanks Chrysalis, can you post your pfsense: Diagnostics, Limiter Info please.
I may have a go with your settings. I am currently using this:

Code: [Select]
Limiter Information
Limiters:
00001:  15.000 Mbit/s    0 ms burst 0
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:  70.000 Mbit/s    0 ms burst 0
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001:  15.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 3ms interval 5ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 1
00002:  70.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 3ms interval 5ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2

TIA
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 10, 2021, 08:05:30 PM
yep I will post it here, I have a config right now that also moves the flows to the queues part of the flow instead of scheduler, so the output will look a bit odd.

Code: [Select]
Limiters:
00001:  73.999 Mbit/s    0 ms burst 0
q131073 6410 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:  18.779 Mbit/s    0 ms burst 0
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
00003:  30.720 Mbit/s    0 ms burst 0
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001:  73.999 Mbit/s    0 ms burst 0
 sched 1 type RR flags 0x0 0 buckets 1 active
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     6406504 9245283343  0    0 67485
00002:  18.779 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 1601 flows 1024 NoECN
   Children flowsets: 2
00003:  30.720 Mbit/s    0 ms burst 0
q65539  20 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
 sched 3 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 3


Queues:
q00001  20 sl. 5 flows (1024 buckets) sched 1 weight 1 lmax 1500 pri 0 droptail
    mask:  0x00 0xffff0000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
 69 ip        158.69.0.0/0             0.0.0.0/0        1       52  0    0   0
212        ip       0                         2a00:1450:4009::/0                                           ::/0        2      120  0    0   0
254 ip       162.254.0.0/0             0.0.0.0/0        1       40  0    0   0
101 ip       151.101.0.0/0             0.0.0.0/0        1       52  0    0   0
233 ip         3.233.0.0/0             0.0.0.0/0        4      401  0    0   0
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
q00003  50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffff0000/0x0000 -> 0x00000000/0x0000

I should point out I patched pfsense with a custom patch to read manual limiter rules I inject to make things quicker.

I will post them below as well, there should be a way to get a matching configuration in the GUI, bear in mind I am not saying this is some kind of god like config, I only since my line fault was fixed decided to try this approach of making the main pipe only a little bit below the maximum throughput.  It wont be good enough for multi threaded downloads that will have too much spill over, thats why I have the 30mbit pipe for ps5/xbox, and I cap in steam.  I might raise that 30mbit pipe to 40mbit however, as that was set when the main pipe was 67mbit.

Code: [Select]
#71780
#73999, without shaping 77mnit on dumeter, 74.5 tbb, with shaping 74.8 dumeter, 72 single threaded, 71.1 tbb
pipe 1 config  bw 73999Kb queue 6410 codel target 5ms interval 100ms noecn
sched 1 config pipe 1 type rr
queue 1 config pipe 1 queue 20 buckets 1024 mask src-ip6 /48 src-ip 0xffff0000 droptail

pipe 2 config  bw 18779Kb droptail
sched 2 config pipe 2 type fq_codel target 5ms interval 100ms quantum 300 limit 1601 flows 1024 noecn
queue 2 config pipe 2 droptail

pipe 3 config  bw 30720Kb queue 50 droptail
sched 3 config pipe 3 queue 20 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 noecn
queue 3 config pipe 3 mask src-ip6 /56 src-ip 0xffff0000 droptail
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: underzone on July 10, 2021, 08:42:44 PM
Very interesting, thanks!
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on July 11, 2021, 05:02:40 AM
So is that why in the early days they limited what speed you could order so strictly, even though your line technically could easily handle it?

For example initially I was only allowed to order 500Kbit, later they allowed 1Mbit and I even tried for 2Mbit which was refused - but somehow they accidentally processed the order anyway and I did indeed get 2Mbit.

Not really.  That was before rate adaptive dsl (radsl) technology became available.   Back then the modems didn't specifically analyse the channels & bit rates to work out the maximum speed that the line was capable of syncing at. 

The speed was fixed rate at the DSLAM and therefore the modem could either sync.... or it couldn't.  If there was insufficient bit load to maintain 2Mbps then the line would just not be able to connect all at.   So there were basic rules in place, mostly based on the attenuation which dictated whether you could get 256/512/1000/2000 kbps.   Obviously there were a few lines which may have had slightly better SNR and sometimes BT could be persuaded to try you on the higher speed. In those days your SNR was very important in deciding the max speed. 

Maxdsl was the first rate adaptive product offered by BTw.  Maxdsl allowed the line to negotiate a sync speed based upon the current line conditions by analysing each frequency bin based upon the target SNRM of 6dB.  Not all modems were able work properly with maxdsl without a f/w update.  iirc thats when I ditched my Solwise SAR110?

adsl2+ natively uses rate adaptive technology, but it also has better coding algorithms for overheads than adsl1 & adsl2 allowing a max 24Mbps.  In theory, the rate adaptive process should have been not much different from any of the adsl2+ products offered by most of the LLU ISPs at the time.  Even though the superior coding algorithm allowed adsl2+ to be more flexible with the target SNRM, most stuck to 6dB or 9dB (Tiscali).  Many of the BTw Maxdsl DSLAMs were actually MSANs capable of adsl2, but we are talking the days when backhaul bandwidth was still a problem - mostly using ATM 155/622Mbps pipes.

The exceptions were (1) Be* who technically didn't have a true DLM, but allowed a target SNRM of 3dB to be set at the MSAN.... and (2) I think it was UKO who  would allow 3dB profiles and also dabbled with SRA. 

Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on July 11, 2021, 05:20:18 AM

There was 4 faults he said he fixed, he said two of them the cable was disintegrating when he touched, the other two were joints, all located close to my property he swapped out the cable close to my house.


Excellent news, glad you got it sorted.  :) 
Had a feeling it may be joint related or damaged cable due to all that noise.  I'm still curious if water ingress on a damaged joint or cable can cause wave like reflections in hlog.  I've got a photo somewhere of a badly oxidised connector in the BT66 caused weird and fluctuating atten, and 2 cables just crumbled when the engineer poked them with his screwdriver. It was very damp inside the box as it faces towards the sea and gets worst of the weather on that side. It started before fttc though and back in the days when we paid little attention to hlog (thats even if the modem would display them)!
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 11, 2021, 05:22:32 AM
Not really.  That was before rate adaptive dsl (radsl) technology became available.   Back then the modems didn't specifically analyse the channels & bit rates to work out the maximum speed that the line was capable of syncing at. 

The speed was fixed rate at the DSLAM and therefore the modem could either sync.... or it couldn't.  If there was insufficient bit load to maintain 2Mbps then the line would just not be able to connect all at.   So there were basic rules in place, mostly based on the attenuation which dictated whether you could get 256/512/1000/2000 kbps.   Obviously there were a few lines which may have had slightly better SNR and sometimes BT could be persuaded to try you on the higher speed. In those days your SNR was very important in deciding the max speed. 

Maxdsl was the first rate adaptive product offered by BTw.  Maxdsl allowed the line to negotiate a sync speed based upon the current line conditions by analysing each frequency bin based upon the target SNRM of 6dB.  Not all modems were able work properly with maxdsl without a f/w update.  iirc thats when I ditched my Solwise SAR110?

adsl2+ natively uses rate adaptive technology, but it also has better coding algorithms for overheads than adsl1 & adsl2 allowing a max 24Mbps.  In theory, the rate adaptive process should have been not much different from any of the adsl2+ products offered by most of the LLU ISPs at the time.  Even though the superior coding algorithm allowed adsl2+ to be more flexible with the target SNRM, most stuck to 6dB or 9dB (Tiscali).  Many of the BTw Maxdsl DSLAMs were actually MSANs capable of adsl2, but we are talking the days when backhaul bandwidth was still a problem - mostly using ATM 155/622Mbps pipes.

The exceptions were (1) Be* who technically didn't have a true DLM, but allowed a target SNRM of 3dB to be set at the MSAN.... and (2) I think it was UKO who  would allow 3dB profiles and also dabbled with SRA. 
Ah, so original ADSL just didn't have any rate adaptation in the spec at all?  For some reason I never considered that.

I guess partly as I focused so much on SRA on ADSL2+ (mostly that nobody used it) that it hadn't occurred to me that plain RA was also a thing and how it was able to sync at discover the "max line rate" in the first place.

Moving to Be was awesome, I ran 3dB SNRm down, 6dB SNRm up for years - despite a chunk of my line being aluminium.  Was a real drag when I moved back to BTw where that wasn't possible any more.

People (especially in the US) bad mouth DSL but its honestly amazing just how much its evolved over the years.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on July 11, 2021, 05:23:21 AM
For the curious, I didnt get any compensation for the missed appointment even though it was mentioned in an email to AAISP.

IMO you are entitled.   They left you hanging around and delayed the repair by best part of a week.   I'd press again.. as I got compensation a few years back for a no-show and that was before they were less clear about it than they are now.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on July 11, 2021, 06:12:08 AM
Ah, so original ADSL just didn't have any rate adaptation in the spec at all?

Thats right.  Going back to pre 2005ish and pre radsl,  then you could have a line that if it were on maxdsl may sync at 1.8Mbps.   But if you put that line on a fixed rate 2Mbps product then it wouldnt be able to sync.   The modem didn't do channel analysis, so couldn't even fall back to 1Mbps or 512Kbps.   The connection would just be dead.   

There was no real Dynamic LM nor DSM...  I dont think there was even a Target SNRM  - no need, because its the Target SRNM which is used to help calculate the sync speed.    Everything was fixed at the DSLAM whereby it was hoped you would have a 6dB SNRm 'buffer zone' sufficient to keep the line error free.  Didn't even have interleaving... only a FAST channel, so a noise burst would take a line down and it may not be able to sync back up again.  There was a guy in the Usergroup who had iirc 1Mbps and his connection could be down for days.   Eventually BTw downgraded the line to 512kbps.   
I'm sure Max would be in his element at the thought of no DLM....  until.... he realised the other technologies that are closely related to it and/or under its control.   Much as I hate the Openreach DLM and its stuck capping, I fully understand how it benefits the majority of lines and why we need a DLM.   
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on July 11, 2021, 06:18:35 AM
People (especially in the US) bad mouth DSL but its honestly amazing just how much its evolved over the years.

When I get time, I've got a post to  make about a friend of mine in the US and her broadband.    My jaw dropped when I found out how much she pays and what speed she got.  No joke, its like they are 10-15 years behind us.  It's worse than Weaver's connection and it's not as if she's far away from civilisation or anything.   
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 11, 2021, 07:06:06 AM
When I get time I've got a post to  make about a friend of mine in the US and her broadband.    My jaw dropped when I found out how much she pays and what she speed she got.  No joke its like they are 10-15 years behind us.  It's worse than Weaver's connection and not as if she's far away from civilisation or anything.   

When Louis Rossman was looking for new premises in New York, he literally said how the ISP he was forced to use in his current building was so bad he couldn't live stream.  In many respects its the high density cities that are the worst (the exact places its cost effective to go 100% FTTP) because there is no mandate for competition.  One building has the good ISP, another building has the bad one.  It really gives you an appreciation for how we deal with anti-competitive behaviour over here.  But were gonna need to split-off the thread if we keep going off topic like this. ;)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 12, 2021, 02:15:45 AM
Not really.  That was before rate adaptive dsl (radsl) technology became available.   Back then the modems didn't specifically analyse the channels & bit rates to work out the maximum speed that the line was capable of syncing at. 

The speed was fixed rate at the DSLAM and therefore the modem could either sync.... or it couldn't.  If there was insufficient bit load to maintain 2Mbps then the line would just not be able to connect all at.   So there were basic rules in place, mostly based on the attenuation which dictated whether you could get 256/512/1000/2000 kbps.   Obviously there were a few lines which may have had slightly better SNR and sometimes BT could be persuaded to try you on the higher speed. In those days your SNR was very important in deciding the max speed. 

Maxdsl was the first rate adaptive product offered by BTw.  Maxdsl allowed the line to negotiate a sync speed based upon the current line conditions by analysing each frequency bin based upon the target SNRM of 6dB.  Not all modems were able work properly with maxdsl without a f/w update.  iirc thats when I ditched my Solwise SAR110?

adsl2+ natively uses rate adaptive technology, but it also has better coding algorithms for overheads than adsl1 & adsl2 allowing a max 24Mbps.  In theory, the rate adaptive process should have been not much different from any of the adsl2+ products offered by most of the LLU ISPs at the time.  Even though the superior coding algorithm allowed adsl2+ to be more flexible with the target SNRM, most stuck to 6dB or 9dB (Tiscali).  Many of the BTw Maxdsl DSLAMs were actually MSANs capable of adsl2, but we are talking the days when backhaul bandwidth was still a problem - mostly using ATM 155/622Mbps pipes.

The exceptions were (1) Be* who technically didn't have a true DLM, but allowed a target SNRM of 3dB to be set at the MSAN.... and (2) I think it was UKO who  would allow 3dB profiles and also dabbled with SRA. 



I still remember my line not meeting the spec BT had specified to allow a 576kbit order to go through, but I was first on the exchange, engineer was excited as me so he tried it anyway and it worked. :)  The frog modem days.

SRA was such an amazing tech for lines that had inconsistent noise levels, it makes me wonder how much of an easier time Weaver would be having e.g. if he had SRA.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 12, 2021, 10:10:29 AM
I still remember my line not meeting the spec BT had specified to allow a 576kbit order to go through, but I was first on the exchange, engineer was excited as me so he tried it anyway and it worked. :)  The frog modem days.

SRA was such an amazing tech for lines that had inconsistent noise levels, it makes me wonder how much of an easier time Weaver would be having e.g. if he had SRA.

I still have my USB ADSL modem knocking around, can't bring myself to throw it out.  Though by the time my exchange was enabled, consumer routers had already become a thing, I bought it separate for diagnostics.

I'm trying to remember what my first router was now, I think it was a D-Link, long before integrated WiFi was a thing. (I actually had a Belkin 802.11b Access Point when they first came out, it was practically unusable)

For many years though I ran on the Netgear DG834.  In fact, the first thread I posted on this forum (https://forum.kitz.co.uk/index.php/topic,7892.msg165125.html) was about that.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 30, 2021, 02:17:36 AM
Had the biggest storm i remember for a while a couple of nights ago, I checked my data the next morning and was lots of ES, but no DLM action, so either the wide area detection worked, or my new ES is so low now that I was able to absorb a few hours of storm and stay below the thresholds, is good. :)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on July 30, 2021, 04:46:05 AM
Fingers crossed.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on September 16, 2021, 02:02:02 AM
Well a new problem, but may be one I have to put up with depending on what happens whilst the line is interleaved.

My line just went down, and I noticed on the aaisp log the rate limit had changed, line is now interleaved but its not due to a burst of errors, Instead since around 9pmish on the 14th (day before yesterday) there has been a constant barrage of errors on the line, seemingly not service impacting as not noticed any actual problems, but obviously DLM doesnt know the actual service impact so am now interleaved.

Sync speed within product spec though so I probably cannot do anything about it unless I find the noise source.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on September 16, 2021, 07:38:07 PM
I replaced cable, modem, psu, turned off ups, tried test socket (like how the new sockets just clip off), turned off my stb, and I dont think the noise is local.  Also unsurprisingly the GEA line test passes.

It seems to have no affect on QLN, hlog, or sync speed itself, just errors on the line, so I am now having to accept been interleaved for the foreseeable future. The bit of good news is even with the 100s of thousands of FEC and the thousands of CRC I was getting on fast path, the interleaving is mitigating 99% of them.  If I had g.inp I expect this would be a non issue.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on September 19, 2021, 06:39:00 PM
An update, I noticed my LAN cables were reporting errors so surely local, I turned of a few more devices and the FEC is zeroed, so I will probably know exactly which device it is very soon.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: kitz on September 22, 2021, 08:30:16 AM
Interesting, so it could well be local.   Hope you manage to isolate.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Alex Atkin UK on September 22, 2021, 09:13:58 AM
I hope its something easily solved, would be annoying if its something essential and expensive.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on September 22, 2021, 05:43:46 PM
I was misled by the FEC going down to 0, that was because of the gap in the graph, seems the FEC momentarily goes down to 0 when there is a gap instead of just cutting off.

Some of the LAN cables are still getting a steady stream of errors, and its across multiple cables, even brand new replacements.  It is only happening on the shorter cat 5 cables, not the thicker long cat 6 I have.

The bitswaps have shot up across the entire VDSL2 range which seems to suggest it isnt just a specific frequency affected.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on October 11, 2021, 02:00:43 PM
Couple more updates, the first which is an old update but I forgot to post it.  I discovered the errors on the LAN stop if I disconnect the modem from the rest of my network, they also stop if I keep the modem connected but disconnect the modem cable.

The second update its a hopeful one, but might be related, there has been a faulty street light outside my property, the council finally got round to investigating it and they say it has a electrical supply fault, I asked them where the power fed from and the direction they pointed in goes right past my BT pole.  I got no idea when it will be fixed though.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Weaver on October 21, 2021, 11:39:04 AM
It sounds like an evil modem, no? It shouldn’t be letting interference through. Is it injecting rfi into the LAN?

If BT are informed, will that help to put pressure on the council?
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on October 21, 2021, 01:03:05 PM
As far as I know the power is fixed now as the street light is working normally again, I have switched out the modem already on basic diagnostics, the noise is I think external somewhere, but I suspect not far from me.

I have accepted it for what it is now and just await until I move to a FTTP area or get FTTP here, the street light was reported for been a nuisance at night as it kept flashing, and was just hopeful it might have been the source of noise.
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Chrysalis on July 11, 2022, 07:53:02 AM
Well, had an outage in the early hours whilst watching F1 replay, I thought I would have a quick look at dslstats a few minutes before this post (I only realised was an outage as AAISP sent me a sms, the stream kind of just got buffery for about a minute), and noticed my SNRM jumped up and I am now hogging the 74mbit banding I caused when I set that very target SNR on the plusnet hub.

Looking back at my line history the FEC errors had a sudden drop of circa 3000 per 5 minutes to around 1-1000 per 5 minutes at about 6am on 9th July 2 days ago, not particularly low, and I have seen many people on this forum reporting been stuck on interleaving with those sort of numbers, however the outage today was me going back on fast path   If I can be bothered to ask for a reset the line should be capable of an 80meg sync again as the attainable is 83, but FTTP feels like its very close now and the line could swing back just as quick as it did 2 days ago so will probably leave it as is.

(https://team-rebellion.net/Chris/DSLStatsWeb/Graphs/2022-07-09/FEC-2022-07-09-00.05.39.png)

(https://team-rebellion.net/Chris/snrm.png)
Title: Re: Crazy fault lost 2/3 sync speed.
Post by: Weaver on July 11, 2022, 08:28:27 AM
Weird. A new one on me.