Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

Lightning has wrecked equipment in some of the houses in the village. Three of my lines are down.

Line 2: WLR3Test WLR3_CIDT_Test dys00556app04:467688470: Fail FAULT - Battery Contact ServiceLevel:2.5, MainFaultLocation:LN, FaultReportAdvised:Y, AppointmentRequired:N, LineStability:, NetworkStability:, StabilityStatement:

Line 3: WLR3Test WLR3_CIDT_Test cbs00556dat03:182580312: Fail FAULT - Dis One Leg In Network ServiceLevel:2.5, MainFaultLocation:CE, FaultReportAdvised:Y, AppointmentRequired:N, LineStability:, NetworkStability:, StabilityStatement:

Line 4: WLR3Test WLR3_CIDT_Test rds00556app02:467066834: Fail FAULT - Dis one leg in network ServiceLevel:2.5, MainFaultLocation:LN, FaultReportAdvised:Y, AppointmentRequired:N, LineStability:, NetworkStability:, StabilityStatement:

[Bowdon]

Wow that doesn't sound good.

Hope they get on to fixing them asap.

[Weaver]

AA has sent someone out and an engineer contacted Janet. An army of OR vans has descended on the NSBFD exchange. No working phone lines in the village apart from my one, apparently. Most lines out in Broadford and even in Kyle of Lochalsh too across the bridge on the mainland. All shops are cash only because they can’t take credit card payments. The mobile phone network is up. I was using 3G last night while my modems were unplugged for safety - Firebrick just went into artificially forced failover.

—
Line 2 is now up again.

[burakkucat]

Hmm . . . That reads like a major incident if the Broadford TE itself is affected.    (I wonder if Black Sheep would like a trip north of the border?)

Those three reports are quite clear as to the problem but I am unable to interpret codes that specify the two separate locations, "MainFaultLocation:LN" and "MainFaultLocation:CE".

What is the status of your equipment, the four VMG1312-B10As and the small switch, the VLAN end-point?

[Weaver]

All my hardware is fine, unplugged asap, even though one strike was heard before we had time to react.

—

Line 3 is up as of 17.05.

That leaves only line 4 down currently. The jobs were marked in the notes as part of a batch of three, thank goodness, so sensibly given to one engineer.

The inhabitants of Heasta were ‘whatsapping’ one another talking about the outage earlier. It’s by far the biggest incident I can remember. On the lightningmaps live strikes map it showed four strikes at least in and around Heasta, including one hit at Na Torran next door and one near Druim Fheàrna - or else in the bay - was seen by Janet - blinded her temporarily. Janet went to the mainland chemist’s phone line was out. I’m surprised that there has been so much damage yet I have had one line up. If the exchanges were taken out then I would expect a different story of course.

[burakkucat]

All my hardware is fine, unplugged asap, even though one strike was heard before we had time to react.

That's good to know.

. . . four strikes at least in and around Heasta, including one hit at Na Torran next door and one near Druim Fheàrna

Google Maps was quite helpful. I just entered the Scots Gaelic, exactly as you had typed, and it showed me the locations. I can now understand why you typed "next door"!

[Weaver]

One hit was given as 1.1 miles away. It’s absolute chaos in the whole area. There were a couple of strikes to the north of here and in the Cuilfhionn mountains, in the Small Isles and as far south as the northern part of Eilean Mhuile (anglicised as Mull).

[Black Sheep]

Hmm . . . That reads like a major incident if the Broadford TE itself is affected.    (I wonder if Black Sheep would like a trip north of the border?)

Those three reports are quite clear as to the problem but I am unable to interpret codes that specify the two separate locations, "MainFaultLocation:LN" and "MainFaultLocation:CE".

What is the status of your equipment, the four VMG1312-B10As and the small switch, the VLAN end-point?

Ha ha .... not likely, Mr Cat .... those days are long gone. 

FI - LN = Local Network or UG (Underground Network, more specifically) ..... CE = Customer Engineer (In other words a multi-skilled engineer as the remote test results can't fully determine if the fault is within the EU's curtilage, or in the OR network nearby the EU's curtilage. 

[burakkucat]

Thank you.    I had a suspicion that the "LN" referenced the local network but couldn't think of a meaning for the "CE".

[Weaver]

Actually, I’m thinking the NTE5 or SSFP might have blown up on line 4. I haven’t got a spare mk 4 handy to swap out.

[burakkucat]

If the NTE5C is determined to be zapped, then the SSFP should also be regarded as dead and the complete set replaced as a whole.

[Weaver]

> If the NTE5C is determined to be zapped, then the SSFP should also be regarded as dead and the complete set replaced as a whole.

Thanks for that. Good point.

I’m hoping the engineer will visit the house then. Perhaps if he or she doesn’t see any signal then they might want to confirm things are actually working.

 

Hang on - I’ve just realised, I told AA the wrong thing - line 2 is the latest line, not line 4; they’re not in chronological order. I feel sure the prior state of affairs was 1, 3, 4. So line 2 is the one with the SSFP, not line 4; I will double check with my beloved. But anyway, so much for that theory. Old threads in kitz should confirm this thinking from history.

So anyway, I was being mad. Line 4 is just down because it hasn’t been fixed yet and there is no SSFP surely.

[Weaver]

I am being even more mad. Line 4 is actually up. An SMS was sent to Janet’s phone to tell her that line 4 but she never told me about it; also I subscribe to the the AA notification-by-Tweet system but it has not been working for a long time.

So that makes all lines now up.

Re line 2:
Yesterday 19:02:02      Track PSTN Fault 5-7-186085713512 Informational Message: 4150 Response Required - Fault Report Cleared   bt
Yesterday 19:02:02      Track PSTN Fault 5-7-186085713512 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 19:02:02      Track PSTN Fault 5-7-186085713512 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 19:02:02      Track PSTN Fault 5-7-186085713512 Status: Open - Clear Unconfirmed   bt
Yesterday 19:01:54      Track PSTN Fault 5-7-186085713512 Notes Field: 08/10/2019 19:01:00

===Point Of Intervention notes===

Between this point...
- Location: Joint on right 150m upheast rd
- Work Point: JB26
And this point...

Plant details...
- Plant affected: JNT100
- Plant type: MC32

Multiple Intervention?: N
===Point Of Intervention notes ends===

(No manually entered closure notes)

=== QBC Summary Start ===Customer Report: End customer advised of no dial tone / voice on the line.\n\n Actions to resolve: Engineer has resolved the fault located at the D-side including aerial cables / lead-in / block terminal.The fault was located outside the end customer's curtilage and shown by cut / damage to wire / cable. The fault was fixed by clearing in joint.\n\n Additional information: Engineer has not visited end customer premises.\n\n Final alternate test results: Final FastTest performed from the customer premise.The test passed on 08/10/2019 18:56:47.=== QBC Summary End ===   bt
Yesterday 19:01:54      Track PSTN Fault 5-7-186085713512 Informational Message: 4465 Please refer to the Notes field for the actual message   bt
Yesterday 19:01:54      Track PSTN Fault 5-7-186085713512 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 19:01:54      Track PSTN Fault 5-7-186085713512 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 19:01:54      Track PSTN Fault 5-7-186085713512 Status: Open - Implementing Solution

Re line 3:
Yesterday 18:41:18      Track PSTN Fault 5-7-186085764565 Informational Message: 4150 Response Required - Fault Report Cleared   bt
Yesterday 18:41:18      Track PSTN Fault 5-7-186085764565 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 18:41:18      Track PSTN Fault 5-7-186085764565 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 18:41:18      Track PSTN Fault 5-7-186085764565 Status: Open - Clear Unconfirmed   bt
Yesterday 18:40:24      Track PSTN Fault 5-7-186085764565 Notes Field: 08/10/2019 18:27:00

===Point Of Intervention notes===

Between this point...
- Location: joint outside 3 school Croft
- Work Point: JB26
And this point...

Plant details...
- Plant affected: JNT100
- Plant type: MC32

Multiple Intervention?: N
===Point Of Intervention notes ends===

(No manually entered closure notes)

=== QBC Summary Start ===Customer Report: End customer advised of no dial tone / voice on the line.\n\n Actions to resolve: Engineer has resolved the fault located at the D-side including aerial cables / lead-in / block terminal.The fault was located outside the end customer's curtilage and shown by soot / blackened wire / cable. The fault was fixed by clearing in joint.\n\n Additional information: Engineer has not visited end customer premises.\n\n Final alternate test results: Final FastTest performed from the customer premise.The test passed on 08/10/2019 18:21:35.=== QBC Summary End ===   bt
Yesterday 18:40:24      Track PSTN Fault 5-7-186085764565 Informational Message: 4465 Please refer to the Notes field for the actual message   bt
Yesterday 18:40:24      Track PSTN Fault 5-7-186085764565 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 18:40:24      Track PSTN Fault 5-7-186085764565 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 18:40:24      Track PSTN Fault 5-7-186085764565 Status: Open - Implementing Solution

Re line 4:

Track PSTN Fault 5-7-186085779396 Notes Field: 08/10/2019 19:09:00 -

===Point Of Intervention notes===

Between this point...
- Location: Jnt 150m on right up heast rd
- Work Point: JB26
And this point...

Plant details...
- Plant affected: JNT100
- Plant type: MC32

Multiple Intervention?: N
===Point Of Intervention notes ends===

(No manually entered closure notes)

=== QBC Summary Start ===Customer Report: End customer advised of no dial tone / voice on the line.\n\n Actions to resolve: Engineer has resolved the fault located at the D-side including aerial cables / lead-in / block terminal.The fault was located outside the end customer's curtilage and shown by cut / damage to wire / cable. The fault was fixed by clearing in joint.\n\n Additional information: Engineer has not visited end customer premises.\n\n Final alternate test results: Final FastTest performed from the customer premise.The test passed on 08/10/2019 19:06:04.=== QBC Summary End ===   bt
Yesterday 19:10:37      Track PSTN Fault 5-7-186085779396 Informational Message: 4465 Please refer to the Notes field for the actual message   bt
Yesterday 19:10:37      Track PSTN Fault 5-7-186085779396 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 19:10:37      Track PSTN Fault 5-7-186085779396 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 19:10:37      Track PSTN Fault 5-7-186085779396 Status: Open - Clear Unconfirmed   bt
Yesterday 19:10:33      Track PSTN Fault 5-7-186085779396 Informational Message: 4150 Response Required - Fault Report Cleared   bt
Yesterday 19:10:33      Track PSTN Fault 5-7-186085779396 Estimated Response Time: 2019-10-09T23:59:59   bt
Yesterday 19:10:33      Track PSTN Fault 5-7-186085779396 Clear: 82.2 In Joint AreaCable (Underground)   bt
Yesterday 19:10:33      Track PSTN Fault 5-7-186085779396 Status: Open - Clear Unconfirmed


Now that I’ve found out about OpenReach Thank an Engineer I submitted the engineer’s name in that form.

[Weaver]

Our engineer was working away sorting out my lines until around 19:10 (presumably BST?) when a lot of people had gone home hours earlier. A long day and probably backlog of chaos continuing tomorrow for all I know.

A brilliant job by BTOR, and by AA who booked an engineer at 08:02 [!]; AA support leapt on it straight away, the moment they were open.

[Weaver]

Lines dropped in the early hours of this morning. I don’t know if this could be due to BTOR working overnight ? Is that possible.

The line 4 sync rate which was bad last week went back to normal when the line was repaired; but last week its downstream had been down to 80% of what it had been before. Now following the blips it’s down to a bit below its normal downstream, as are all the three repaired lines now. Each of lines 2,3,4 should be around 2.8 - 2.9 Mbps downstream at least:

Live sync rates:
  #1: down 2854 kbps, up 666 kbps
  #2: down 2496 kbps, up 547 kbps
  #3: down 2652 kbps, up 579 kbps
  #4: down 2563 kbps, up 505 kbps

The upstream rates are miraculous though : by a mile higher than anything ever seen before at 666k (best prev 560k). And line 3 upstream is now cured, its upstream speed massively increased - from say 350k to 579k - which brings it level with the others. And that too is presumably going to bring an enormous improvement, so you would hope; more nearly level speeds without one causing problems for TCP because it is so much slower that the round trip times for that pipe are weird. The sum of the IP PDU rates upstream @ 96.5% modem loading factor and 0.884434 protocol efficiency is 1.96Mbps which is significantly higher than the highest ever seen before, around 1.6-1.7Mbps (depending on how sickly line 3 was). So if you correct that figure for TCP and IP header overheads that should give an idea for a maximum possible TCP combined throughput in an ideal world, and something line 97% of

However speedtest2.aa.net.uk does not give good results at @ 1.28 Mbps upstream.

Firebrick current upstream rate limiters' IP PDU tx rates (egress speeds), in-force right now ::
  #1: 568416 bps
  #2: 466852 bps
  #3: 494164 bps
  #4: 431006 bps
Total combined rate: 1.960438 Mbps

Fractional speed contributions:
  #1: 28.994%  [█████████████ ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒]
  #2: 23.814%  [███████████ ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒]
  #3: 25.207%  [███████████ ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒]
  #4: 21.985%  [██████████ ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒]

--
Obtained from live-querying the Firebrick.

So the speedtest result is pretty awful at only 68.6% of the expected throughput figure for IPv6+TCP+timestamps or 67% for IPv4+TCP no timestamps. I just don’t know why this is so at one time it was fine, then it wasn’t and now we don’t have the speed inequalities to blame. Earlier I was absolutely certain that data corruption was to blame, and I suppose that is possible, maybe the sync rates are incredibly high because some kit was turned off and that made an atypically quiet noise environment which - speculating further - was not sustainable because after the sync rate upstream had been established the noise sources came back leaving the links struggling to cope. So will need to look at stats.

	TCP			TCP @ loading	TCP bps @ 100%
TCP TS+IPv6 efficiency:	95.200000%	of IP PDU
TCP+IPv6 efficiency:	96.000000%	of IP PDU
TCP TS+IPv4 efficiency:	96.533333%	of IP PDU
TCP+IPv4 efficiency:	97.333333%	of IP PDU
TCP TS+IPv6 rate:	84.198113%	of sync rate		1866337 bps	1934031 bps	; @1.27Mbps = 0.68584
TCP+IPv6 rate:	84.905660%	of sync rate		1882020 bps	1950283 bps	; @1.27 Mbps = 0.68012
TCP TS+IPv4 rate:	85.377358%	of sync rate		1892476 bps	1961118 bps	; @1.27 Mbps = 0.67636
TCP+IPv4 rate:	86.084906%	of sync rate		1908160 bps	1977370 bps	; @1.27Mbps = 0.67080