Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Chrysalis]

So yesterday I discovered a odd problem, every 30 seconds I was getting packet loss.  At first I thought it was just the internet, but was my entire LAN.

In addition it coincided with a spike of traffic I seen on my main desktop, it was only a spike to around 700kB/sec but a spike noticeable over idle traffic.

I spent ages trying to get to the bottom of it, a few key things.

When I disconnected my second openwrt switch from my LAN, the packet loss stopped, although spikes were still going to my PC.
I also discovered disconnecting my firewall from the main switch, also stopped the packet loss but in addition also stopped the spikes.
I looked at my firewall (pfsense) and the culprit was upnp, something known to be a security risk, but it was enabled in the past as it was the only way I could get UNO to work online multiplayer.  As soon as I disabled upnp, the spikes stopped, and everything is fine again with all equipment connected.

So the conundrum I have been left with is how such a small spike of internet traffic can kill a gigabit LAN, can internet traffic cause a broadcast storm somehow?  I didnt inspect the traffic after I discovered it was upnp, so I have no information on what the payload was.  I briefly enabled it again today and the spikes have stopped, but have disabled it again given what happened.

Welcome any thoughts.

[Weaver]

How did you see the traffic spikes ? That is, what tool did you use to see them.

It occurs to me that 700kB/s could easily be 1Gbps if a total amount of burst traffic is averaged out over a suitable time quantum; 1Gbps is transmitted for a short period of time and then ceased and then two different kinds of measurements are made at different time resolution accuracies.

[Reformed]

You've a switching loop somewhere that caused a storm. UPnP among other things uses multicast and if your switch is either not smart or not configured properly you can have issues.

Assuming your switches are smart enable RSTP or STP on them, and check on your equipment to make sure nothing connected to both switches is bridging them.

[Chrysalis]

How did you see the traffic spikes ? That is, what tool did you use to see them.

It occurs to me that 700kB/s could easily be 1Gbps if a total amount of burst traffic is averaged out over a suitable time quantum; 1Gbps is transmitted for a short period of time and then ceased and then two different kinds of measurements are made at different time resolution accuracies.

I have dumeter on my desktop, also confirmed in task manager. (although task manager lags).

[Chrysalis]

You've a switching loop somewhere that caused a storm. UPnP among other things uses multicast and if your switch is either not smart or not configured properly you can have issues.

Assuming your switches are smart enable RSTP or STP on them, and check on your equipment to make sure nothing connected to both switches is bridging them.

Thats my thoughts also switch related.  Both switches are openwrt and support STP, but I read STP has a performance hit, should I ignore that and just turn it on?

There is nothing connected to both switches directly.  However my modem (in bridge mode) is connected to my firewall directly (which is connected to first switch) and also to the second switch, the second switch is a LAN connection for collecting stats (since I moved to pppoe the stat collecting via wan cable has broke), and the cable to my firewall is the WAN bridge.

[sdawson35]

Spanning Tree wont cause that much of performance hit (especially if your network is small).

My home network is quite complex with multiple vlans, switches , router , firewall , multiple operating system drvices , iOT etc and all running with spanning tree enabled no issues.

I did however have an oddity quite a while back that saw big spikes ( I use netxms to monitor my network ) intermittently and randomly , turned out to be Window Updates , one of the settings was set to update from other windows devices (or some such) and every so often it poll my network for other windows devices and then try to do an update . Turned that setting off and no more issues .

Just for context I have a Cisco xdsl router , Cisco switches, TP Link Wifi access points, Palo Alto firewall , 8 iOT devices (security) and 4 servers 

[Alex Atkin UK]

I did however have an oddity quite a while back that saw big spikes ( I use netxms to monitor my network ) intermittently and randomly , turned out to be Window Updates , one of the settings was set to update from other windows devices (or some such) and every so often it poll my network for other windows devices and then try to do an update . Turned that setting off and no more issues .

That's funny, as I always keep that option on and it NEVER ONCE updated from other machines on the network, though I have no idea if it actually looks or not.

[Chrysalis]

Ok thank you, I will enable it on both switches and report back.

[Reformed]

STP will consume notable CPU once as it builds a topology of the network. After that the resource consumption is minimal and periodic. A single path to each destination in the network is selected and other ports to the same destination are not used removing loops on the switches.

[Chrysalis]

All seems good, copying to truenas 984mbit, copying from 977mbit.

The second switch only needed about two minutes for the CPU load to go back to idle, main switch about five minutes, I think because on that one was two bridges.

[Alex Atkin UK]

And from the logs you can figure out which ports are bridging and fix it.

[Reformed]

Only one device is attached to both switches to complete the loop. That device is a bridge that has to be connected to both as far as I have read so STP should keep it sane.

[Chrysalis]

What has confused me a little, in openwrt the STP toggle is only present on interfaces that have a bridge configured, so e.g. I enabled it for both LAN and guest on my main switch/AP, but on the second switch I could only enable it for LAN as the guest is just the VLAN not also bridged to a AP.

I am hoping this is globally enabling STP rather than just on the internal openwrt bridges, ultimately if none of these internal bridges were configured there would be no means of turning STP on.

[Alex Atkin UK]

Seeing as STP is a way to prevent network loops over switches/bridges, why would you need it on anything that isn't a switch/bridge?

[Chrysalis]

Seeing as STP is a way to prevent network loops over switches/bridges, why would you need it on anything that isn't a switch/bridge?

My first LAN broadcast storm had a bridge not on the switch.

The bit that confused me is that you can still create a loop without these internal bridges. But potentially with no means of enabling STP.

I could only enable STP as the ethernet switch is bridged with wifi interfaces on 3 of the 4 interfaces. (on second switch there is no guest AP as I dont use wifi on that switch, so hence not been able to turn on STP on its guest LAN interface).

I have probably misunderstood the full way STP works, so I will just leave it turned on and be happy with that.