Multicast Storm

[Grenstauf]

I've been reading some on here but haven't seen exactly this so I'm throwing it out there.

Over this past weekend I got a call that nothing worked on the network. Not likely but ok. It took me a while to get back there and sure enough everything was locked up with switches in weird light patterns. No logons, no dhcp, no internet! No nothing! Local consoles on various servers were fine but no network access for anything. Started pulling plugs to try and isolate a source and with some reconfiguration I got one switch configured with core services but no clients. I started adding things back until I was able to see where it was coming from and had a look with MS Network Monitor.

Whenever a particular link from another switch was added in things would get ugly. Network monitor showed me multicast packets that just keep growing until they swamped everything. Usually only takes a few minutes to build up. Network Monitor reports MAC addresses for source and destination but neither of the MACs reported exist on this network and they are from an unknown manufacturer.

At the far end of this cable is an extension to the main network that has not had any recent changes in a couple of years. There is a little router that isolates a student lab and a bunch of office systems. The lab is Linux mint and the offices are Win7. This whole event started at 10 o'clock at night and it's pretty much a sure thing no one was in the building on a Friday night. After multiple power cycles on everything possible...it just went away.

The first thing I thought of was a really nasty virus but after many checks and updates nothing of that sort. There's only about 20 systems with 4 printers and everything has been checked many times since with not even a hint of a problem.

I now have Wireshark standing by and ready to capture whenever it should start up again but this just really creeps me out and I'm glad it was on a weekend. A few dozen users looking over my shoulder is no fun.

If anybody has seen anything like this I'd sure be happy to hear what you found.

[Oberon.Smite]

Post this on Tom's Hardware, where they actually have half a brain for this.

[Grenstauf]

actually im posting it on every tech support website i can think of just to get a variety of different responses.

[KTFO|SGTmoody]

I've been reading some on here but haven't seen exactly this so I'm throwing it out there.

Over this past weekend I got a call that nothing worked on the network. Not likely but ok. It took me a while to get back there and sure enough everything was locked up with switches in weird light patterns. No logons, no dhcp, no internet! No nothing! Local consoles on various servers were fine but no network access for anything. Started pulling plugs to try and isolate a source and with some reconfiguration I got one switch configured with core services but no clients. I started adding things back until I was able to see where it was coming from and had a look with MS Network Monitor.

Whenever a particular link from another switch was added in things would get ugly. Network monitor showed me multicast packets that just keep growing until they swamped everything. Usually only takes a few minutes to build up. Network Monitor reports MAC addresses for source and destination but neither of the MACs reported exist on this network and they are from an unknown manufacturer.

At the far end of this cable is an extension to the main network that has not had any recent changes in a couple of years. There is a little router that isolates a student lab and a bunch of office systems. The lab is Linux mint and the offices are Win7. This whole event started at 10 o'clock at night and it's pretty much a sure thing no one was in the building on a Friday night. After multiple power cycles on everything possible...it just went away.

The first thing I thought of was a really nasty virus but after many checks and updates nothing of that sort. There's only about 20 systems with 4 printers and everything has been checked many times since with not even a hint of a problem.

I now have Wireshark standing by and ready to capture whenever it should start up again but this just really creeps me out and I'm glad it was on a weekend. A few dozen users looking over my shoulder is no fun.

If anybody has seen anything like this I'd sure be happy to hear what you found.

 

Hi,  Sounds like somthing is flodding your network with MAC addresses. Thus all the devices work overtime to try and learn all these new MAC's and then their MAC tables get full, At which point they start sending trafic to all ports.

 

First thing that enterted my head was,  Has someone created a loop somewhere ?  This could be a likely cause,  If the network isnt correclty seperated with VLANs  and STP running between routers and switches you can easly create a loop.

 

The other thing could be that there is a large network somethere thats connected to your core services which is overloading it with MAC and broadcasts.

 

Have you a network map that you can post ? , Including Vlans and if STP is enabled between core and edge devices.

 

Aslo,  Do you keep a syslog off all devices in the network?  This could help you solve the problem by looking back at all network events? 

 

If your using CISCO you can look up the even log of each device by doing a :  sh log | inc beg (put month IE NOV) (put date, IE 25)  EG:  sh log | inc NOV 25

[KTFO|SGTmoody]

Whenever a particular link from another switch was added in things would get ugly. Network monitor showed me multicast packets that just keep growing until they swamped everything. Usually only takes a few minutes to build up.

 

Loop, Find where the cable goes, Look at the events on both switches,  Idealy if your free to plug and play to solve the issue,  Lable and then unplug all the cables from this switch.  Then, one by one, keep adding more with about 30 sec between each one untill the switch ceates the problem,  then trace cable back to the other end. Now you have the offending devices you can work on preventing it,  Idealy enable  Spaning Tree on all switches to prevent this in the future and the use of VLANS to seperate your core sevices from general computers and important networks, with the use of routers to connect networks togeather.

 

Hope this helps a little,

[Footy]

you often get things like this when managed switches and connected with unmanaged ones, or a typically fun one is ip phones.

generally you will want to enable a few things on the edge of your managed network;

BPDU Guard (often easy to integrate with portfast)

storm-control (use it to shut down ports if broadcast traffic exceeds your limits)

 

Another reason I see this happen is because of excessive switch chaining. spanning tree can only calculate loops on default settings over a network with a diameter of 7, so splitting up your L2 network with routers can help or even just connecting your switches in a few simple star configurations and linking the core of each star can help.

[Darren]

Post this on Tom's Hardware, where they actually have half a brain for this.

 

Lol, speak for yourself. So rude. Looks like OP got answers.

[Oberon.Smite]

Lol, speak for yourself. So rude. Looks like OP got answers.

You don't know this community. And how was it rude? The ratio of IT pros/network specialists there is way larger than here, no denying that.

[Neil]

Oh boy oh boy I love a possible STP problem.

Unplug everything. Then turn it off. And replug. haha

Do you have any dell computers on your network? They're known to uhh... multicast a lot of awesome stuff.

Do you have any VMWare virtual servers running anywhere on the network? What do the MACs look like? Could be spoofed, or generated. You can usually break them down and find out the hardware/vm that generated them if that's the case.