Factory Networking Problem

[Huntsman]

Hi, I have plenty of experience setting up network for small businesses (50 ish desktops and devices) as a freelancer back in college. I just got employed in a R&D company and am tasked with solving the company's networking problem. Don't ask why a software engineer is tasked to do this kinda work, tldr is me and my big mouth and me being a new hire.

 

So my company has a few labs which contains 40~ ish test equipment. Each of which periodically grabs a small file from a NAS (a cheap looking dlink one which overheats now and then) and runs test. And the previous contractor thinks that it is a good idea to daisy chain switches all over the place and use cheap tiny switches to get the job done. So you could imagine finding a 5 port or 4 port dlink plastic switch scattered all over the place daisy chained together. Performance is of course not great. Fetching files from the crappy NAS takes forever and often times out. That causes alot of delays and frustration among the senior engineers.

 

I can solve the NAS problem pretty easily by building something like a FreeNAS box with redundancy and caches.

 

Networking side is where I'm at a loss. My original idea was to have one cable per machine to a central location where I can put all the switches but that is not feasible as the labs are quite far apart and really big. But at the same time, I know that I can't just put switches all over the place like the previous contractor so now I'm stumped. Is there some way to connect all the switches together to form a big switch? How does those big companies do this? I don't think any manufacturer sell like 200 ports switch, at least not to my knowledge.

 

I have absolutely no experience in this kinda large scale network but I do build small networks for SOHO so I'm hoping the LTT networking gurus can give me some advise on how to go about this. I'm willing to learn and explore new things.

 

Thanks in advance!

[brwainer]

If the locations are too spread out to wire everything back to a single location, then you should identify multiple areas where you can serve as many hosts possible reasonably. So maybe you have 4 or 5 24-port switches, each serving the equipment nearby. These "Individual Distribution Frames" (IDF) each have one or two uplink cables back to the "Main Distribution Frame" (MDF) where the NAS and router are located. 

 

For all switches, you want to be running STP or RSTP, and they should also have a loop detection feature. Loops on the same switch or between switches are the biggest cause of network performance issues, and only Smart or Managed switches can protect you against them. Note that with these protections though, they only ensure that there is no loops, they don't always choose the right port to disable when given a choice of ports that are in conflict.

[Huntsman]

15 hours ago, JaredM54 said:

Yeah, sometimes people will cut corners and then you end up with switches and hubs everywhere, which as you can see can cause tons of problems. Normally you will have your "core" switches, which then spread out to different areas, for example floors in a building. As for needing more ports than a single 48 port switch, you can "stack" them together, basically becoming one big switch. Or, just use uplinks, most of the time 10Gb. You have the right idea with putting the switches in a central place. So I would consider where it makes sense to have switch closets (like near labs and rooms of offices), then bring those back to a single switch. Take a look at Ubiquiti's switches, either Edgeswitches or Unifi. (Depending on your budged of course look at other brands, but that's where I'd start.)

 

14 hours ago, brwainer said:

If the locations are too spread out to wire everything back to a single location, then you should identify multiple areas where you can serve as many hosts possible reasonably. So maybe you have 4 or 5 24-port switches, each serving the equipment nearby. These "Individual Distribution Frames" (IDF) each have one or two uplink cables back to the "Main Distribution Frame" (MDF) where the NAS and router are located. 

 

For all switches, you want to be running STP or RSTP, and they should also have a loop detection feature. Loops on the same switch or between switches are the biggest cause of network performance issues, and only Smart or Managed switches can protect you against them. Note that with these protections though, they only ensure that there is no loops, they don't always choose the right port to disable when given a choice of ports that are in conflict.

 

First of all, thanks for the response. I had a somewhat similar idea but didn't know what sort of hardware would I need.

 

Currently there are 5 labs and each of them contains around 40 machines. All of them would periodically (but simultaneously) grab a small text file from a NAS. My idea is to put a 48 port switch in each lab and run one cat6 back to a "main" switch. The NAS would be connected directly to the "main" switch. But I am kinda doubting myself as to what's the difference between the my idea and the current predicament. This is technically also daisy chaining switches right? Or is there a proper way to do this? Maybe some specialized switches that has some high bandwidth interconnect between a "main" and "daughter" switches?

 

The network won't have a steady over the time usage but very high peak of burst usage as all the machines fetch the same text file at the same time. After that one burst of activity, the network will just stay idle.

[leadeater]

3 hours ago, Huntsman said:

Currently there are 5 labs and each of them contains around 40 machines. All of them would periodically (but simultaneously) grab a small text file from a NAS. My idea is to put a 48 port switch in each lab and run one cat6 back to a "main" switch. The NAS would be connected directly to the "main" switch. But I am kinda doubting myself as to what's the difference between the my idea and the current predicament. This is technically also daisy chaining switches right? Or is there a proper way to do this? Maybe some specialized switches that has some high bandwidth interconnect between a "main" and "daughter" switches?

 

The network won't have a steady over the time usage but very high peak of burst usage as all the machines fetch the same text file at the same time. After that one burst of activity, the network will just stay idle.

Connecting switches in a hub and spoke type configurations is very common. Where the difference comes in from just slapping switches willy nilly and chaining to the point of death versus doing it properly is just purely in planning and using the correct equipment.

 

The fewer the hops the better of course but no hops is generally impossible for anything except the most small and basic network.

 

It's generally considered safe to place switch locations up to 80m away (cable run length) from devices. This allows for 20m of patch cords to go from the wall to the device and from the patch panel to the switch, 100m is maximum distance for the Ethernet standard.

 

Your current plan for a 48 port switch in each lab will work well and won't cost too much to do. If however some of the labs are close enough together for a more central switch location 2 or more 48 port switches that support stacking would be nicer. Also for each switch location I would take 2 cable runs back to the core switch using link aggregation, more for redundancy than performance as it sounds this isn't a big requirement.

 

I would probably contribute most of the problems to an under spec'd NAS but the network layout certainly won't be helping and depending on the devices being hubs or switches making it worse.

[Lurick]

I would also suggest looking into the future and seeing if the lab's will ever have more computers and also how much uplink bandwidth will be needed out of each lab to other locations. If you think that more than a single 1Gb uplink will be enough I would suggest switching to fiber and getting a switch with a few 10Gb uplink ports instead and also having the ability to stack the switches is a good idea for future-proofing the network.

 

Depending on how much the company is willing to spend and how much knowledge you have with networking in general there are a myriad of options. Obviously one of the top choices would be Cisco switches such as the 2960-X flavor for mostly layer 2 functionality and then the 3560-X or 3750-X or even newer ones like the 3650 or 3850 series switches for Layer 2 and Layer 3 functionality. Another option if you aren't very familiar with the CLI would be something like Meraki switches which are crazy easy to maintain and operate but still offer all the features you would need and then some. Of course stepping down a bit there are options such as the Netgear ProSafe line of switches, which should offer similar functionality as well or even Ubiquiti switches although I'm not as familiar with these I know the Netgear does have a 10Gb option I imagine the Ubiquiti line would have some as well.

[Huntsman]

5 hours ago, leadeater said:

Connecting switches in a hub and spoke type configurations is very common. Where the difference comes in from just slapping switches willy nilly and chaining to the point of death versus doing it properly is just purely in planning and using the correct equipment.

 

The fewer the hops the better of course but no hops is generally impossible for anything except the most small and basic network.

 

It's generally considered safe to place switch locations up to 80m away (cable run length) from devices. This allows for 20m of patch cords to go from the wall to the device and from the patch panel to the switch, 100m is maximum distance for the Ethernet standard.

 

Your current plan for a 48 port switch in each lab will work well and won't cost too much to do. If however some of the labs are close enough together for a more central switch location 2 or more 48 port switches that support stacking would be nicer. Also for each switch location I would take 2 cable runs back to the core switch using link aggregation, more for redundancy than performance as it sounds this isn't a big requirement.

 

I would probably contribute most of the problems to an under spec'd NAS but the network layout certainly won't be helping and depending on the devices being hubs or switches making it worse.

This might be a stupid question but regarding the cable run length, how is the run length calculated? Is it from NAS to main switch then to daughter switch then to machine? Or just daughter switch to machine? If it's the former then I'm afraid I would be operating out of spec then.

 

Also, yea I would try upgrading the NAS first and see if the situation improves. The dlink el-cheapo NAS is currently placed in the boss's room. Network wise it's connected to a switch which is branched from a "main" switch at the center of the building. The current "main" switch is another white plastic el-cheapo 4 port dlink. Somebody has gotta knock some sense into these people lol

 

2 hours ago, Lurick said:

I would also suggest looking into the future and seeing if the lab's will ever have more computers and also how much uplink bandwidth will be needed out of each lab to other locations. If you think that more than a single 1Gb uplink will be enough I would suggest switching to fiber and getting a switch with a few 10Gb uplink ports instead and also having the ability to stack the switches is a good idea for future-proofing the network.

 

Depending on how much the company is willing to spend and how much knowledge you have with networking in general there are a myriad of options. Obviously one of the top choices would be Cisco switches such as the 2960-X flavor for mostly layer 2 functionality and then the 3560-X or 3750-X or even newer ones like the 3650 or 3850 series switches for Layer 2 and Layer 3 functionality. Another option if you aren't very familiar with the CLI would be something like Meraki switches which are crazy easy to maintain and operate but still offer all the features you would need and then some. Of course stepping down a bit there are options such as the Netgear ProSafe line of switches, which should offer similar functionality as well or even Ubiquiti switches although I'm not as familiar with these I know the Netgear does have a 10Gb option I imagine the Ubiquiti line would have some as well.

According to my boss the place is never going to get expanded. The only thing that changes are the test machines but the amount of machines per room will never be likely to increase. They're all big thermal chambers for stress testing lol, not easy to lug around.

 

And since all of them are accessing the same file simultaneously, I think it wouldn't be a bandwidth problem? Would it be some sort of congestion problem having to serve one machine at a time? Correct me if I'm wrong tho..

 

All machines at all labs would pull a single text file from the NAS simultaneously, then do their testing, after which they will all push results to the NAS. Rinse and repeat.

[Lurick]

4 minutes ago, Huntsman said:

This might be a stupid question but regarding the cable run length, how is the run length calculated? Is it from NAS to main switch then to daughter switch then to machine? Or just daughter switch to machine? If it's the former then I'm afraid I would be operating out of spec then.

 

Also, yea I would try upgrading the NAS first and see if the situation improves. The dlink el-cheapo NAS is currently placed in the boss's room. Network wise it's connected to a switch which is branched from a "main" switch at the center of the building. The current "main" switch is another white plastic el-cheapo 4 port dlink. Somebody has gotta knock some sense into these people lol

 

According to my boss the place is never going to get expanded. The only thing that changes are the test machines but the amount of machines per room will never be likely to increase. They're all big thermal chambers for stress testing lol, not easy to lug around.

 

And since all of them are accessing the same file simultaneously, I think it wouldn't be a bandwidth problem? Would it be some sort of congestion problem having to serve one machine at a time? Correct me if I'm wrong tho..

 

All machines at all labs would pull a single text file from the NAS simultaneously, then do their testing, after which they will all push results to the NAS. Rinse and repeat.

 

The spec is rated on a device A to device B length, the signal gets regenerated at every hop along the way. If they aren't going to expand the labs them you should be good with a single 48 port switch with a few uplink ports. As for bandwidth constraints I don't see any for pulling down the information if it's a small text file but I would suggest adding multiple links to the server so it can load balance properly and even having two uplinks ports to the main switch will allow things to be hashed and uploaded/downloaded faster. Where I do see constraints is when the devices upload their results, are we talking a small amount of information or 10s of gigabytes of information after each test per machine?

[leadeater]

8 minutes ago, Huntsman said:

This might be a stupid question but regarding the cable run length, how is the run length calculated? Is it from NAS to main switch then to daughter switch then to machine? Or just daughter switch to machine? If it's the former then I'm afraid I would be operating out of spec then.

It's calculated from network port to network port, meaning switch to switch or switch to PC. The 100m is the maximum length the signal can be transmitted before it degrades too much, once received it can then be re-transmitted to travel a further 100m and so on.

[leadeater]

13 minutes ago, Huntsman said:

And since all of them are accessing the same file simultaneously, I think it wouldn't be a bandwidth problem? Would it be some sort of congestion problem having to serve one machine at a time? Correct me if I'm wrong tho..

 

All machines at all labs would pull a single text file from the NAS simultaneously, then do their testing, after which they will all push results to the NAS. Rinse and repeat.

At a guess I would think it's the disk performance of the NAS that isn't up to scratch to service that many devices, CPU performance could also be an issue.

 

As @Lurick asked, how big is the result data as that could be a significant amount of network traffic if 200~ devices are all sending that data to the NAS at the same time. Even a small amount of bandwidth, say 10Mbps per device, would add up to a lot once multiplied by 200. 200 x 10 is more than a single 1Gbps link can handle. However most cheap basic NAS's do not perform well enough to fully saturate a 1Gbps link so without addressing both the issue won't really be fully resolved.

 

I would typically do some monitoring to pin down the exact cause of an issue but the current switches you have will have no ability to do this, you might have better luck with the NAS though. Most allow you to look at CPU load and network throughput on their management web interface.

[Huntsman]

14 minutes ago, leadeater said:

It's calculated from network port to network port, meaning switch to switch or switch to PC. The 100m is the maximum length the signal can be transmitted before it degrades too much, once received it can then be re-transmitted to travel a further 100m and so on.

If that's the case, I should be fine. Does this rule of thumb apply to all network cable type? I'm planning to go cat6.

16 minutes ago, Lurick said:

 

The spec is rated on a device A to device B length, the signal gets regenerated at every hop along the way. If they aren't going to expand the labs them you should be good with a single 48 port switch with a few uplink ports. As for bandwidth constraints I don't see any for pulling down the information if it's a small text file but I would suggest adding multiple links to the server so it can load balance properly and even having two uplinks ports to the main switch will allow things to be hashed and uploaded/downloaded faster. Where I do see constraints is when the devices upload their results, are we talking a small amount of information or 10s of gigabytes of information after each test per machine?

 

3 minutes ago, leadeater said:

At a guess I would think it's the disk performance of the NAS that isn't up to scratch to service that many devices, CPU performance could also be an issue.

 

As @Lurick asked, how big is the result data as that could be a significant amount of network traffic if 200~ devices are all sending that data to the NAS at the same time. Even a small amount of bandwidth, say 10Mbps per device, would add up to a lot once multiplied by 200. 200 x 10 is more than a single 1Gbps link can handle. However most cheap basic NAS's do not perform well enough to fully saturate a 1Gbps link so without addressing both the issue won't really be fully resolved.

 

I would typically do some monitoring to pin down the exact cause of an issue but the current switches you have will have no ability to do this, you might have better luck with the NAS though. Most allow you to look at CPU load and network throughput on their management web interface.

The download is 1kB text file, the upload is several MB excel file. I do plan to build a FreeNAS box to serve as central NAS and I will look into the link aggregation or some load balancing stuff for this. Is there anything in particular I should look for?

[leadeater]

2 minutes ago, Huntsman said:

If that's the case, I should be fine. Does this rule of thumb apply to all network cable type? I'm planning to go cat6.

 

The download is 1kB text file, the upload is several MB excel file. I do plan to build a FreeNAS box to serve as central NAS and I will look into the link aggregation or some load balancing stuff for this. Is there anything in particular I should look for?

Yes, same maximum length whether it's cat5/cat5e/cat6/cat6a etc. If you need longer than 100m in a single hop that's when you move to fibre.

 

Link aggregation should do the trick, linux/FreeNAS has other nic teaming options that don't require switch support that will also work i.e. balance-alb.

https://en.wikipedia.org/wiki/Link_aggregation#Driver_modes

[Huntsman]

Sorry for not replying for so long, was busy with work. So after some digging in my spare time, I've found these switches.

http://www.tp-link.com/en/products/details/TL-SL1226.html#specifications

http://www.cisco.com/c/en/us/products/collateral/switches/small-business-smart-switches/data_sheet_c78-610061.html (SF300-24 (SRW224G4-K9-UK)

https://www.hpe.com/h20195/v2/gethtml.aspx?docname=c05051651

 

However, the TP-Link one cost 1/3 of the Cisco. What sort of benefit would the Cisco bring to the table in my setup? Because all the test machines are only 100mbps, they can all connect to the normal 100mbps ports. I plan to use the gigabit ports as uplink, to join the switches to the main gigabit switch.

 

Also, since the max speed I'm doing is the trunk cable from daughter switches to main switch, which is gigabit, would cat5e be enough?