My network is bigger than yours - Redundant Networking Upgrade / Server Room Update

[jakkuh_t]

Watch on Floatplane.com! https://www.floatplane.com/post/ESb4R1MR8c

Watch on YouTube:

Check out Dell EMC Networking Equipment! https://lmg.gg/dell-networking
- Buy a Dell EMC S5232F-ON Network Switch: https://geni.us/oWrjcG

 

Check out the Hammond Manufacturing H1 Series of Data Center Rack/Cabinets: https://lmg.gg/hammond-h1

 

Buy RackStuds at lttstore.com!: https://lmg.gg/studs

 

Buy FLEXOPTIX Universal Networking Gear:
- 100G QSFP28 DAC Cable: https://lmg.gg/100g-dac
- Universal Transceivers: https://lmg.gg/flexoptix
- FLEXBOX Transceiver & DAC Programmer: https://lmg.gg/flexbox

 

Buy InfiniteCables Gear:
- Ultra-Thin CAT6a Ethernet Patch Cables: https://lmg.gg/ultrathin-patch-cables
- Bulk CAT Ethernet Cable: https://lmg.gg/bulk-ethernet
- 24 Port Patch Panels: https://lmg.gg/patch-panel
- Single Mode Fiber Cables: https://lmg.gg/singlemode-fiber
- Multi Mode Fiber Cables: https://lmg.gg/multimode-fiber

 

Buy Ubiquiti UniFi Gear:
- Enterprise 48 PoE Network Switch: https://lmg.gg/enterprise48poe
- Pro 48 PoE Network Switch: https://lmg.gg/pro48poe
- Enterprise XG 24 Network Switch: https://lmg.gg/xg24
- CloudKey Enterprise: https://lmg.gg/cloudkey-enterprise
- Protect NVR Video Recorder: https://lmg.gg/nvr
- Protect Security Cameras: https://lmg.gg/unifi-protect
- WiFi 6E Access Points: https://lmg.gg/unifi-wifi

 

Purchases made through some store links may provide some compensation to Linus Media Group.

[bal723]

im jealous.....

[Sanedish]

Ok that is actually Amazing...

Not that bal723 is jealous i didnt mean that

[JogerJ]

I did not expect FlexOptix to make an appearance on this channel. Their major customers are mostly big data centers and contractors, the idea would be they just buy these transceivers in bulk and they're reprogrammable to any different standards/firmwares needed for whatever hardware they're installing on-site. For the uninitiated they also sell pre-programmed transceivers so you don't necessarily need the FlexBox.

I had the chance to meet the people behind FlexOptix at a fair, they also gave me a tour of their compound in Germany. They're really cool people who make really cool products.

[Needfuldoer]

Why (besides tradition) is the bigger server room a switch closet, and the little under-the-stairs switch closet a server room?

[dogwitch]

3 hours ago, Needfuldoer said:

Why (besides tradition) is the bigger server room a switch closet, and the little under-the-stairs switch closet a server room?

cough linus and his height...

[Skipple]

Yeah that's the face of a man who spent all fucking day in a server closet. 

 

[dogwitch]

3 hours ago, Skipple said:

Yeah that's the face of a man who spent all fucking day in a server closet. 

 

i have that face atm...  creating a data base of a collection i have .

both online seeable and offline version. with pics!

oh this is with 650 items.... that have variants of ...

125 down  of the 650...... all has to be down before middle of next week....

that can be updated in real time. while am at event buying and selling the stuff....

 

tbh i  take the in server closet then what i am doing now...... off to working on it ... sad face and noises.

oh and that the smallest collection i have...

the other two collections are 10k and up!

 

[GodAtum]

Finally. Most companies have a rack for prod and one for dev.

[Needfuldoer]

1 hour ago, GodAtum said:

Finally. Most companies have a rack for prod and one for dev.

Every company has dev equipment.

 

Lucky companies also have production equipment.

[Total Epicnessᴭᴺ]

I feel for the poor guy who eventually has to replace one of the middle switches in the middle of the spaghetti.

[silentdragon95]

Kinda wild that LMG does not appear to have redundant internet connections. When Covid hit and my internet connection via Coax-Cable became even more unreliable than it was before, I went and got an additional DSL connection. It's technically a little less than half the speed of the Coax connection (200Mbit vs. 500Mbit), but it's had zero downtime or lagspikes and the speed doesn't randomly decrease in the evening (yay, shared mediums). OPNsense handles load balancing and failover, which works great, as long as you don't need native IPv6. Somehow there seems to be no proper way to implement load balancing of two connections in IPv6 yet. Anyway.

 

It's funny how even LMG seems to be running into the classic "we used to be a small company, now we're large, our infrastructure just sort of grew uncontrolled and now we're starting to hit a bunch of walls" problem. It happens to the best of us.

[ToboRobot]

Getting a second "redundant" internet connection that uses the same backhaul to the local exchange isn't as redundant as people think.  

For real redundancy, it would have to be separate fibre that doesn't share the same route or conduit.  And that is very expensive, and overkill.  Even putting Starlink or cellular backup is unlikely to be worth it.  Driving from the office to home is always a backup option.

[silentdragon95]

1 hour ago, ToboRobot said:

Getting a second "redundant" internet connection that uses the same backhaul to the local exchange isn't as redundant as people think.  

Fair enough. That's why I did it, one is via coax cable, the other one uses the phone line and so they do in fact route to different exchanges. I suppose they are still routed under the same road in front of my house at some point (obviously), so if someone manages to rip up the entire road right where I live I'd still have problems, but for me it is more than redundant enough. If I really needed a plan C, I do have unlimited mobile data, but 4G speeds at home only reach about 25Mbit or so.

 

I also just kind of enjoy overkill solutions I guess.

[ToboRobot]

24 minutes ago, silentdragon95 said:

Fair enough. That's why I did it, one is via coax cable, the other one uses the phone line and so they do in fact route to different exchanges. I suppose they are still routed under the same road in front of my house at some point (obviously), so if someone manages to rip up the entire road right where I live I'd still have problems, but for me it is more than redundant enough. If I really needed a plan C, I do have unlimited mobile data, but 4G speeds at home only reach about 25Mbit or so.

 

I also just kind of enjoy overkill solutions I guess.

Exactly.  Real redundant links, have separate demarcation points and entirely separate links and different upstream providers.  

 

[Needfuldoer]

3 hours ago, ToboRobot said:

Getting a second "redundant" internet connection that uses the same backhaul to the local exchange isn't as redundant as people think. 

Yup, I ran into this at work. We had two fiber circuits that were allegedly "fully diverse paths", but they were with the same carrier.

 

They both went down once when there was a fiber cut hundreds of miles away.

 

They went down again when somebody misconfigured a router somewhere between us and the head office.

 

We also lost our ability to log into anything when these circuits went down, because they didn't spec us out with a local domain controller. ("You won't need one, your Internet connection will never go down, those are fully diverse paths.") The first time, I asked if we were going to get local DCs for "when the circuit goes down again", and the corporate IT people just stared daggers at me. The second time, I asked again, and they were a little less sure of themselves. The third time, we got a local DC and they started talking with a different carrier to migrate our failover circuit onto.

 

Moral of the story? If your "redundant" circuits are on the same carrier, they're not redundant. Also have a local domain controller so your people can at least log into their PCs and use software hosted on-site, even if you're using Microsoft 365 or corporate domain controllers.

[silentdragon95]

1 hour ago, ToboRobot said:

Exactly.  Real redundant links, have separate demarcation points and entirely separate links and different upstream providers. 

I still meet two out of three criteria, I'd say that's pretty good for a home solution that's essentially unnecessary since any sane person would have just taken the hit in speed and switched to the DSL connection entirely

 

On none of the three occasions in the last 2 years where the coax connection went down for an extended amount of time due to somebody accidentally digging through the line somewhere (again, yay, shared mediums) the DSL connection was affected. After all, it's entirely different infrastructure.

 

Can't say that the municipal government was as lucky, someone there clearly must have thought that getting two internet connections with the same provider was a good idea (Narrator: It wasn't.) They ended up running a cable across a street to wire a residential DSL connection into the fiber ring between their various buildings, so users could at least send and receive e-mail. They specifically told users not to use it for anything else though as it would have been overwhelmed easily.

[mynameisjuan]

20 minutes ago, Needfuldoer said:

Moral of the story? If your "redundant" circuits are on the same carrier, they're not redundant.

 

There are multiple level and scopes to resiliency and redundancy, and increase in cost and complexity for each level introduced. Just because the redundancy is not geo-diverse and geo-redundant does not mean it's not redundant.

 

Redundant circuits from the same SP is redundancy, just more so on the basic level, and does provide benefits on it's own. Depending on the SP's topology/design, this may provide decent redundancy excluding the potential of a SPOF in a fiber bundle.

 

At minimum, LTT should have a second circuit even if the only option is with the same provider. At least it will cover instances when their FW craps out or they need to perform work during hours. While costs of downtime are dependent on other factors, their business relies on the revenue which requires connectivity.

[CWP]

5 hours ago, silentdragon95 said:

so if someone manages to rip up the entire road right where I live I'd still have problems

Ahh yes, the North American Fiber-Seeking Backhoe. I know them (and their ancestors that favoured copper) far too well.

[Abhijeet Bhardwaj]

I just signed up to say this:

Dell networking is good until you do not treat it as a Core switch, as soon as you do that it becomes crap. I have 2 X S5232F in VLT running as core and 4 X S5248F running as distribution. All of them run OS10 enterprise and don't mind me saying this but their OS is trash, you are better off using SONIC rather than OS10 because half of the time support would not know how to fix stuff and blame it on your setup. We have had so many issues with STP like RPVST limitation on OS10 would break the STP between core and access layer (all Cisco). Then we finally settled on RSTP for Dells and RPVST for all access but a new issue arose where access would become STP root. Another issue and a major one with VLT, one of switch would go into kernel panic and taking down the whole network regardless of how many redundant links you have, it just doesn't work. Another major issue related to STP is somehow when access becomes root, core starts discarding packets resulting in dropped voice and internet connection, adding to the amazement Dell have no idea how to fix it. Lastly, I saw Jake referring to ACLs I'll leave that to you to find out how ACLs works in Dell OS10, just a fun-fact you are better off of running ACLs in your firewall rather than core which is what I ended up doing. 

 

It has became so bad that the organisation I work for, the IT staff is busy most of the time fixing networking stuff. We are already evaluating Cisco 9600s and 9500s and Arista (doubtful) to replace this piece of junk. Word of advice, if you can then return it and get Cisco, Arista, Aruba or Juniper and if you cannot then stop paying for OS10 and re-image it to Sonic.

[DaITguy]

all things considered, Jake got a whole rack to experiment on at work. that's sick.