Our server room ACTUALLY Caught Fire Explained

[nicklmg]

 

Buy a UPS

On Amazon (PAID LINK): https://geni.us/Z5bB1XS

On Newegg (PAID LINK): https://geni.us/30PYIB

 

Buy a Network Card

On Amazon (PAID LINK): https://geni.us/b3YSW

On Newegg (PAID LINK): https://geni.us/rrVX

[Jtalk4456]

@nicklmg I've been wanting to get a UPS for a while now, but i don't know much about them. While I appreciate the link, could you guys maybe do a video or even just a forum post as a what to buy for UPS's. Like for my example, I am planning for 2 mid range pcs, router, modem, printer, speaker, and landline phone. I really don't know how to figure out what size I need, what features to look for etc.

[GDRRiley]

I'll be honest this screams eaton had issues given they changed how they did it.

[TetraSky]

So wait, I love the magnetic doors. But, wouldn't that technically allow anyone who wanted to... Into the server room without unlocking the actual door? Or is the back completely blocked off from the front by the heavy weight champion that is the UPS with zero possibility around it ?

[asquirrel]

3 minutes ago, Jtalk4456 said:

@nicklmg I've been wanting to get a UPS for a while now, but i don't know much about them. While I appreciate the link, could you guys maybe do a video or even just a forum post as a what to buy for UPS's. Like for my example, I am planning for 2 mid range pcs, router, modem, printer, speaker, and landline phone. I really don't know how to figure out what size I need, what features to look for etc.

My recommendation: Buy a kill-a-watt (https://www.amazon.com/P3-P4400-Electricity-Usage-Monitor/dp/B00009MDBU) (or maybe the newer version?) and plug each device you plan to use into it. If you KNOW that you won't exceed a 10 amp draw then you can just connect the power strip directly into the kill-a-watt. Then, run things as you'd expect them to be during a power failure. IE: if you plan to play a game on one PC but the other is browsing reddit, do that while playing music.

 

If you can't plug everything in all at once (and maybe consider not doing that because the device WILL explode if you exceed 10A draw), just add the various numbers together. IE: PC1 + modem + router = 450W, PC2 + printer + speakers + landline = 150W means you need 150 + 450W == 600W.

 

Then go to the kill-a-watt and see what the "Watts" mode displays. That's how much power draw you have. "Right...but, how big of a UPS do I need?" How long do you want to keep your devices running in the case of a power failure? Do you plan to be home when a failure occurs, or out of the house? If you want to have everything shut down cleanly even when you are out of the house you'll need to spend a bit more and get what is called a 'smart UPS'. The 'Smart' part is basically a USB cable and some software you install that automatically shuts your PC down when the battery level gets too low. Preventing (barring bad setups/scenarios) the battery from running out of power before your devices can cleanly power off.

 

If you plan to be at home during any power outage and only need 5 minutes of power, then you can get a much smaller UPS. Let's start with an example UPS: https://www.apc.com/shop/us/en/products/APC-Back-UPS-650/P-BE650G1

 

It says it has 390W or 650VA. Well, the VA rating has to do with the internals of the unit. The closer the VA and Watt rating are the better, but the watt rating is the one you care about. So that UPS can only output 390W at max, so it's too small for your 600W load. But you could buy two of them and split up the power load between the two of them.

"Cool, but how long will it run for?" APC has a nice calculator on their website, but let's work it out ourselves. The battery is rated for 79 Watt-hours (Battery Volt-Amp-Hour Capacity). You want 600W for 5 minutes, so how many watt-hours is that? 5 (minutes) / 60 (minutes per hour) = 0.0833 (% of an hour you need) * 600W (a 600 Watt-hour battery would run your 600W load for 1 hour, assuming 100% efficiency, but more on efficiency later on) = 50 Watt-hours to run your 600W load for 5 minutes. "But wait, the website says a 390W load would only last 3 minutes?" Efficiency. I'll get to that.

 

"Oh, so that UPS *can* run my load!" No. It cannot. Think about it this way: You have a car with a top speed of 30 MPH but a 100 gallon fuel tank. It can only ever go 30 MPH (deliver 390W) but it can keep doing that for over a longer time than you need because of the 100 gallon gas tank (the 79 Watt-hour rating). Same concept applies to batteries; capacity and maximum delivery rate are two VERY different things!

 

Efficiency: Remember that VA rating? A VA is a volt-amp. A Watt is a volt-amp-second. IE: 1 volt, at 1 amp, for 1 second is 1 watt. So that UPS has a 390 Watt rating, but it built to the 650VA spec. What that actually means is that the unit *draws* 650W from the battery, but can only deliver 390W of that power to your devices due to efficiency losses in the power conversion. 390/650 = 60% conversion efficiency. So that 79 Watt-hour rating is actually: 79 * .6 = 47.4 Watt-hours. To deliver 390W for 3 minutes only needs 19.5 Watt-hours, so I'm not sure where the extra 27.9 Watt-hours go. Maybe I missed something / fail to understand something. Anyone got some insight here?

 

One other feature you MUST have: Sine-wave output. The PSUs in your computer will randomly shut down if you use a cheap/low-grade UPS that uses simulated sine-wave power output. This is because the PSU itself is doing a thing called Power Factor Correction, and unless it gets a natural sine wave on the power input, it will shut down to protect itself. Name brand UPS (APC, CyberPower, etc) will generally come with this feature as standard. Be sure to ask/check though.

[LinusTech]

17 minutes ago, TetraSky said:

So wait, I love the magnetic doors. But, wouldn't that technically allow anyone who wanted to... Into the server room without unlocking the actual door? Or is the back completely blocked off from the front by the heavy weight champion that is the UPS with zero possibility around it ?

It will lock.

[Curious Pineapple]

1 hour ago, GDRRiley said:

I'll be honest this screams eaton had issues given they changed how they did it.

Yupp, if factory fasteners can come loose without "servicing" then there's problems. Sounds like they gave out a free unit to look like a nice helpful company with the stipulation that Linux blames everyone but Eaton just to keep up an image.

[Required]

YEY i got a Eaton a while ago to!

Mine rune perfect! I got a extension Module to and now the Uptime is around 5h when the Power fail.

[GDRRiley]

21 minutes ago, Curious Pineapple said:

Yupp, if factory fasteners can come loose without "servicing" then there's problems. Sounds like they gave out a free unit to look like a nice helpful company with the stipulation that Linux blames everyone but Eaton just to keep up an image.

lets see weights hundreds of pounds and doesn't move.

they didn't give him a free one and he wasn't exactly happy about blaming the installer.

[Kedohawyr]

Something I've never understood from the moment of the original move vlog is why Linus, when presented with a huge warehouse and the ability to design the floorplan almost from scratch, looked at it and said "I want you to build a tiny cramped closet under the stairs so I can put the important servers containing my company's work in there".

 

[Nystemy]

The side access doors are a very nice improvement to ease of reach. The idea of putting locks on them is good as well.

 

I would though also cut out a few fan holes, install some grills, and put in some fans that are collecting dust on a warehouse shelf.

After all, that would make the space a lot more ventilated. And also make an air path from the grill on the server room door, through the servers, and then the UPS and out the back.

Though, LMG seems to maybe be in need of something a bit less hastily thrown together as far as server rooms go.
I myself would make it a bit fancy, for the sake of videos, as in put a large glass wall in front of it, have 2-3 racks + UPS placed next to each other, and then also provide an access corridor on the back. Make the glass wall into a sliding door to quickly provide ample access. And also make it into a warm/cold aisle setup just to please the comment section.

It would potentially be fairly small all things considered. The glass wall could be fairly close to the front of the racks, like 30-50 cm away. It isn't like you are going to run a blade center in it after all. Then we have the racks, lets say they are about a meter deep. (Typical racks can though be anywhere from like 60cm to 120cm deep...) And then a corridor on the back, here I would make it maybe 1.2-1.5 meters deep, to make camera work easier. Giving a total depth of around 2.5-3 meters.

In terms of width, the space would need to have an access door to the back, lets say its a fairly standard 60-80cm wide. Then 3 racks + UPS, each needing about 50cm each. Giving a total width of 2.1-2.3 meters, maybe make it a nice even 2.5 and give some blank space on either side to make the scene a little less cramped, maybe put in some vents.

The space would very easily serve both as a server room for the office, but also as a fancy set background piece. Not to mention that it also gives room to expand things like render/encoding servers, data base servers and accompanying storage arrays, network gear, security camera servers, etc. Not to mention server related projects. Also providing a nicer environment to service and upgrade the servers in.

Sound deadening is though going to be an issue to contend with, though, a large fairly thick piece of glass dampens most sounds rather efficiently, unless there is gaps around the edges...

[Required]

He  want show his incompetence.

[Luscious]

My first UPS purchase was well over 10 years ago. I picked up a small 450VA unit from Staples that I've been using to power my less power hungry stuff - landline, modem/router/switch, printer, powered USB charging hub, small NAS box. It keeps my essentials powered during an outage and keeps me connected to the outside world in case of an emergency.

 

For the actual PC I had a harder time. Most tower UPS units you find in store top out at 1000 watts - no good when I'm rocking a 1600W PSU to run a quad SLI rig. So I was able to hunt down a tower variant of a smart-UPS 220VA made by APC. It's a big/heavy sucker but it can run a 1900W load making it ideal to plug in my PC, monitor, DAC and speakers. A 20 minute run time is more than adequate to shut down gracefully.

 

One thing I do wish was that UPS manufacturers got away from cheap/heavy SLA and transitioned to lithium batteries - I am CONSTANTLY having to replace them because at best they will only handle 150-200 or so cycles. Lithium can easily go 10x that. It's an additional expense and time/labor headache I'd like to go without.

 

For extended power outages I also have a GoalZero Yeti 3000, maybe not for running the PC, but to keep my fridge/freezer on. It's easy to move around and can be solar charged too.

[sub68]

2 hours ago, LinusTech said:

It will lock.

here Linus I found some impact gymbal things

https://www.amazon.com/dp/B0002SRAQ2

then use something like this to put on your drill

https://www.amazon.com/Socket-Adapter-Square-Driver-Silver/dp/B07S4LHZXF

[GDRRiley]

1 hour ago, Kedohawyr said:

Something I've never understood from the moment of the original move vlog is why Linus, when presented with a huge warehouse and the ability to design the floorplan almost from scratch, looked at it and said "I want you to build a tiny cramped closet under the stairs so I can put the important servers containing my company's work in there".

At the time I doubted they expect the server room to get so busy. Had they realized the library would go to waste I'd bet in a redesign some of it would go there.

remember too this was their first custom built space and it has changed a lot over the years

[zENjA]

TD:DR

Yes your Serverroom is really small to have a Rack, Switch Box and a UPS in there.

Cabeling

Think about zero u PDU's with C13/14 or even C19/20 outlets for your servers. Some also can be switched and/or monitored. Also have 2, A from Grid & B from the UPS, as it's not your scale to have two UPS Systems. Get an ATS Switch for single PSU devices, so they could benefit from both sides.

Also you could set the UPS to 220-230V to get better efficiancy out of your server PSU's. If you only use the PDU Outlets, you could not plug a 120v device in by accident, because the plugs are different.

 

Long version:

I plan/build/operate colocation Datacenters at a scale of 90-120MW per site / ~5-15MW per Building. It's like "Unboxing Canada's BIGGEST Supercomputer!" but at scale.

UPS Systems at scale, sometimes have problems, but they should have internal redundancy and not be loaded over 45% and for efficiancy not under 30%. I believe that you have get a double conversion UPS, so you could install an ATS or STS Switch between the powerwallbox and your UPS. This way you could switch the powersource of the UPS from Grid to Backup Generator if needed.

Don't know how often this happens in your area, but here in germany I only had one unplaned outage for 4min at home this year (+ one announced for maintanance).

In the DC we normally have not outages, because of the 110kv / 3 Transformer n+1 setup, but we get all shit that comes over the line, like spikes from lightnings or even get hit by lightnings. In that case it happened that all buildings disconnects themself from the side grid, run on UPS and starts their generators (mostly 16-24 Zylinders with ~35-70l of cubic capacity at 1,8-2,4MW of power per generator). Under full load they want ~ 500l / ~ 3 barrel of super light heating oil (like diesel) each per hour and we have at least enough onside to power them for 72h.

Our UPS Systems get a refresh after 8 years, then the batterys "only" have 80-90% of capacity left. I've get some (18) ot these and have them connected to an aged Dell 5500kVA to get longer runtime. I use Yuasa NPL's and at work we use Yuasa SWL UPS Blocks. Mostly 56-60 in a row, to get high DC Voltages for the UPS, because it's more efficiancy and / and because this way, we get lower amp's and don't have to use "thick" cabels - ok they are still hand thick.

All our UPS Systems are seperated from the battery blocks, they are in different rooms. At scale there is not only eaton, also Vertiv (Emerson Network Power), Schneider Electric (APC).

[Required]

Another problem the home Circuit breaker trip at 3,2KW load.

It rember me at this Video:

 

[Beerzerker]

This reminds me:

I once had to some of the fire protection setup in a server room where I was one of the maintenance techs. Got the job because I was the guy in charge of onsite testing, inspection and some repairs to the building's fire protection/sprinkler system.

Of course there were many things I coudn't do, with me just being the on-site guy.

Anything like that a security contractor took care of or a sprinkler system contractor did the work.

In this instance the "Thing" was to get the room as airtight as possible so the extinguishant once released would remain long enough in sufficient concentration to smother a fire if one broke out, the standard at that time was for the room to be filled with it for 10 minutes before it dissapated to a level that woudn't be effective by the atmospheric percentage of it detected.

It wasn't Halon, it was another type of agent they used (Can't recall the name of it - That was a few years ago) but definitely a newer type than Halon but just as dangerous.

 

I had to remove all the ceiling tiles, paint and seal them, then after setting them back in the ceiling go around the edges of each one with silicone to complete the sealing process for them.

Also had to check and seal any gaps/cracks in the wall, doorways and so on until the entire room was as airtight as possible. Took about a week and a half but got it done and the room passed when tested.

I will say one requirement we had was even if the room could be/was locked there had to be some means in place that a person could get out if that ever happened.

Which there was.

 

Things happen and all you can do is deal with it when it does and hope for the best.

 

EDIT:

I'll add that the buss bar connections (Nuts and bolts/studs) could have (Note I stressed could, not that they actually did) come loose over time due to the thermal expansion and contraction of the metal(s) during use, slowly causing metal fatigue and the proccess of thermal expansion and contraction will induce it, literally making the metal(s) stretch/deform to a small degree.

 

I've personally seen and dealt with it while doing my former job, mostly with things like electrical motors and so on but with things in breaker boxes and the like as well, just not nearly as often.

 

This is an issue that's not unheard of, for example homes that had aluminum wiring in them in the past were bad about this exact thing occuring and that's what it turned out to be, causing house fires due to connections that got loose over time.

This expansion and contraction of the materials "Worked" the connections and they eventually loosened up because of it, no longer having the intial torque they had at the beginning - Loose connections will heat up and you know what's next.

That's why you no longer see new homes using aluminum wiring for this very reason - Aluminum expands and contracts thermally by a good amount.

 

Even though this isn't aluminum it's still subject to the same basic effects because the connections gets warm, cools some, then warms up again while in use, depending on the loads it has to carry during it's life which will vary of course. If a connection isn't "Quite" up to torque spec's initially it can do this afterwards at any time.

It's always a good idea to check such when installed initially and at least annually to ensure they are and remain at the required torque spec's to avoid it. 

[maksakal]

I'm surprised your insurance let you get away with this. You're lucky the fire didn't get further then the inside of the unit, if the fire would've spread there's no way your insurance would've paid as this is clearly a risk you yourself created by improperly installing high-power equipment.

There is a much bigger lesson to be learned here, about electrical and fire safety.

As for the question who's fault it was, it was neither Brian's nor Eaton's. Eaton clearly says:
 

Quote

4. Torque the screws holding all input and output power conductors to the values specified in Table 7 on

page 30.
5. Wire the UPS for the proper input voltage as shown in Figure 28.

 

Also the manual specifies that you're supposed to use bus-bar extenders to wire your mains.

So the installer is expected to open up the unit and wire the internals according to your regional power and how you're expecting to use it (1 phase, 2 phase or 3 phase). They also clearly say you're supposed to torque all the bus-bars down according to specs. The screws on the bus-bars probably weren't there because Eaton expects you to wire them up with bus-bar extenders or to wire them up for additional battery cabinets. You can hook up all kids of stuff to those bars. You can use a bypass power module, an external bypass. Depending on your input voltage and configuration there's several ways you can hook those bas-bars. If you're not using any of those then you're supposed to torque the bars down yourself.

So it's Brian's fault? No Brain isn't a product engineer nor is he familiar with the product. He's an electrician who's been asked to wire up a device to the mains. To be honest no electrician is going to open up a device and mess with the internals, it's just a liability waiting to happen. BUT! Brain you know damn well you're supposed to consult the manual before turning a device on for the first time.

Neither of the two were at fault here, it is LTT's responsibility to make sure devices are commissioned, maintained, inspected and used properly. This means inspections and certifications. Making sure a product engineer or somebody who is familiar with the product is present or is installing the product.

Here in The Netherlands (were we probably have the strictest codes and requirements) you're required to annually inspect everything and re-certify everything every 5 years. It's probably a good idea to have the LTT legal department read the insurance policy because something tells me you will be required to-do annual inspections for the insurer to pay up.

As an electrical engineer I'm always surprised to find how lightly people tend to think about electricity. Just connect these 2 wires and it works! Right? Yes it works now, but will it stay working for the next 10 to 20 years. Once it's wired up nobody will take the time to inspect it as long as it stay's working, only when the device fails will you look at it.

Improper wire-gauge and improper connections are the 2 culprits when it comes to catastrophic failure. At first the device will work, not showing the fact that there is an improper connection and that it's arcing. In a few years it will start to show unexplained failures, suddenly stop working and then start working again without any reason and then one day, poof! Catastrophic failure.

I think this is a good wake-up call for LTT, you guys got off easy! Heck you even had a video and a discount from Eaton as a reward. Nexttime it won't be this easy, maybe it's time to start taking things like workplace safety and fire safety more seriously. You can start by doing an inspection of all your buildings, I'm sure your insurance company and local fire department will be glad to help you with that.

 

[Jtalk4456]

6 hours ago, TetraSky said:

So wait, I love the magnetic doors. But, wouldn't that technically allow anyone who wanted to... Into the server room without unlocking the actual door? Or is the back completely blocked off from the front by the heavy weight champion that is the UPS with zero possibility around it ?

I'd imagine it blocks the way in, but even if it didn't, I'd be more concerned how someone got far enough into their building undetected to get to the server room. If someone is that determined, I'd guess they can get past the door to the server room

 

6 hours ago, asquirrel said:

~Too long to leave the whole quote~

Thanks for the info. I'll look into getting a killawatt, I've heard of them before. How do I account for equipment I do not currently own but want to plan for in the future? Would it be easier if not a bit more expensive to plan based on the max rated draw for a device than using a kill a watt, giving myself some wiggle room? Or is that making too much wiggle room?

[ahuckphin]

Somehow my most memorable moment from this video for me is at 1:50. Using a server room as storage? Cold spares, maybe but toilet paper? 

[TempestCatto]

23 minutes ago, bindydad123 said:

but toilet paper? 

Because of corona virus. Cold spares are to fight against murder hornets.

[asquirrel]

22 minutes ago, Jtalk4456 said:

I'd imagine it blocks the way in, but even if it didn't, I'd be more concerned how someone got far enough into their building undetected to get to the server room. If someone is that determined, I'd guess they can get past the door to the server room

 

Thanks for the info. I'll look into getting a killawatt, I've heard of them before. How do I account for equipment I do not currently own but want to plan for in the future? Would it be easier if not a bit more expensive to plan based on the max rated draw for a device than using a kill a watt, giving myself some wiggle room? Or is that making too much wiggle room?

Depends on the device. A 1000W microwave will, more or less, always draw 1000W. A PC with a 650W power supply could draw anywhere from 50W to 750W depending on what hardware is in there, whether it's sleeping or powered on, a game is running, etc. And no, that isn't a typo. PSUs are rated by the power they can *deliver* not the power they can take. An 80+ certified PSU (if it barely qualified) would only be 80% efficient *at best* (layman explanation of ratings here: https://www.velocitymicro.com/blog/what-is-psu-efficiency-and-why-is-it-important/ ) So that 650W PSU would actually be drawing closer to 815W from the wall (650/.8) if you managed to fully load the PSU. The Kill-a-watt would show you the 815W number, which is what makes it so useful (in general, not just for UPS sizing). Contrast that with an 80+ Gold PSU, which would be 650/.87 = 750W. Basically an entire CPU worth of power saved (well, a gaming CPU).

 

But yea, I'd really need to know what, specifically, you were planning to buy. Some items will have fairly static power draws and others will not. I will save you the time though: if you tell me another computer, the answer is "I can't know that". Depends on what is in the computer and what the computer is doing. My PC, right now, typing this message to you, us using about 135W. But if I start watching a movie it will use around 170W. And if I play Factorio it will use 315W.

[anylettuce]

4 hours ago, maksakal said:

I'm surprised your insurance let you get away with this. You're lucky the fire didn't get further then the inside of the unit, if the fire would've spread there's no way your insurance would've paid as this is clearly a risk you yourself created by improperly installing high-power equipment.

There is a much bigger lesson to be learned here, about electrical and fire safety.

As for the question who's fault it was, it was neither Brian's nor Eaton's. Eaton clearly says:
 

 

Also the manual specifies that you're supposed to use bus-bar extenders to wire your mains.

So the installer is expected to open up the unit and wire the internals according to your regional power and how you're expecting to use it (1 phase, 2 phase or 3 phase). They also clearly say you're supposed to torque all the bus-bars down according to specs. The screws on the bus-bars probably weren't there because Eaton expects you to wire them up with bus-bar extenders or to wire them up for additional battery cabinets. You can hook up all kids of stuff to those bars. You can use a bypass power module, an external bypass. Depending on your input voltage and configuration there's several ways you can hook those bas-bars. If you're not using any of those then you're supposed to torque the bars down yourself.

So it's Brian's fault? No Brain isn't a product engineer nor is he familiar with the product. He's an electrician who's been asked to wire up a device to the mains. To be honest no electrician is going to open up a device and mess with the internals, it's just a liability waiting to happen. BUT! Brain you know damn well you're supposed to consult the manual before turning a device on for the first time.

Neither of the two were at fault here, it is LTT's responsibility to make sure devices are commissioned, maintained, inspected and used properly. This means inspections and certifications. Making sure a product engineer or somebody who is familiar with the product is present or is installing the product.

Here in The Netherlands (were we probably have the strictest codes and requirements) you're required to annually inspect everything and re-certify everything every 5 years. It's probably a good idea to have the LTT legal department read the insurance policy because something tells me you will be required to-do annual inspections for the insurer to pay up.

As an electrical engineer I'm always surprised to find how lightly people tend to think about electricity. Just connect these 2 wires and it works! Right? Yes it works now, but will it stay working for the next 10 to 20 years. Once it's wired up nobody will take the time to inspect it as long as it stay's working, only when the device fails will you look at it.

Improper wire-gauge and improper connections are the 2 culprits when it comes to catastrophic failure. At first the device will work, not showing the fact that there is an improper connection and that it's arcing. In a few years it will start to show unexplained failures, suddenly stop working and then start working again without any reason and then one day, poof! Catastrophic failure.

I think this is a good wake-up call for LTT, you guys got off easy! Heck you even had a video and a discount from Eaton as a reward. Nexttime it won't be this easy, maybe it's time to start taking things like workplace safety and fire safety more seriously. You can start by doing an inspection of all your buildings, I'm sure your insurance company and local fire department will be glad to help you with that.

 

got the model # of the one that had isses? Looking at a manual now. It has an in and out. The out looks like it goes to possibly a breaker panel so that panel us also on a ups. Not the internal connections. So far I haven't seen anything about opening it up and tightening the bus bar connections.

so far I would say 75% eaton issue, 25% linus. Not enough ventilation and " service of the unit"

[maksakal]

1 hour ago, anylettuce said:

got the model # of the one that had isses? Looking at a manual now. It has an in and out. The out looks like it goes to possibly a breaker panel so that panel us also on a ups. Not the internal connections. So far I haven't seen anything about opening it up and tightening the bus bar connections.

so far I would say 75% eaton issue, 25% linus. Not enough ventilation and " service of the unit"

Look again, the manual clearly says you're supposed to wire up a load ditribution panel and optionally can wireup an external bypass module. Depending on your line voltage and number of phases you're expected to wire the device accordingly. Both chapters 3 and 4 of the manual cover this in great detail, not sure how you could've missed it.