Why doesn't LTT have a proper server room?

[GodAtum]

When they built out their offices years ago as seen on YT, I'm surprised Linus didn't spec a proper server room. Even the office I work in (about 100 people, 2 floors), has a dedicated server room with raised floors, air con and fire systems for 2 racks.

[Lurick]

Well they originally built it 5-6 years ago when it was much less dense and I don't think they had the foresight to plan for such density or expansion. It would have made sense but probably also cost a ton of money at the time which they didn't have when first building it out.

[Needfuldoer]

My guess is, it was a convenient spot to put the infrastructure at the time that grew organically to the point where it's out of space but would take a substantial amount of work to move. It was a huge improvement from the en-suite master bathroom at the Langley house, but they've long since outgrown it.

 

When they built the main studios, there was a staircase on the 'outside' of the original two-story office space and the server closet took up the space underneath it. Since that's where the servers and switches wound up, that's where all the network wiring for the building (at the time) landed. Since the network wiring was there, and the rack was there, new servers went there. It's also right next to the edit den, which made high speed networking relatively easy several years ago.

 

Now that they have the Lab 2 space, and it sounds like the logistics department and hardware 'warehouse' is moving there, they've got an opportunity to "do it right" with the space in the main studio unit.

 

What I would do is build a new server room with more accessibility, proper cooling, and better sound isolation. Leave space for at least three full height, full depth server racks and the Eaton UPS. Cooling should be handled by at least redundant mini-splits, each of which can handle the entire heat load of the room (so one can be taken offline for service without having to worry about heat). The existing server closet can then become a switch closet, housing just a couple rack mounted switches and a rack mounted UPS. These will support the existing structured cabling in the building, with a couple high speed backbone connections back to the new server room.

[Guest]

They have one now and I only know because it's in another channel's video when someone visited lmg. 

 

For security reasons they probably didn't make a fuss about it despite having one. 

 

Or at least I'm pretty sure that was lmg

[Neroon]

1 hour ago, Lurick said:

Well they originally built it 5-6 years ago when it was much less dense and I don't think they had the foresight to plan for such density or expansion. It would have made sense but probably also cost a ton of money at the time which they didn't have when first building it out.

That's weird though. When they moved they had a lot of gear already in their bathroom, and since they moved for considerable growth, 1 would have assumed they would have thought about that.

[tkitch]

1 hour ago, Neroon said:

That's weird though. When they moved they had a lot of gear already in their bathroom, and since they moved for considerable growth, 1 would have assumed they would have thought about that.

No, not really.

They mostly still fit into a single rack, including patch panels.  If they were spacious, they'd probably not fill 2 racks.  That's not a lot of gear.  

[Neroon]

33 minutes ago, tkitch said:

No, not really.

They mostly still fit into a single rack, including patch panels.  If they were spacious, they'd probably not fill 2 racks.  That's not a lot of gear.  

You want to be able to work in a room though, allow for growth etc. 

[sub68]

Remember LTT started the house bathroom server probably when they were still not a company the warehouse setup was probably when they started to realize they are a company.

I am pretty sure the server room was speced to have petabyte project and that when they didnt realize.

1. petabyte project might fail

2.they needed more storage then previous years.

5 hours ago, GodAtum said:

air con and fire systems

LTT does have them, air con for awhile, fire was more recently after the UPS fire.

[wanderingfool2]

I would mostly say it's because they were computer people who didn't really have to think about network/server stuff properly for a business...or rather their mentality seems to be, lets worry about it later.

 

I was watching back when it was just Luke and Linus in the garage...back when the forum was on something like a test bench and they tossed an USB that landed inside and broke the server for a while.

 

When they originally were creating the server room, I knew it would be less than optimal given their propensity to add stuff.  Since they were building most of what they had there, it would have been simple to actually have a proper room dedicated to it, away from the filming and easily accessible for AC/future cooling technology.

[Salv8 (sam)]

21 hours ago, Needfuldoer said:

they've got an opportunity to "do it right" with the space in the main studio unit.

i hope that linus will hire someone-one either part or full time to take care of the new and existing infrastructure, it's a mess and needs to have someone take care of it. plus as much as wendel is a nice guy, he can't be expected to come to the rescue whenever a server fails and anthony/jake/alex/whoever is babysitting the servers that day, doesn't know what else to do.

they have gotten to that point and i have hope when linus does another round of hireing he'll advertise a spot for an IT admin.

with the petabyte project having trouble and a brand new building full of new employee's, it's gonna have to happen sooner or later. i just hope sooner for linus's sake.

[wasab]

They should seriously just move their servers to the cloud, using things like Amazon AWS, Microsoft Azure, or google cloud like everyone is doing.

[GuiltySpark_]

28 minutes ago, wasab said:

They should seriously just move their servers to the cloud, using things like Amazon AWS, Microsoft Azure, or google cloud like everyone is doing.

They're not going to get the 20+Gb/s read/write they need for their editors with cloud based servers. 

[Needfuldoer]

7 hours ago, wasab said:

They should seriously just move their servers to the cloud, using things like Amazon AWS, Microsoft Azure, or google cloud like everyone is doing.

That doesn't work when you're primarily dealing with multi-gig video files.

 

I work for a much larger media company than LMG; even at "just" 1080i "the cloud" is for slower-than-real-time transfers between sites or to a long-term tape archive at corporate.

 

Besides that, cloud hosting the entire Vault would cost them an arm and a leg. Probably more than building a proper tape library and hiring a media manager. (And certainly more than "our good buds at SuperMicro and Seagate hooked us up with a 3 PB SAN".) They can always pull their finished video off YouTube or Floatplane if they absolutely have to.

[Neroon]

2 hours ago, Needfuldoer said:

That doesn't work when you're primarily dealing with multi-gig video files.

 

I work for a much larger media company than LMG; even at "just" 1080i "the cloud" is for slower-than-real-time transfers between sites or to a long-term tape archive at corporate.

 

Besides that, cloud hosting the entire Vault would cost them an arm and a leg. Probably more than building a proper tape library and hiring a media manager. (And certainly more than "our good buds at SuperMicro and Seagate hooked us up with a 3 PB SAN".) They can always pull their finished video off YouTube or Floatplane if they absolutely have to.

Linus has said that they do that most of the time anyway. It's easier for them to grab clips from Youtube.

[GuiltySpark_]

3 minutes ago, Neroon said:

Linus has said that they do that most of the time anyway. It's easier for them to grab clips from Youtube.

Indeed, however this doesn't help with with new projects where multiple editors need immediate and local-like access to raw footage. They use this workflow heavily.

[Needfuldoer]

26 minutes ago, Neroon said:

Linus has said that they do that most of the time anyway. It's easier for them to grab clips from Youtube.

If it's faster to find videos online than in the Vault, there's an opportunity to improve their local workflow. A media management system (like Telestream Kumulate  or Grass Valley STRATUS, for example) would provide the editors a better experience with more accessibility than a file dump full of raw footage.

 

But that would be a significant expense, and it wraps back to my argument that they really need a full-time media manager and IT admin at this point.

[dogwitch]

39 minutes ago, Needfuldoer said:

If it's faster to find videos online than in the Vault, there's an opportunity to improve their local workflow. A media management system (like Telestream Kumulate  or Grass Valley STRATUS, for example) would provide the editors a better experience with more accessibility than a file dump full of raw footage.

 

But that would be a significant expense, and it wraps back to my argument that they really need a full-time media manager and IT admin at this point.

i think after the build out of labs and such. They  most likely will look into that. due  to labs itself. you will need redundant data copy and such. before ltt writer etc access it.

 

my experience is .

i have a small database (took me 3 months to create) daily 5 hours sat down. (once had a 3 days worth of work flush down the drain due to a buggy cloud back up folder).

its back up as of writing these in  8 locations . both hard copy(usb and such) and cloud. Around 30% of the info inside of it. Said sites etc    i og found the info . Simple are gone. Zero archive of their data and such.

 

 

( book publishing etc. stuff ) is the data.

[Neroon]

5 hours ago, Needfuldoer said:

If it's faster to find videos online than in the Vault, there's an opportunity to improve their local workflow. A media management system (like Telestream Kumulate  or Grass Valley STRATUS, for example) would provide the editors a better experience with more accessibility than a file dump full of raw footage.

 

But that would be a significant expense, and it wraps back to my argument that they really need a full-time media manager and IT admin at this point.

I know, the whole process right now is far too time intensive, both working with the files and find them, would need to be way quicker for those quick clips to be worth it.

[Tams]

Only they will know, but if we're speculating:

1. They never intended and planned for it to get this big and convoluted.
2. Server stuff is expensive.
3. They get much of their server stuff for 'free' through sponsorship, so they work with what they are given/can get.
   1. Note how they don't really do anything server side unless they get sent something (apparently usually due to asking).
4. The space to expand the server room is more valuable as space for something else.
5. Disruption and inertia.  They'd likely have to redesign the studio/office layout to accommodate a larger server room.*
   1. Delayed and fewer videos.  Perhaps some of lower quality.
   2. It takes a lot of effort and time.
   3. Cost.

*They have quite extensively redesign the studio/office, but that mostly seems to have been office switching.

[Tams]

On 8/19/2022 at 4:46 PM, Salv8 (sam) said:

i hope that linus will hire someone-one either part or full time to take care of the new and existing infrastructure, it's a mess and needs to have someone take care of it. plus as much as wendel is a nice guy, he can't be expected to come to the rescue whenever a server fails and anthony/jake/alex/whoever is babysitting the servers that day, doesn't know what else to do.

they have gotten to that point and i have hope when linus does another round of hireing he'll advertise a spot for an IT admin.

with the petabyte project having trouble and a brand new building full of new employee's, it's gonna have to happen sooner or later. i just hope sooner for linus's sake.

They probably could afford to hire a server admin at this point, but needing to be living in Canada (and in the Vancouver area), and be willing to be in videos as well as a good host (easy to understand) of them is likely a massive hurdle.  Most server admins really do not fit that.

 

No doubt there's some hubris and unwillingness of Linus to let it go as well, as he seems to really like being a server layman.

 

As for Wendel.  Surely he isn't doing it for free?  If he is, well that's on him.

[Yoinkerman]

As someone who looks after a lot of communication rooms, they much much more often than not are a closet under the stairs 

[BondiBlue]

On 8/18/2022 at 8:02 AM, fpo said:

They have one now and I only know because it's in another channel's video when someone visited lmg. 

 

For security reasons they probably didn't make a fuss about it despite having one. 

 

Or at least I'm pretty sure that was lmg

What? They've done well over a dozen dedicated videos about their server room or the hardware that lives inside it. Here's a playlist on the LTT channel that has several of them: 

 

 

That playlist doesn't have them all. There have been videos about the server room going back to before LMG moved into the building more than 7 years ago. 

[Salv8 (sam)]

11 hours ago, Tams said:

and be willing to be in videos as well as a good host (easy to understand) of them is likely a massive hurdle.  Most server admins really do not fit that.

not necessarily, some employee's aren't ever in videos or have been in one video for less then a sec, usually it's the employee's taking care of things such as financials. if the future employee isn't really great on camera, Anthony can take the dude's spot.

and even then, not all IT Admins are introverts, i know one guy who works as one who's introverted and thats it.

11 hours ago, Tams said:

As for Wendel.  Surely he isn't doing it for free?  If he is, well that's on him.

i don't know wendel personally nor do i watch his vids regularly, but he comes off as the kind of guy who'll try and help someone with their problem even if he gets nothing outta the interaction. a lot of IT Admins are like that since they have to do user helpdesk frequently as a part of their job. no matter how large the company is at some point you will do help desk and it helps to be the kind of guy who likes to help.

[wanderingfool2]

11 hours ago, Tams said:

They probably could afford to hire a server admin at this point, but needing to be living in Canada (and in the Vancouver area), and be willing to be in videos as well as a good host (easy to understand) of them is likely a massive hurdle.  Most server admins really do not fit that.

I'd imagine in the worst case it's more of on camera briefly in background (unless the person really wanted to)....as they realistically need someone who does the day to day stuff, like helping out other people around the office (like account department when they need specific software).  Sure they might be involved a bit in a video or two making the recommendations...but more likely the let them do their stuff, and then fix it up/monitor it after the fact.

 

Their last issue they had with the vault was a direct result of not paying enough attention to their equipment (which an IT person would do).

 

It would probably be easy for them to find someone, as there aren't many places where you have access to so much hardware (seriously, some places just getting permission for even a backup solution is like pulling teeth)...I'd imagine it would be an System admin's dream.

 

On 8/19/2022 at 7:02 PM, wasab said:

They should seriously just move their servers to the cloud, using things like Amazon AWS, Microsoft Azure, or google cloud like everyone is doing.

So many places get burned by going to the cloud.  Also, "like everyone is doing" isn't correct.  Content creators storing TB of data are unlikely doing that....using an "everyone is doing" it approach is really shortsighted that can easily get you in trouble when running a company...as it doesn't properly reflect what the needs of each business might be.

 

Like I did a quick calculator.aws quote.  At like 500TB (they have over that amount remember) the monthly cost is $8k (with gigabit speeds) [and I didn't really configure it right]

Google cloud, quickly looking it is $20k/month (PiB amount)

Azure, $1k/month (PiB) but it's 20 cents per GB of read.  So even downloading a 10 gig project file (and lets face it, it's like hundreds per project not tens)...at 10 that's $2 to redownload the file.  If they wanted to download the entire contents of it, it would cost $20k.

 

So in no way is it reasonable to go to cloud for them

 

[Akima]

I don't think i've ever seen a rapidly growing company plan out the perfect comms room that scales with growth. There are too many "unknown unknowns" variables that can throw any sensible amount of planning out of the window.

 

It's kind of already been covered, but if they'd planned for example, 5 videos a week, with an average fileset size, you'd be able to somewhat plan predicable storage needs and scale over whatever period of time set. But throw an unplanned variable, like the opportunity to move from 4K footage to 8K footage, then your origional storage growth schedule is no longer valid.

 

What do you do? Miss the opportunity to "improve" footage quality, or try and make best use of the space your given at the expense of the space being suboptimal.

 

Given the nature of their work and the regular occurrence of unplanned "opportunties" to add new servers/features/functions, I can entirely understand why the server space is the way it is.