Petabyte Project Installation Vlog

[manikyath]

so.. just yesterday i touched more worth of server than linus installed in this video..

 

working in IT really ruins the wow factor of stuff like this

[nicklmg]

3 hours ago, Mandrewoid said:

IDK what their offsite backup is, but If I were Linus, it would be with AWS, and so then it's basically unlimited. Relatively cost effective. 

AWS is not cost-effective... We've looked into them for a few different ideas and always found that their pricing is obscene compared to alternatives. They're one of the most expensive options available as far as I know...

 

Don't get me wrong - they offer a ton of different services and their reliability is second to none from what I've heard, but for our purposes it really seems overkill.

 

We have a private offsite backup in a rack that's about 3 hrs away from the office by car.

[risk]

5 hours ago, unijab said:

explain how it is cost effective...

Not claiming it's cost effective, it's probably hard for Linus to estimate that as well given how maintaining all the things is a revenue stream for him and not strictly opex.

Rule of thumb for datacenter hardware is that for every dollar of hardware you put in, you end up paying a dollar per year to run it and maintain it (overtime it becomes cheaper to replace existing capacity with new more efficient capacity hardware, than it is to keep running existing hardware, roughly after 2-3 years depending on Moore's law in a particular area). 

 

Ignoring the transfer costs, storing a petabyte in the cloud costs about $7.5k - $10k per month with Google Coldline/Nearline .. not sure about other providers.

 

If you estimate 1:1 capex:opex and ignore the network transfer costs;  at 50k USD / petabyte of storage capex .. his cloud storage bill for a year, would probably cost about the same. (IRL network costs money too, but then again you don't use all your storage at the same time and you only pay for what you use, but then you don't need a magical sysadmin to mount a cloud service, or at the very least cloud is less hassle than having to replace and rebuild raids; then again if you need access to the footage quickly you probably want the data on your LAN not on the internet; then again you could also conceivably render videos in the cloud).

 

IMHO Cloud has the potential to be cheaper, easier, faster and more reliable than any use case, ... but it's not there yet and home networking is not there yet for that to be true for 99.999% use cases, maybe in 5 years.

 

(disclaimer: I work on cloud)

[Ezio Auditore]

14 hours ago, Daniel644 said:

on a serious note, could we get rid of the orange overlay during outros, it honestly feels cheaper then just seeing the host with thing overlayed.

What I guess is that since Youtube started adding video overlay at the ending of the video the suggestion would appear on the screen which would hide the host so they resorted to orange outro

[Zodiark1593]

I probably missed it, but the server uses ECC RAM, correct? From what I've read, ECC is strongly recommended for use with the ZFS file system. 

[Tieox]

Linus mentions the water-cooled server needing to be at the bottom? but correct me if I am wrong he then places the petabyte stuff at the bottom? so where did the WC server go @LinusTech

[arbellason55]

15 minutes ago, Kierax said:

Linus mentions the water-cooled server needing to be at the bottom? but correct me if I am wrong he then places the petabyte stuff at the bottom? so where did the WC server go @LinusTech

At the end it is under Petabyte stuff if you look closely.

[hogfather]

That's quite a lot TB-age lost due to Windows and Unraid/ZFS.  Is that justifiable? Can the loss be, well, lessened somehow?

 

BTW, AWS at these amounts of data is not viable at all. 

[2FA]

4 hours ago, Zodiark1593 said:

I probably missed it, but the server uses ECC RAM, correct? From what I've read, ECC is strongly recommended for use with the ZFS file system. 

The Storinator servers all come with ECC RAM, that is of course assuming Linus didn't put his own hardware in the chassis.

[unijab]

14 hours ago, risk said:

...

Ignoring the transfer costs, storing a petabyte in the cloud costs about $7.5k - $10k per month with Google Coldline/Nearline .. not sure about other providers.

 

If you estimate 1:1 capex:opex and ignore the network transfer costs;  at 50k USD / petabyte of storage capex .. his cloud storage bill for a year, would probably cost about the same

....

 

 $7.5k - $10k per month is 90k - 120k per year. 

So you're saying it would only be about 50k per year starting out since the full petabyte wont be used in the cloud?

[unijab]

6 hours ago, hogfather said:

That's quite a lot TB-age lost due to Windows and Unraid/ZFS.  Is that justifiable? Can the loss be, well, lessened somehow?

 

BTW, AWS at these amounts of data is not viable at all. 

 

On 45 drives website, they say they normally put 15 drives in RaidZ2. that is almost 87% usable space. More space get used for redundancy when the gluster is setup... but not sure off the top of my head.

[aymen]

OMG its cost a lot but have fun of it guys  

[Reece Boddington]

On 04/04/2017 at 8:43 PM, nicklmg said:

Buy Seagate ECAP 10TB drives on Amazon: http://geni.us/QQ8BG6d

 

Learn more about the 45Drives Storinator: http://geni.us/6x4qLo

 

 

We got a PETABYTE worth of drives from Seagate and some badass enclosures from 45Drives... let's get 'em installed.

 

It isn't 1PB its 778TB...

[manikyath]

On 5-4-2017 at 8:16 AM, nicklmg said:

AWS is not cost-effective... We've looked into them for a few different ideas and always found that their pricing is obscene compared to alternatives. They're one of the most expensive options available as far as I know...

 

Don't get me wrong - they offer a ton of different services and their reliability is second to none from what I've heard, but for our purposes it really seems overkill.

 

We have a private offsite backup in a rack that's about 3 hrs away from the office by car.

AWS is a good option if you need all the data to be "hot" all the time, which video creators like yourself usually dont. offsite backup in an "actually well fitted out" datacenter is probably the best to go, especially since most of them need several failures in a row to even go dark at all, and actual data loss due to infrastructure failure is essentially zero.

 

as for long term archiving, tapes are still amazing

[nicklmg]

1 hour ago, ABXY Gaming said:

It isn't 1PB its 778TB...

It's 1PB of drives!

5 minutes ago, manikyath said:

AWS is a good option if you need all the data to be "hot" all the time, which video creators like yourself usually dont. offsite backup in an "actually well fitted out" datacenter is probably the best to go, especially since most of them need several failures in a row to even go dark at all, and actual data loss due to infrastructure failure is essentially zero.

 

as for long term archiving, tapes are still amazing

Good point. And yes tapes are something we're certainly looking into  haha

[manikyath]

Just now, nicklmg said:

Good point. And yes tapes are something we're certainly looking into  haha

contact sevadus, he's been looking into tapes (and other options including AWS) for long term archiving, it's been a while since he did the math, but i'm sure he still has the excel sheet.

[Reece Boddington]

5 minutes ago, nicklmg said:

It's 1PB of drives!

Good point. And yes tapes are something we're certainly looking into  haha

 

In the video, you said that the cluster could be expanding so you could just add another server.

[nicklmg]

18 minutes ago, ABXY Gaming said:

In the video, you said that the cluster could be expanding so you could just add another server.

That is correct.

[FelixC]

On 2017-04-04 at 3:15 PM, N.tony said:

4:49 "Um... couple of screws fell onto my boob" - did you hear that too? 

The bigger question is... how did the screw get into her boot in the first place!

[risk]

On 4/6/2017 at 0:26 AM, unijab said:

 

 $7.5k - $10k per month is 90k - 120k per year. 

So you're saying it would only be about 50k per year starting out since the full petabyte wont be used in the cloud?

No, I'm estimating operating expences at 50k a year in value for the petabyte server pair: it's the power cost + space cost + room cooling cost+ room noise insulation cost + security cost + installation labor + keeping expertise on hand in the form of some sysadmin cost to maintain the thing and deal with it when things go wrong (for accountants sysadmins are basically insurance that pile of electrically working hardware is suitable for business needs throughout that 1year).

 

Linus can not only afford to pay his own salary but also can make out of it through videos, what limits the size of his business is how much he can replicate himself in that regard, and how much he wants to.

 

But what you said about not paying for what you don't use in the cloud is also true.

 

Linus will likely fill up the server within next 6 months to a year though, making the investment more efficient.

[unijab]

I'd love to know how much power those beasts draw from the wall @ idle and during an average load during the day....

[Gavin Penney]

WTB:

zpool status -v

zpool list -v

[georgemr]

I would like to know the configuration of all the hard drives! And how did you manage to get those read and write speeds? 

[ky56]

Does anyone know what specific model they were? I was trying find out if they used SATA or SAS. The amazon link goes to a list of drives.

[GDRRiley]

1 hour ago, ky56 said:

Does anyone know what specific model they were? I was trying find out if they used SATA or SAS. The amazon link goes to a list of drives.

sata. the exact model shouldn't matter as seagate has rebranded a lot of stuff over the last 3 years