LTT Storage Rankings

[MG2R]

How do you use 57 Tb?

He has the secret formula to craft a unicorn out of soap stored in an encrypted fashion.

 

Also: millions of family pictures

[Jarsky]

How do you use 57 Tb?

 

 

13th Place Here. 

 

Just like to say, if I had that much, I totally wouldn't be deleting stuff that i've been having too for the last year, to get new content 

Pretty easy to fill that much space unfortunately 

[Hellboy]

i've got about 15 TB + about 10TB in a second server

 

server 1

case -  cooloermaster haf-X

cpu - amd Phenom X6 1090T

ram - 16GB DDR3

motherboard - asus crosshair V formula

OS - windows 7

controller - internal

Raid - nothing yet, but coming

network - internal

 

server 2

cpu - AMD sempron 145

Ram - 4GB DDR3

motherboard - asus M5A78L-M LX

OS - windows 7

controller - internal

Raid - nothing yet, but coming

network - internal

 

 

its a job on putting these 2 servers into 1, as soon as i have bouht the right parts

so work in progress

 

i have now built me a new server now with additional 38 TB storage

as in total i now have 58 TB of storage

 

the new server contains

 

case : fractal design array R2

CPU : intel celeron G555

MB : gigabyte GA-Z77N-wifi

raid: IBM M1015

SSD : C300

HDD 1: 2*3 TB WD green  (JBOD)

HDD 2: 8*4 TB seagate barracuda  (raid 5)

SW: windows server 2008 R2

 

 

 

Edited August 15, 2014 by alpenwasser
fixed images

[Patrick3D]

Hardware
CASE: Raidmax Seiran (red)
PSU: Silverstone Strider 550Watt
MB: MSI FM2-A85XA-G65
CPU: AMD A4-5300 APU
HS: Stock AMD heatsink
RAM: 16GB DDR3 1600
RAID CARD 1: Rosewill RC-211 (1x DVD & 1x Blu-Ray)
RAID CARD 2: Rosewill RC-227 (2x HDDs)

/DEV/SDA: 150GB Seagate Barracuda (old drive, pulling data off of it, not counted in total size) 
/DEV/SDB: 3TB External USB drive
/DEV/SDC: 60GB Kingston SSD 
/DEV/SDD: 3TB WD Red (LVM01)
/DEV/SDE: 2TB WD Black (LVM01)

/DEV/SDF: 1TB WD Black (LVM01)

/DEV/SDG: 4TB Seagate Barracuda (LVM02)

/DEV/SDH: 2TB Seagate Barracuda (LVM02)

/DEV/SDI: 2TB External USB
Total Capacity: 12TB (not counting SSD, external USB, or old drives)

 

Software and Configuration:
My server is running Ubuntu 13.10 and I'm using lvm2 via the kvpm GUI to make my arrays. 
My main volume is 6TB (3TB+2TB+1TB) and called LVM01-Archive (mounted as "Archive")

My 2nd volume is 6TB (4TB+2TB) and called LVM02-Archive (mounted as "Backup")

 

Usage:
I use the storage for movies/TV/photos/games, other computers in the apartment access it over the network and remotely through FTP.
 
Backup:
I use a tool called RsyncBackup to synchronize LVM01 with LVM02, it is not automated, I just run it manually whenever there is new data to synchronize. I use USB drives to backup photos and other non-replaceable data as well as Ubuntu One and burned DVDs.
 
Additional info:
The low end AMD APU handles everything beautifully, I went a little overboard with RAM but it was cheap when I bought it and I wanted to have spare for VM usage. The drives are installed in a pair of 4x3 iStarUSA trayless hot-swap bays BPN-DE340SS-RED. The SSD is in a single, separate, iStarUSA hot-swap bay (between the two optical drives) for easy replacement in case it ever dies. There is also one DVD drive and one Blu-Ray drive for backing up media. I'm using LVMs for my own education (I never heard of them until 2 years ago), so far no issues. They're easy to grow, but when I replace drives I rebuild from scratch to be safe. Haven't had a drive die yet in 2 years, only been replacing for additional capacity. Need to add another pair soon as I'm down to less than 200GB free.
 
Photo's:

(Note: old photo, number of drives has changed)

 

[alpenwasser]

Backup:

I use a tool called RsyncBackup to synchronize LVM01 with LVM02, it is not automated, I just run it manually whenever there is new data to synchronize. I use USB drives to backup photos and other non-replaceable data as well as Ubuntu One and burned DVDs.

Burned DVDs? Now that's some major year 2000 mojo going on dear sir.

But yeah, nice rig.

[MG2R]

Burned DVDs? Now that's some major year 2000 mojo going on dear sir.

But yeah, nice rig.

I'm still waiting for the first rig that implements tape drives.

[looney]

I'm still waiting for the first rig that implements tape drives.

I´m still very tempted to get a tape library  

[MG2R]

I´m still very tempted to get a tape library

Have been contemplating the idea myself for some time, just don't have the money... Would love a box full of tapes though

[alpenwasser]

Have been contemplating the idea myself for some time, just don't have the money... Would love a box full of tapes though

Haha, exact same thing here, a box of tapes would be so awesome, but it would

also cost me an arm, a leg and probably something else I'm not willing to part

with.

[MG2R]

Haha, exact same thing here, a box of tapes would be so awesome, but it would

also cost me an arm, a leg and probably something else I'm not willing to part

with.

The moment they're able to transplant my brain into a machine, they may take my body. It's a crappy one anyway

[alpenwasser]

The moment they're able to transplant my brain into a machine, they may take my body. It's a crappy one anyway

Haha, I have been saying the same thing for years (ever since I played

Ending No. 2 in the original Deus Ex IIRC).

But, as long as I still need this body, I would prefer it stay in one

piece...

[RandomNOOB]

You need to update the list @looney 

 

This is quite interesting  http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/ 

[alpenwasser]

You need to update the list @looney 

 

This is quite interesting  http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/ 

Very interesting article, and it matches nicely with the Google study I

came across a few years back: The primary killer of HDDs seems to be old

(or not-so-old, depending on the drives) age.

Also nice to see Hitachi are doing so well in the Enterprise sector, let's

just hope they continue to be that good now that they're owned by WD, I'll

definitely look into Hitachi drives for my next storage pool expansion (though

that is still a while off).

[The Syndicate]

What do they keep in these things?

[alpenwasser]

What do they keep in these things?

For me, it's mostly TV shows, movies and documentaries (so, mostly high-def

video files), along with lots and lots of pictures.

[The Syndicate]

For me, it's mostly TV shows, movies and documentaries (so, mostly high-def

video files), along with lots and lots of pictures.

what's the file size of an average movie file you have?

[alpenwasser]

what's the file size of an average movie file you have?

Depends. A 42 minute TV episode can vary from anything between 750 MiB and

1.8 GiB, depending on quality; if you get a file that has a BluRay as its

source this can go up to 2.5 GiB. So, for a show with more than 20 episodes

per season, this adds up rather quickly.

Movies usually start at around 4.4 GiB (the versions I'm interested in) and

go up from there. Can go up to 20 GiB for a good quality version of a long

movie.

Pictures are not to be taken lightly either though. I have a few automated

scripts which make it very easy to download lots of pics from specific websites,

and once you go into the tens of thousands of files they add up as well.

Also, some space websites provide extremely high-res TIFF versions of their

files, which can be several hundred megabytes per image.

Lastly, I dabble in photography, and those RAW image files are not exactly

small...

[MG2R]

Lastly, I dabble in photography, and those RAW image files are not exactly

small...

24 * 1.000.000 / 8 = 3 MB / Mpixel? Something like that?

[alpenwasser]

24 * 1.000.000 / 8 = 3 MB / Mpixel? Something like that?

Canon's RAW files (*.CR2) are losslessly compressed, so they equal out to

something between 22 MiB and 30 MiB in most of my files.

Having an 18 MP sensor, your calculation would come out to about 51.5 MiB

for the uncompressed image for my camera (well, 54,000,000 bytes, to be exact).

But then I usually create a TIFF (an 8 bit one, if I recall correctly) from

the CR2 file for further processing, which is precisely 53,924,336 bytes in

size, thus matching very nicely your calculation.

After having done what I need to do to the TIFF, I then import the image

into GIMP and create an XCF file, which usually varies in size from anything

between 20 MiB and about 50 MiB.

Then I normally export it to JPEG or PNG files (resolution-adjusted), which are

a few hundred KiB to a few MiB.

Yes, I realize there's potential for optimization here. I could compress

TIFFS once I am through with them, or delete them altogether and merely

save the profile which would allow me to recreate them exactly from the

CR2 again.

One might also argue for deletion of the CR2 input files, but I don't

really want to part with the original source of the images.

In any case, I haven't been at this photography business very long, so

I might still look into what possibilities I have at some point, but

for the time being I have the space to store all that stuff, so it doesn't

really make too much of a difference if I waste a few dozen GiB with image

files.

[MG2R]

Yes, I realize there's potential for optimization here. I could compress

TIFFS once I am through with them, or delete them altogether and merely

save the profile which would allow me to recreate them exactly from the

CR2 again.

I was merely trying to give The Syndicate an idea of what "not exactly small" is If you want to save space, just delete everything except for the source and the end product when you're (temporarily) done with the project. Since everything you have there is derived from the source, you can always recreate it if necessary. Given that most of the actions you do are repetative, you could script it, making the time needed for re-creating the files relatively small

[alpenwasser]

I was merely trying to give The Syndicate an idea of what "not exactly small" is If you want to save space, just delete everything except for the source and the end product when you're (temporarily) done with the project. Since everything you have there is derived from the source, you can always recreate it if necessary. Given that most of the actions you do are repetative, you could script it, making the time needed for re-creating the files relatively small

No problem, and yeah, I think this would probably be helped by scripts

to some extent (not quite sure if all programs I'm using lend themselves

to being scripted, but if the need ever truly arises I shall look into

it; I prefer scripts over repetitive manual labour ).

[AidanJMessenger]

Pictures are not to be taken lightly either though. I have a few automated

scripts which make it very easy to download lots of pics from specific websites,

and once you go into the tens of thousands of files they add up as well.

Also, some space websites provide extremely high-res TIFF versions of their

files, which can be several hundred megabytes per image.

 

 

What websites are you downloading photos from and for what reason? Sounds like something I should start doing, so I have plenty of stock photos etc on hand. What's your use for such a thing?

[alpenwasser]

What websites are you downloading photos from and for what reason? Sounds like something I should start doing, so I have plenty of stock photos etc on hand. What's your use for such a thing?

For space pictures, I have mirrored the eso.org site, esa.int and the hubble

archive (or part of it, not sure anymore). Many of their best images are

available as high-res TIFFs, which is pretty awesome, but takes up lots of

space.

Aside from that I have also grabbed a few thousand airplane pics from af.mil.

Most recently, I downloaded the entire archive from wallbase.cc (~800k

images), though I have not sorted through them yet. Took a few days, but

luckily their wallpapers have consecutively numbered consistent filenames,

so it was very easy to do (other sites are not so convenient and require

quite a bit of regex magic to accomplish this).

I also have written a script for subscribing to threads on 4chan, but TBH

there's a lot of horrible pseudo-pr0n and celebrity worshipping stuff on

there, so getting to the good stuff (and occasionally there really is)

requires a bit of work.

The reason? I'm a collector I suppose, and grabbing an entire archive, then

sorting through it and deleting the images I do no want is easier and faster

than looking through the catalogue on the web and 'right click->save as' for

each image I want.

[looney]

Updated the list,

I might have to upgrade my system soon as I intend to stay in the top 10 on my own list

[Alexdaman]

I just upgraded my server to 52Tb so it could use an update  

 

also have a mini server from HP will post pictures soon|