Type of drive for project

[MartinIAm]

Hello, I am working on a project and I want to make a storage server, that will hold the avatar (png, jpeg, etc.) of the users that signup. Each avatar has a max of 2mb size limit. Should I use 2.5 hard drives, if so what speed (7.2k, 10k, 15k(sas btw)), or should I use cheaper enterprise sas ssds for this because they can access any data at almost anytime? Also this will most likely be in raid 5 or 6 (If raid 5 they will be split into groups of drives) so I think that gives a read speed advantage. What should I choose?

[APasz]

What sort of project scope are we talking here?
Unless this server is mobile for some reason, there is no benefit to 2.5" drives.

If all it's doing is storing sub 2.5mb pfps then a 5.4k or 7.2k RPM drive is most likely fine.

An SSD would only be useful if there is likely to be massive rushes of requests (eg 1000's) on a regular basis.

[mariushm]

You're overthinking things.

Figure out first how many avatars you're gonna have, how fast the number will increase, and so on ... make a projection/estimation/whatever you want to call it.

You're dealing with small files, usually you will probably have 50-200 KB files, most of them, and then you'll have some lower percentage of .. let's say 10-20% of users .. with avatars bigger than 1 MB.

Let's say you're gonna have 100k of users or 100k of avatars ... figure out how many are active, how many post, how many pages search engines will browse and access throughout the day... most likely a lower percentage of users will be active. 

 

Let's go with the 100k avatars with an average of 200 KB per picture... that means around 50 GB of images if my quick math is right.

These days, you have 8 GB and 16 GB memory sticks, there's 60$ motherboards in which you can install 64 GB of memory, there's Threadripper which has 8 memory slots and supports up to 128-256 GB of memory.

Personally, I would get a 500 GB or 1 TB nvme drive with MLC or TLC memory... nvme for the IOPS and lower latency, for the random requests.

I would install 32 GB or more of memory in the server, and use 24+ GB to cache avatars in memory ... you can rely on the operating system to cache them, or you can use solutions like memcached, Redis, or database memory tables to cache the most often served images... that will hold tens of thousands of avatars live in ram.

A smart web server like nginx has reverse proxy and caching modules to do it completely on its own, without relying on php scripts or whatever.

This way, only the less often accessed images are requested from storage, reducing the number of requests per second.

You don't need enteprise SSD, unless you're dealing with 500k - 1m avatars or more , you also don't need raid or various such combinations (and i would argue you'd be better off having multiple light servers all connected to the storage on a NAS or something like that, and you'd use load balancer to route users to random light servers in such scenario)

A single nvme SSD can serve hundreds of small files per second which should be enough as the large majority of images would continue to be served directly from ram

For data safety, I would have a mechanical drive or a second SSD and some backup / rsync make copies / mirror of the main ssd once an hour , and I'd schedule a daily rsync/ incremental backup to a separate or remote server just in case the server blows up or both drives die.

The SSDs don't wear out from reads, only from writes... so as these are static files and don't change very often the ssd won't wear out any time soon.

 

So basically, what I would do is fill the drive with ram, and that will cache most often served images, use a nvme for hot storage, a mechanical drive or second m.2 for hot spare/backup/whatever.

If the server is just for serving static files, it can be some cheap cpu like an athlon 200ge or a ryzen 3000g , paired with a cheap MB with lots of ram (can be lower frequency, as for serving static files you don't need super fast ram)

You also don't care much about ECC as the world won't end if an image will be served to a visitor with a few pixels corrupted, if there's some bit flipped in the ram sticks.

 

Here's an example with 64 GB of memory ... but you can start with as little as 1 stick of ram and add sticks of ram as you feel its needed.. if the ssd can't keep up, you can add ram..

 

PCPartPicker Part List

Type	Item	Price
CPU	AMD Athlon 3000G 3.5 GHz Dual-Core Processor	$54.99 @ B&H
Motherboard	Gigabyte B450M DS3H Micro ATX AM4 Motherboard	$72.98 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Storage	Western Digital Blue SN550 500 GB M.2-2280 NVME Solid State Drive	$64.98 @ Amazon
	Prices include shipping, taxes, rebates, and discounts
	Total	$429.23
	Generated by PCPartPicker 2020-02-20 07:55 EST-0500

 

Also note that renting dedicated servers these is relatively cheap, you can rent servers for as little as 25-30$ a month, with multiple hard drives and 8-16 GB of memory ... or you can rent VPSes for less than that, but will less disk space (ex 50-100 GB)

It may be cheaper than buying your own server and colocating it or using your electricity and internet connection for it.

 

[MartinIAm]

10 hours ago, mariushm said:

You're overthinking things.

Figure out first how many avatars you're gonna have, how fast the number will increase, and so on ... make a projection/estimation/whatever you want to call it.

You're dealing with small files, usually you will probably have 50-200 KB files, most of them, and then you'll have some lower percentage of .. let's say 10-20% of users .. with avatars bigger than 1 MB.

Let's say you're gonna have 100k of users or 100k of avatars ... figure out how many are active, how many post, how many pages search engines will browse and access throughout the day... most likely a lower percentage of users will be active. 

 

Let's go with the 100k avatars with an average of 200 KB per picture... that means around 50 GB of images if my quick math is right.

These days, you have 8 GB and 16 GB memory sticks, there's 60$ motherboards in which you can install 64 GB of memory, there's Threadripper which has 8 memory slots and supports up to 128-256 GB of memory.

Personally, I would get a 500 GB or 1 TB nvme drive with MLC or TLC memory... nvme for the IOPS and lower latency, for the random requests.

I would install 32 GB or more of memory in the server, and use 24+ GB to cache avatars in memory ... you can rely on the operating system to cache them, or you can use solutions like memcached, Redis, or database memory tables to cache the most often served images... that will hold tens of thousands of avatars live in ram.

A smart web server like nginx has reverse proxy and caching modules to do it completely on its own, without relying on php scripts or whatever.

This way, only the less often accessed images are requested from storage, reducing the number of requests per second.

You don't need enteprise SSD, unless you're dealing with 500k - 1m avatars or more , you also don't need raid or various such combinations (and i would argue you'd be better off having multiple light servers all connected to the storage on a NAS or something like that, and you'd use load balancer to route users to random light servers in such scenario)

A single nvme SSD can serve hundreds of small files per second which should be enough as the large majority of images would continue to be served directly from ram

For data safety, I would have a mechanical drive or a second SSD and some backup / rsync make copies / mirror of the main ssd once an hour , and I'd schedule a daily rsync/ incremental backup to a separate or remote server just in case the server blows up or both drives die.

The SSDs don't wear out from reads, only from writes... so as these are static files and don't change very often the ssd won't wear out any time soon.

 

So basically, what I would do is fill the drive with ram, and that will cache most often served images, use a nvme for hot storage, a mechanical drive or second m.2 for hot spare/backup/whatever.

If the server is just for serving static files, it can be some cheap cpu like an athlon 200ge or a ryzen 3000g , paired with a cheap MB with lots of ram (can be lower frequency, as for serving static files you don't need super fast ram)

You also don't care much about ECC as the world won't end if an image will be served to a visitor with a few pixels corrupted, if there's some bit flipped in the ram sticks.

 

Here's an example with 64 GB of memory ... but you can start with as little as 1 stick of ram and add sticks of ram as you feel its needed.. if the ssd can't keep up, you can add ram..

 

PCPartPicker Part List

Type	Item	Price
CPU	AMD Athlon 3000G 3.5 GHz Dual-Core Processor	$54.99 @ B&H
Motherboard	Gigabyte B450M DS3H Micro ATX AM4 Motherboard	$72.98 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Memory	Crucial Ballistix Sport LT 16 GB (1 x 16 GB) DDR4-3000 Memory	$59.07 @ Amazon
Storage	Western Digital Blue SN550 500 GB M.2-2280 NVME Solid State Drive	$64.98 @ Amazon
	Prices include shipping, taxes, rebates, and discounts
	Total	$429.23
	Generated by PCPartPicker 2020-02-20 07:55 EST-0500

This looks pretty good and yea for the price it seems really good, but for just one drive I would do raid 1 and get another nvme drive with that. This should work for most people, but for me buying an enterprise server would be better because it has simply better reliability, even for storing something as simple as avatars. Probably for me it would work best with 15k or 10k drives with 600gb or more of storage per drive and put them in a 16 drive chassis, split them up into 2x raid 5 groups cloning each other to have better reliability. Also go with maybe 128gb of ram or more per server for cache because they are hard drives.

10 hours ago, mariushm said:

Also note that renting dedicated servers these is relatively cheap, you can rent servers for as little as 25-30$ a month, with multiple hard drives and 8-16 GB of memory ... or you can rent VPSes for less than that, but will less disk space (ex 50-100 GB)

It may be cheaper than buying your own server and colocating it or using your electricity and internet connection for it.

I like to host everything myself and buying used older servers is just a better deal in the end for me because I already have an infrastructure built, and virtual private servers just blow my whole idea of not using cloud services. Using electricity for me isn't a problem as electricity is relatively cheap where I live (about 6 cents per kw) and I also have solar panels and a generator which means I can stay off the grid for a while. Also, my internet kinda sucks right now (250mbps down 15mbps up), but I'm planning to get a fiber line within the next year so that isn't a concern for me.

 

In the end, as I said I will be just putting a bunch of 10k or 15k (or maybe even 7.2k rpm cause it's cheaper per gb) 2.5 inch sas hdds with 600gb or more per drive in 2x raid 5 arrays with 2 different raid controllers (clone) for maximum redundancy in a 16 drive chassis (maybe Dell r720xd) and use 128gb ecc of memory or more for cache per server and run like that. Currently, I'll start with one server and then expand in the future to more and use load balancing for avatars.

[mariushm]

11 hours ago, MartinIAm said:

This looks pretty good and yea for the price it seems really good, but for just one drive I would do raid 1 and get another nvme drive with that. This should work for most people, but for me buying an enterprise server would be better because it has simply better reliability, even for storing something as simple as avatars. Probably for me it would work best with 15k or 10k drives with 600gb or more of storage per drive and put them in a 16 drive chassis, split them up into 2x raid 5 groups cloning each other to have better reliability. Also go with maybe 128gb of ram or more per server for cache because they are hard drives.

In my opinion it's time to stop thinking mechanical hard drives. SSDs are reliable and for this particular application, they make more sense as you have basically no seek latency (no read/write heads to move to various locations)

It makes very little sense to look at a 16 drive chassis, seems like a waste of rack space. I'd look into a 1-2U server with 4 or more 2.5" drive bays, so that you can install a couple SSDs. Really though, you could install SSDs anywhere, just get a double sided sticky tape and tape the SSD to the case if you want to... they don't care.

For redundancy you can think of RAID but then it's probably gonna be easier to do it on SATA compared to nvme.

Mechanical drives fail more often and they'll be slower, even with 10k or 15k rpm.... SSDs fail less but often they fail suddenly so it makes sense to have backups... luckily they're so fast it's easy to just dump an image of the ssd every few hours if you want, or to do incremental backups.

 

 

 

WebHostingTalk forum has an advertising section, with a Hardware sub-section and you can often find decent deals there. 

For example, there's a company now clearing their Supermicro stock : https://www.webhostingtalk.com/showthread.php?t=1796350

 

Even the 125$ one would work for you considering it's just serving static content. Add a 10g ethernet/fiber card and if you want, add another 6 sticks of 16 GB for a total of 192 GB of memory and you're good to go.

 

You could have 2 of them (you're buying a 2nd server and 96GB of memory for <150$, sounds like a good deal) and use one of the gigabit ports as dedicated link between them and have requests from users randomly spread across both servers and the servers could keep each other's storage in sync so one would be a backup of the other and so on...

We have the below servers available immediately if collected/same DC, +$100/server if you need this shipped.

POST tested before being de-racked and moved/handed over to you. Sold as is. Most include RAID Cachevault/Battery.

Please email info [@] syn.ltd.uk or DM if interested.

Los Angeles, CA (Psychz)
------------
2U Supermicro SC825TQ-R740LPB Server (8 Bay 3.5" Chassis) (X9DRI-F)
2x Intel Xeon E5-2620
6x 16GB (96GB)
8x Caddies
LSI MegaRAID 9271-8i
PSU / Rail Kit
$200

2U Supermicro 826E16-R1200LPB Server (12 Bay 3.5" Chassis) (X8DT6)
2x Intel Xeon L5640 2.26GHz
6x 16GB (96GB)
12x 3.5" Caddies - No HDDs
LSI MegaRAID 9271-8i
PSU / Rail Kit
$125

2U Supermicro 826E16-R1200LPB Server (12 Bay 3.5" Chassis) (X8DT6)
2x Intel Xeon X5650 2.67GHz
6x 16GB (96GB)
12x 3.5" Caddies - No HDDs
LSI MegaRAID 9266-8i
PSU / Rail Kit
$150

 

 

 

[MartinIAm]

10 hours ago, mariushm said:

In my opinion it's time to stop thinking mechanical hard drives. SSDs are reliable and for this particular application, they make more sense as you have basically no seek latency (no read/write heads to move to various locations)

It makes very little sense to look at a 16 drive chassis, seems like a waste of rack space. I'd look into a 1-2U server with 4 or more 2.5" drive bays, so that you can install a couple SSDs. Really though, you could install SSDs anywhere, just get a double sided sticky tape and tape the SSD to the case if you want to... they don't care.

For redundancy you can think of RAID but then it's probably gonna be easier to do it on SATA compared to nvme.

Mechanical drives fail more often and they'll be slower, even with 10k or 15k rpm.... SSDs fail less but often they fail suddenly so it makes sense to have backups... luckily they're so fast it's easy to just dump an image of the ssd every few hours if you want, or to do incremental backups.

 

 

 

WebHostingTalk forum has an advertising section, with a Hardware sub-section and you can often find decent deals there. 

For example, there's a company now clearing their Supermicro stock : https://www.webhostingtalk.com/showthread.php?t=1796350

 

Even the 125$ one would work for you considering it's just serving static content. Add a 10g ethernet/fiber card and if you want, add another 6 sticks of 16 GB for a total of 192 GB of memory and you're good to go.

 

You could have 2 of them (you're buying a 2nd server and 96GB of memory for <150$, sounds like a good deal) and use one of the gigabit ports as dedicated link between them and have requests from users randomly spread across both servers and the servers could keep each other's storage in sync so one would be a backup of the other and so on...

We have the below servers available immediately if collected/same DC, +$100/server if you need this shipped.

POST tested before being de-racked and moved/handed over to you. Sold as is. Most include RAID Cachevault/Battery.

Please email info [@] syn.ltd.uk or DM if interested.

Los Angeles, CA (Psychz)
------------
2U Supermicro SC825TQ-R740LPB Server (8 Bay 3.5" Chassis) (X9DRI-F)
2x Intel Xeon E5-2620
6x 16GB (96GB)
8x Caddies
LSI MegaRAID 9271-8i
PSU / Rail Kit
$200

2U Supermicro 826E16-R1200LPB Server (12 Bay 3.5" Chassis) (X8DT6)
2x Intel Xeon L5640 2.26GHz
6x 16GB (96GB)
12x 3.5" Caddies - No HDDs
LSI MegaRAID 9271-8i
PSU / Rail Kit
$125

2U Supermicro 826E16-R1200LPB Server (12 Bay 3.5" Chassis) (X8DT6)
2x Intel Xeon X5650 2.67GHz
6x 16GB (96GB)
12x 3.5" Caddies - No HDDs
LSI MegaRAID 9266-8i
PSU / Rail Kit
$150

 

i agree with you on this one. I think it's time that hdds are getting outdated and that people should use ssds more, which is something good to keep in mind. But I was thinking to get a 2u server put half on ssd and the other half on hdd and when a user loads an avatar that avatar gets cached for like 24 hours and then gets moved back into the hard drives. Maybe also have a backup server also for the every avatar that is only SATA hdds possible even 5400rpm. I also already have a 10gb switch and a 10gb sfp+ network card is pretty cheap.

[mariushm]

11 minutes ago, MartinIAm said:

i agree with you on this one. I think it's time that hdds are getting outdated and that people should use ssds more, which is something good to keep in mind. But I was thinking to get a 2u server put half on ssd and the other half on hdd and when a user loads an avatar that avatar gets cached for like 24 hours and then gets moved back into the hard drives. Maybe also have a backup server also for the every avatar that is only SATA hdds possible even 5400rpm. I also already have a 10gb switch and a 10gb sfp+ network card is pretty cheap.

If you'll configure logging on your web server, you could write a script or small tool to parse the access log and get all the image downloads and calculate how often each image was accessed in the last n hours.

Then, sort by number of hits, and delete the least accessed files from SSD.

 

However, again, i don't see why you would have to delete pictures from a SSD when you can buy a 1-2 TB SSD (and there's enterprise ssds up to 8 TB or so) and also cache most accessed in ram.. the SSD won't become slower if you fill it up and by the time you have 1-2 TB worth of avatars, you can simply add more SSDs into the server

 

Also why "move back to the hard drive" ... there's 16 TB hard drives these days ... and there's no need to move from hard drive to ssd and back, just leave everything on the hard drive if it's more than the SSD space.

 

[MartinIAm]

15 minutes ago, mariushm said:

If you'll configure logging on your web server, you could write a script or small tool to parse the access log and get all the image downloads and calculate how often each image was accessed in the last n hours.

Then, sort by number of hits, and delete the least accessed files from SSD.

 

However, again, i don't see why you would have to delete pictures from a SSD when you can buy a 1-2 TB SSD (and there's enterprise ssds up to 8 TB or so) and also cache most accessed in ram.. the SSD won't become slower if you fill it up and by the time you have 1-2 TB worth of avatars, you can simply add more SSDs into the server

 

Also why "move back to the hard drive" ... there's 16 TB hard drives these days ... and there's no need to move from hard drive to ssd and back, just leave everything on the hard drive if it's more than the SSD space.

 

Haha, not manually. There is a disk caching service built into the database program I use and probably also ram cache. Also my idea of this is lets say somebody makes an account and uploads an avatar that is 2mb big and never logs on again and nobody else sees that avatar ever again for another year. In a big project this will happen a lot and it's just better to store it on something that isn't as costly per gb as ssds. Let's say 100,000 people are viewing the same page and more keep loading in and loading the page and reloading it, that would be best on ram, then 5,000 people are viewing this other person's page over 5 hours this would either work on ssd or on ram, if 100 people view one person's account in 10 hours it would be "ok" to put it on the ssd for 24 hours, if 1 person views one person's account in half a year then I would store it on the ssd for 24 hours and once a day has by then it puts it back into the main storage. Quite simple.