Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Hard drive question.

I found a new little hobby recently. It's creating massive files on my drives, so I got a brand new WD Green 2tb M.2. In the first week of installing it, according to Crystal Disk, I wrote about 20tb of data to it. The videos record at roughly 200-240gb of data an hour, so it writes pretty quickly. I could not find the expected terabytes written life expectancy of the drive anywhere.

But my real question is, with the amount of data I'm going through, at roughly 22tb a week, and creating at 5gb a minute, what kind of drives should I be looking at? I don't know much about drives tb lifetime write expectancy, be it 3.5 drive, 2.5 or m.2. And if it happens to be 3.5 drives with the best, could they handle properly writing data that fast for a few hours straight? 

Thanks for any help/replies in advance!

Link to comment
Share on other sites

Link to post
Share on other sites

Posted (edited)

First, an M.2 is not a hard drive, it is a solid state drive. And at the rate you're going, that WD Green M.2 drive will be toast in 1-3 years or less. Solid state can only go through so many write cycles before it's done. You are much better off with an actual HARD drive. At 22TB / week, 5GB /  minute, I'd recommend the Western Digital Blue or Black series. Black are faster and can be had in similar or higher capacity. I got a 6TB for $170 a few months ago, they're probably around $180-$200 now. But you can catch them on sale. 5400 rpm is okay for storage, but you really need 7200 rpm for speed.

Edited by An0maly_76
Revised, more info

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

Depending on the users need and the sustained load it could also be justifiable to look into a WD Gold series HDD

Link to comment
Share on other sites

Link to post
Share on other sites

22tb a week = ~1144 tb a year. 

 

So if a 5 year life is wanted, thats about 5.7pb of writes the drive needs. This puts you past basically any consumer drive like the 860 pro that is rated at 4.8pb or writes

 

You probably want a server grade drive. Id get something like a samgsung pm1725 or simmilar. You can find good deals on those sometimes and get about 20pB of rated writes. Or something like a intel s3700.

 

But also why such big videos. For most uses you can compress them much more with almost no quality difference.

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, An0maly_76 said:

First, an M.2 is not a hard drive, it is a solid state drive. And at the rate you're going, that WD Green M.2 drive will be toast in 1-3 years or less. Solid state can only go through so many write cycles before it's done. You are much better off with an actual HARD drive. At 22TB / week, 5GB /  minute, I'd recommend the Western Digital Blue or Black series. Black are faster and can be had in similar or higher capacity. I got a 6TB for $170 a few months ago, they're probably around $180-$200 now. But you can catch them on sale. 5400 rpm is okay for storage, but you really need 7200 rpm for speed.

I realized after I posted that rather then "hard" drive, I should have put storage. But I figured everyone here would just inherently understand the difference. And at 130 bucks, I'm only so worried about the life expectancy of the drive, as I find that very cheap for a couple terabytes of M.2. But you're sure 7200 is fast enough to handle this with no loss or skips in video output?

Link to comment
Share on other sites

Link to post
Share on other sites

Posted (edited)
2 minutes ago, ThisIsChew said:

I realized after I posted that rather then "hard" drive, I should have put storage. But I figured everyone here would just inherently understand the difference. And at 130 bucks, I'm only so worried about the life expectancy of the drive, as I find that very cheap for a couple terabytes of M.2. But you're sure 7200 is fast enough to handle this with no loss or skips in video output?

 

Plenty of us run 7200s without issue. It's when you start running RAID arrays you should worry. Trust me, if you try to generate media content with a 5400 rpm drive, you are going to be hating life. Not that it won't do it, but it will take what seems like forever in comparison to a 7200. Also, you won't care about life expectancy of the drive until it fails in the middle of a project, and SSDs are known to do that when at the end of the line.

 

Edited by An0maly_76
Revised, more info

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

According to WD's datasheet, that drive is rated for 100 TB of write endurance, so you've gone through about a fifth of that already. Although it will probably last longer than the datasheet says, I wouldn't trust it to hold out for more than a few months at this rate. Magnetic hard drives have much higher write endurance, but you'll need an array of many to keep up with that speed, which offsets the reliability gains.

 

Unless you change something about how you're using these drives, you'll need to switch to enterprise solid state drives if you want to keep writing like this for an extended period of time.

 

What is this hobby? Recording security footage in uncompressed 4K RAW?

¯\_(ツ)_/¯

 

 

Desktop:

Intel Core i7-11700K | Noctua NH-D15S chromax.black | ASUS ROG Strix Z590-E Gaming WiFi  | 32 GB G.SKILL TridentZ 3200 MHz | ASUS TUF Gaming RTX 3080 | 1TB Samsung 980 Pro M.2 PCIe 4.0 SSD | 2TB WD Blue M.2 SATA SSD | Seasonic Focus GX-850 Fractal Design Meshify C Windows 10 Pro

 

Laptop:

HP Omen 15 | AMD Ryzen 7 5800H | 16 GB 3200 MHz | Nvidia RTX 3060 | 1 TB WD Black PCIe 3.0 SSD | 512 GB Micron PCIe 3.0 SSD | Windows 11

Link to comment
Share on other sites

Link to post
Share on other sites

3 minutes ago, Electronics Wizardy said:

22tb a week = ~1144 tb a year. 

 

So if a 5 year life is wanted, thats about 5.7pb of writes the drive needs. This puts you past basically any consumer drive like the 860 pro that is rated at 4.8pb or writes

 

You probably want a server grade drive. Id get something like a samgsung pm1725 or simmilar. You can find good deals on those sometimes and get about 20pB of rated writes. Or something like a intel s3700.

 

But also why such big videos. For most uses you can compress them much more with almost no quality difference.

Really appreciate the specific drives to look into, so thanks for that. But it's a brand new thing to me, and I don't know much of anything about in regards to output quality, so I started at probably entirely too high. I'll probably slowly began to drop down the quality, but it'll probably take me a decent amount of time to learn what I need to see the output and notice quality difference. Not too much of an real eye for these things. And I'm kinda beyond stupid to these things. 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, ThisIsChew said:

Really appreciate the specific drives to look into, so thanks for that. But it's a brand new thing to me, and I don't know much of anything about in regards to output quality, so I started at probably entirely too high. I'll probably slowly began to drop down the quality, but it'll probably take me a decent amount of time to learn what I need to see the output and notice quality difference. Not too much of an real eye for these things. And I'm kinda beyond stupid to these things. 

What codecs are settings are you using? For 4k recoring 200mbit is normally more than enough with modern codecs, and that's much less data.

Link to comment
Share on other sites

Link to post
Share on other sites

Posted (edited)
8 minutes ago, BobVonBob said:

What is this hobby? Recording security footage in uncompressed 4K RAW?

Sounds a bit like what I do, live screen recording DVDs to storage so I don't have to mess with discs when I want to watch a movie. DVDs are cheaper than most streaming pay-per-view. I figured that out after about a month on Prime Video. I had to use 9600kbps @ 1920x1080p, and that's about 2-3GB per hour. I compress the final, but the raw recording is what eats up space, hence I got a 6TB. Yes, I've heard of Handbrake and DVDFab, but they have not yielded satisfactory results for me. Lots of pauses, etc. and Handbrake is generating rainbow-colored gibberish in the shapes of objects and characters as they move. Being autistic, I don't have the patience or inclination to figure that BS out.

Edited by An0maly_76
Revised, more info

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, An0maly_76 said:

live screen recording DVDs to storage

Y'know you can just rip them, right? MakeMKV is your friend.

Main System (Byarlant): Ryzen 7 3800XT | Asus B350-F Strix | EK 240mm Basic AIO | 16GB G.Skill DDR4 3200MHz CAS-14 | XFX RX 5600 XT THICC II | Samsung 960 PRO 512GB / Samsung 970 EVO 500GB / UMIS SSD 256GB / Crucial MX500 2TB / Samsung 840 EVO 250GB / WD White 7200RPM 8TB | Corsair RM750X | Mellanox ConnectX-3 10G NIC | Hyte Y60 Case | Dell U3415W Monitor | Microsoft Modern Keyboard

 

Laptop (Narrative): Lenovo Flex 5 81X20005US | Ryzen 5 4500U | 16GB RAM (soldered) | Vega 6 Graphics | SKHynix P31 1TB NVMe SSD | Intel AX200 Wifi (all-around awesome machine)

 

TrueNAS Server (Veda): Xeon E3-1241v3 | Supermicro X10SLL-F | Corsair H60 | 32GB Micron DDR3L ECC 1600MHz | 4x 10TB WD Whites / 4x 14TB Seagate Exos / 2x 1TB HGST 2.5" / 1x Samsung PM961 128GB SSD / 1x Kingston 16GB SSD | Seasonic Prime Fanless 500W | Mellanox ConnectX-3 10G NIC | LSI 9207-8i LBA | Fractal Design Node 804 Case (side panels swapped to show off drives)


Media Center/Video Capture (Jesta): Core i7-2600 | Asus H77M-PRO | Noctua NH-L12S | 16GB Crucial DDR3 | EVGA GTX750Ti SC | Sandisk UltraII SSD 64GB / Seagate 1.5TB HDD | Corsair CX450M | Hauppauge ImpactVCB-PCIe | Syba USB3.1 Gen 2 Card | LG UH12NS30 BD-ROM | Silverstone Sugo SG-11 Case

 

Camera: Sony ɑ7II (w/ Meike Grip) | Sony SEL24240 | Samyang 35mm ƒ/2.8 | Sony SEL50F18F | Sony SEL2870 (kit lens) | PNY Elite Perfomance SDXC cards


Tablet (---): Samsung Galaxy Tab A 8"
 

Spoiler

Laptop (Rozen-ZuluSony VAIO VPCF13WFX | Core i7-740QM | 8GB Patriot DDR3 | GT 425M | Kingston 120GB SSD | Blu-ray Drive | Intel 7260 Wifi (lived a good life, retired with honor)


Tablet (ReGZ): Asus T102HA (BIOS clock doesn't tick, loses time when sleep/off) (I kill tablets with disturbing regularity)

Tablet (Unicorn): Surface Pro 2 (battery will reset total capacity to current charge, leading Windows to think it's always 100% charged until it dies)

Tablet (Loto): Dell Venue 8 Pro (screen discoloration issues, wouldn't update to Windows 10)

Tablet: iPad 2 16GB (WiFi died, basically useless after that)

Testbed/Old Desktop (Kshatriya): Xeon X5470 @ 4.0GHz | ZALMAN CNPS9500 | Gigabyte EP45-UD3L | 8GB Nanya DDR2 400MHz | XFX HD6870 DD | OCZ Vertex 3 Max-IOPS 120GB | Corsair CX430M (?) | HooToo USB 3.0 PCIe Card | NZXT H230 Case (mostly intact, but some parts have been scavenged)

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, AbydosOne said:

Y'know you can just rip them, right? MakeMKV is your friend.

Yeah, I was adding my reasons for this in my last post. Some are copy protected and somehow manage to screw with the codec that ripping software uses.

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

21 minutes ago, Electronics Wizardy said:

What codecs are settings are you using? For 4k recoring 200mbit is normally more than enough with modern codecs, and that's much less data.

Nvenc H.264 with CQP at 12 for rate control. I was read to go at MKV, and then I have to switch that over to MP4 to do anything with the file later on, which creates a separate file of it that's MP4, so double the writing. Read never go straight to MP4, but that would have cut the drive writing in half. 

Link to comment
Share on other sites

Link to post
Share on other sites

26 minutes ago, BobVonBob said:

According to WD's datasheet, that drive is rated for 100 TB of write endurance, so you've gone through about a fifth of that already. Although it will probably last longer than the datasheet says, I wouldn't trust it to hold out for more than a few months at this rate. Magnetic hard drives have much higher write endurance, but you'll need an array of many to keep up with that speed, which offsets the reliability gains.

 

Unless you change something about how you're using these drives, you'll need to switch to enterprise solid state drives if you want to keep writing like this for an extended period of time.

 

What is this hobby? Recording security footage in uncompressed 4K RAW?

It's videos through OBS. Sorry to ask something of you, but I couldn't find that sheet anywhere. Would you mind to link it? Would really help me with my search for future drives. Crystal disk puts the drive health at 98%, which I would have figured took into account what the drive should be able to do, but I guess it does something entirely different then what I thought. 

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, ThisIsChew said:

Nvenc H.264 with CQP at 12 for rate control. I was read to go at MKV, and then I have to switch that over to MP4 to do anything with the file later on, which creates a separate file of it that's MP4, so double the writing. Read never go straight to MP4, but that would have cut the drive writing in half. 

12 is really high, set it to a much lower bitrate and the quality will be basically the sam.e 

 

You should be able to have obs make a mp4 Im pretty sure. Why not do that.

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Electronics Wizardy said:

12 is really high, set it to a much lower bitrate and the quality will be basically the sam.e 

 

You should be able to have obs make a mp4 Im pretty sure. Why not do that.

I did read 12 was high, but I wanted to get the best out of it when I started. I could increase it bit by bit, but I don't wanna get to a point where it starts to look worse in areas that I wouldn't notice, or would just simply oversee. And I read going straight to MP4 could have a higher chance of file corruption. Something like a single hiccup could corrupt the entire file output. I don't know too much, just starting to learn all this. 
Every new thing you get into in computers, just comes with such an overwhelming amount of things to learn and challenges to solve.

I appreciate all y'alls help.

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, ThisIsChew said:

I did read 12 was high, but I wanted to get the best out of it when I started. I could increase it bit by bit, but I don't wanna get to a point where it starts to look worse in areas that I wouldn't notice, or would just simply oversee. And I read going straight to MP4 could have a higher chance of file corruption. Something like a single hiccup could corrupt the entire file output. I don't know too much, just starting to learn all this. 
Every new thing you get into in computers, just comes with such an overwhelming amount of things to learn and challenges to solve.

I appreciate all y'alls help.

What bitrate are you seeing? What resolution and frame rate? From a quick calculation thats about 660mbit, and you could reduce it by 6x and probably won't notice any quality difference. 

 

Id just stick with mp4 unless you have issues when testing. What program is using the files? Can the program take mkv files?

Link to comment
Share on other sites

Link to post
Share on other sites

One other thing of note -- With PCIe3, sustained writing will likely be okay, but a PCIe4 would get quite hot.

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

28 minutes ago, An0maly_76 said:

One other thing of note -- With PCIe3, sustained writing will likely be okay, but a PCIe4 would get quite hot.

The drive it's going to is sitting about 48 idle, and 61 is what I've seen it up to. Haven't had crystal up for temps when active, so it could be slightly higher. It marks the 61 as red, so that's not great.

Link to comment
Share on other sites

Link to post
Share on other sites

38 minutes ago, Electronics Wizardy said:

What bitrate are you seeing? What resolution and frame rate? From a quick calculation thats about 660mbit, and you could reduce it by 6x and probably won't notice any quality difference. 

 

Id just stick with mp4 unless you have issues when testing. What program is using the files? Can the program take mkv files?

3840x2160, 60fps. 56 minute was lower in size at only 174gb, but data rate was 446748kbps, with total bitrate at 447010kbps.

The program I've been using is Vegas pro 19, which wouldn't let me drop in with MKV. 

Link to comment
Share on other sites

Link to post
Share on other sites

3 minutes ago, ThisIsChew said:

The drive it's going to is sitting about 48 idle, and 61 is what I've seen it up to. Haven't had crystal up for temps when active, so it could be slightly higher. It marks the 61 as red, so that's not great.

Some PCIe4s are known to reach 100C if written at full speed for long enough. Like I said, from a longevity standpoint, you're much better off with an actual HARD drive, as opposed to solid state. Solid state is faster in the moment, but does not last as long when large amounts of data are written in a short period of time, as in your exampled use.

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

33 minutes ago, ThisIsChew said:

3840x2160, 60fps. 56 minute was lower in size at only 174gb, but data rate was 446748kbps, with total bitrate at 447010kbps.

The program I've been using is Vegas pro 19, which wouldn't let me drop in with MKV. 

Thats a pretty high bitrate for that res and codec. Id try something like 100-200mbit and the quality should be about the same 

 

And id use mp4 unless you have issues using mp4

Link to comment
Share on other sites

Link to post
Share on other sites

45 minutes ago, An0maly_76 said:

Some PCIe4s are known to reach 100C if written at full speed for long enough. Like I said, from a longevity standpoint, you're much better off with an actual HARD drive, as opposed to solid state. Solid state is faster in the moment, but does not last as long when large amounts of data are written in a short period of time, as in your exampled use.

I didn't know enterprise drives were a thing. A 20tb WD Gold, 7200rpm, rated for 550tb a year seems perfect for me. And it's only 400 bucks. I'll do a test run or two with a 7,200rpm drive I have lying around to make sure, but this seems like a great deal of help from y'all.

 

Enterprise drive and 20tb for only 400 sems like ridiculously cheap. Here's hoping my friend.

Link to comment
Share on other sites

Link to post
Share on other sites

15 minutes ago, Electronics Wizardy said:

Thats a pretty high bitrate for that res and codec. Id try something like 100-200mbit and the quality should be about the same 

 

And id use mp4 unless you have issues using mp4

I'll definitely need to look at lower quality, because even bumping up from where I'm at, should still have near perfect quality. Especially since the time it takes to create proxy files to actually edit these files. It takes an insane amount of time to create those.

I'll need to do more research on MKV vs MP4 though. It creates double the writing this way, but I'm fine with the cost and time if it gives me an extra layer of protection. So much to learn man. I don't know how y'all do it, knowing all these random tidbits. 

Link to comment
Share on other sites

Link to post
Share on other sites

Posted (edited)
5 minutes ago, ThisIsChew said:

I didn't know enterprise drives were a thing. A 20tb WD Gold, 7200rpm, rated for 550tb a year seems perfect for me. And it's only 400 bucks. I'll do a test run or two with a 7,200rpm drive I have lying around to make sure, but this seems like a great deal of help from y'all.

 

Enterprise drive and 20tb for only 400 sems like ridiculously cheap. Here's hoping my friend.

Keep in mind, two things are important factors with HDDs... RPM, and cache. Larger cache = faster throughput. Think of it as a buffer for what the machine knows it will be reading, that the HDD can be commanded to run ahead and prepare it. Some 7200s do not have a cache at all, or a very large one if they do. 128MB is good, 256MB is better. There are some out there with more, but they are pricey, of course.

 

MTBF (Mean Time Between Failures) is important also, but a foregone conclusion in this day and age. It was a bigger deal when we were dealing with 20MB - 160MB drives, not, 4TB - 20TB.

Edited by An0maly_76
Revised, more info

MODERATE TO SEVERE AUTISTIC, COMPLICATED WITH COVID FOG

 

Due to the above, I've likely revised posts <30 min old, and do not think as you do.

THINK BEFORE YOU REPLY!

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share


×