Hard drive question.

[ThisIsChew]

I found a new little hobby recently. It's creating massive files on my drives, so I got a brand new WD Green 2tb M.2. In the first week of installing it, according to Crystal Disk, I wrote about 20tb of data to it. The videos record at roughly 200-240gb of data an hour, so it writes pretty quickly. I could not find the expected terabytes written life expectancy of the drive anywhere.

But my real question is, with the amount of data I'm going through, at roughly 22tb a week, and creating at 5gb a minute, what kind of drives should I be looking at? I don't know much about drives tb lifetime write expectancy, be it 3.5 drive, 2.5 or m.2. And if it happens to be 3.5 drives with the best, could they handle properly writing data that fast for a few hours straight? 

Thanks for any help/replies in advance!

[An0maly_76]

First, an M.2 is not a hard drive, it is a solid state drive. And at the rate you're going, that WD Green M.2 drive will be toast in 1-3 years or less. Solid state can only go through so many write cycles before it's done. You are much better off with an actual HARD drive. At 22TB / week, 5GB /  minute, I'd recommend the Western Digital Blue or Black series. Black are faster and can be had in similar or higher capacity. I got a 6TB for $170 a few months ago, they're probably around $180-$200 now. But you can catch them on sale. 5400 rpm is okay for storage, but you really need 7200 rpm for speed.

Edited July 2, 2022 by An0maly_76
Revised, more info

[MrMase]

Depending on the users need and the sustained load it could also be justifiable to look into a WD Gold series HDD

[Electronics Wizardy]

22tb a week = ~1144 tb a year. 

 

So if a 5 year life is wanted, thats about 5.7pb of writes the drive needs. This puts you past basically any consumer drive like the 860 pro that is rated at 4.8pb or writes

 

You probably want a server grade drive. Id get something like a samgsung pm1725 or simmilar. You can find good deals on those sometimes and get about 20pB of rated writes. Or something like a intel s3700.

 

But also why such big videos. For most uses you can compress them much more with almost no quality difference.

[ThisIsChew]

4 minutes ago, An0maly_76 said:

First, an M.2 is not a hard drive, it is a solid state drive. And at the rate you're going, that WD Green M.2 drive will be toast in 1-3 years or less. Solid state can only go through so many write cycles before it's done. You are much better off with an actual HARD drive. At 22TB / week, 5GB /  minute, I'd recommend the Western Digital Blue or Black series. Black are faster and can be had in similar or higher capacity. I got a 6TB for $170 a few months ago, they're probably around $180-$200 now. But you can catch them on sale. 5400 rpm is okay for storage, but you really need 7200 rpm for speed.

I realized after I posted that rather then "hard" drive, I should have put storage. But I figured everyone here would just inherently understand the difference. And at 130 bucks, I'm only so worried about the life expectancy of the drive, as I find that very cheap for a couple terabytes of M.2. But you're sure 7200 is fast enough to handle this with no loss or skips in video output?

[An0maly_76]

2 minutes ago, ThisIsChew said:

I realized after I posted that rather then "hard" drive, I should have put storage. But I figured everyone here would just inherently understand the difference. And at 130 bucks, I'm only so worried about the life expectancy of the drive, as I find that very cheap for a couple terabytes of M.2. But you're sure 7200 is fast enough to handle this with no loss or skips in video output?

 

Plenty of us run 7200s without issue. It's when you start running RAID arrays you should worry. Trust me, if you try to generate media content with a 5400 rpm drive, you are going to be hating life. Not that it won't do it, but it will take what seems like forever in comparison to a 7200. Also, you won't care about life expectancy of the drive until it fails in the middle of a project, and SSDs are known to do that when at the end of the line.

 

Edited July 2, 2022 by An0maly_76
Revised, more info

[BobVonBob]

According to WD's datasheet, that drive is rated for 100 TB of write endurance, so you've gone through about a fifth of that already. Although it will probably last longer than the datasheet says, I wouldn't trust it to hold out for more than a few months at this rate. Magnetic hard drives have much higher write endurance, but you'll need an array of many to keep up with that speed, which offsets the reliability gains.

 

Unless you change something about how you're using these drives, you'll need to switch to enterprise solid state drives if you want to keep writing like this for an extended period of time.

 

What is this hobby? Recording security footage in uncompressed 4K RAW?

[ThisIsChew]

3 minutes ago, Electronics Wizardy said:

22tb a week = ~1144 tb a year. 

 

So if a 5 year life is wanted, thats about 5.7pb of writes the drive needs. This puts you past basically any consumer drive like the 860 pro that is rated at 4.8pb or writes

 

You probably want a server grade drive. Id get something like a samgsung pm1725 or simmilar. You can find good deals on those sometimes and get about 20pB of rated writes. Or something like a intel s3700.

 

But also why such big videos. For most uses you can compress them much more with almost no quality difference.

Really appreciate the specific drives to look into, so thanks for that. But it's a brand new thing to me, and I don't know much of anything about in regards to output quality, so I started at probably entirely too high. I'll probably slowly began to drop down the quality, but it'll probably take me a decent amount of time to learn what I need to see the output and notice quality difference. Not too much of an real eye for these things. And I'm kinda beyond stupid to these things. 

[Electronics Wizardy]

1 minute ago, ThisIsChew said:

Really appreciate the specific drives to look into, so thanks for that. But it's a brand new thing to me, and I don't know much of anything about in regards to output quality, so I started at probably entirely too high. I'll probably slowly began to drop down the quality, but it'll probably take me a decent amount of time to learn what I need to see the output and notice quality difference. Not too much of an real eye for these things. And I'm kinda beyond stupid to these things. 

What codecs are settings are you using? For 4k recoring 200mbit is normally more than enough with modern codecs, and that's much less data.

[An0maly_76]

8 minutes ago, BobVonBob said:

What is this hobby? Recording security footage in uncompressed 4K RAW?

Sounds a bit like what I do, live screen recording DVDs to storage so I don't have to mess with discs when I want to watch a movie. DVDs are cheaper than most streaming pay-per-view. I figured that out after about a month on Prime Video. I had to use 9600kbps @ 1920x1080p, and that's about 2-3GB per hour. I compress the final, but the raw recording is what eats up space, hence I got a 6TB. Yes, I've heard of Handbrake and DVDFab, but they have not yielded satisfactory results for me. Lots of pauses, etc. and Handbrake is generating rainbow-colored gibberish in the shapes of objects and characters as they move. Being autistic, I don't have the patience or inclination to figure that BS out.

Edited July 2, 2022 by An0maly_76
Revised, more info

[AbydosOne]

1 minute ago, An0maly_76 said:

live screen recording DVDs to storage

Y'know you can just rip them, right? MakeMKV is your friend.

[An0maly_76]

Just now, AbydosOne said:

Y'know you can just rip them, right? MakeMKV is your friend.

Yeah, I was adding my reasons for this in my last post. Some are copy protected and somehow manage to screw with the codec that ripping software uses.

[ThisIsChew]

21 minutes ago, Electronics Wizardy said:

What codecs are settings are you using? For 4k recoring 200mbit is normally more than enough with modern codecs, and that's much less data.

Nvenc H.264 with CQP at 12 for rate control. I was read to go at MKV, and then I have to switch that over to MP4 to do anything with the file later on, which creates a separate file of it that's MP4, so double the writing. Read never go straight to MP4, but that would have cut the drive writing in half. 

[ThisIsChew]

26 minutes ago, BobVonBob said:

According to WD's datasheet, that drive is rated for 100 TB of write endurance, so you've gone through about a fifth of that already. Although it will probably last longer than the datasheet says, I wouldn't trust it to hold out for more than a few months at this rate. Magnetic hard drives have much higher write endurance, but you'll need an array of many to keep up with that speed, which offsets the reliability gains.

 

Unless you change something about how you're using these drives, you'll need to switch to enterprise solid state drives if you want to keep writing like this for an extended period of time.

 

What is this hobby? Recording security footage in uncompressed 4K RAW?

It's videos through OBS. Sorry to ask something of you, but I couldn't find that sheet anywhere. Would you mind to link it? Would really help me with my search for future drives. Crystal disk puts the drive health at 98%, which I would have figured took into account what the drive should be able to do, but I guess it does something entirely different then what I thought. 

[Electronics Wizardy]

2 minutes ago, ThisIsChew said:

Nvenc H.264 with CQP at 12 for rate control. I was read to go at MKV, and then I have to switch that over to MP4 to do anything with the file later on, which creates a separate file of it that's MP4, so double the writing. Read never go straight to MP4, but that would have cut the drive writing in half. 

12 is really high, set it to a much lower bitrate and the quality will be basically the sam.e 

 

You should be able to have obs make a mp4 Im pretty sure. Why not do that.

[ThisIsChew]

1 minute ago, Electronics Wizardy said:

12 is really high, set it to a much lower bitrate and the quality will be basically the sam.e 

 

You should be able to have obs make a mp4 Im pretty sure. Why not do that.

I did read 12 was high, but I wanted to get the best out of it when I started. I could increase it bit by bit, but I don't wanna get to a point where it starts to look worse in areas that I wouldn't notice, or would just simply oversee. And I read going straight to MP4 could have a higher chance of file corruption. Something like a single hiccup could corrupt the entire file output. I don't know too much, just starting to learn all this. 
Every new thing you get into in computers, just comes with such an overwhelming amount of things to learn and challenges to solve.

I appreciate all y'alls help.

[Electronics Wizardy]

4 minutes ago, ThisIsChew said:

I did read 12 was high, but I wanted to get the best out of it when I started. I could increase it bit by bit, but I don't wanna get to a point where it starts to look worse in areas that I wouldn't notice, or would just simply oversee. And I read going straight to MP4 could have a higher chance of file corruption. Something like a single hiccup could corrupt the entire file output. I don't know too much, just starting to learn all this. 
Every new thing you get into in computers, just comes with such an overwhelming amount of things to learn and challenges to solve.

I appreciate all y'alls help.

What bitrate are you seeing? What resolution and frame rate? From a quick calculation thats about 660mbit, and you could reduce it by 6x and probably won't notice any quality difference. 

 

Id just stick with mp4 unless you have issues when testing. What program is using the files? Can the program take mkv files?

[An0maly_76]

One other thing of note -- With PCIe3, sustained writing will likely be okay, but a PCIe4 would get quite hot.

[ThisIsChew]

28 minutes ago, An0maly_76 said:

One other thing of note -- With PCIe3, sustained writing will likely be okay, but a PCIe4 would get quite hot.

The drive it's going to is sitting about 48 idle, and 61 is what I've seen it up to. Haven't had crystal up for temps when active, so it could be slightly higher. It marks the 61 as red, so that's not great.

[ThisIsChew]

38 minutes ago, Electronics Wizardy said:

What bitrate are you seeing? What resolution and frame rate? From a quick calculation thats about 660mbit, and you could reduce it by 6x and probably won't notice any quality difference. 

 

Id just stick with mp4 unless you have issues when testing. What program is using the files? Can the program take mkv files?

3840x2160, 60fps. 56 minute was lower in size at only 174gb, but data rate was 446748kbps, with total bitrate at 447010kbps.

The program I've been using is Vegas pro 19, which wouldn't let me drop in with MKV. 

[An0maly_76]

3 minutes ago, ThisIsChew said:

The drive it's going to is sitting about 48 idle, and 61 is what I've seen it up to. Haven't had crystal up for temps when active, so it could be slightly higher. It marks the 61 as red, so that's not great.

Some PCIe4s are known to reach 100C if written at full speed for long enough. Like I said, from a longevity standpoint, you're much better off with an actual HARD drive, as opposed to solid state. Solid state is faster in the moment, but does not last as long when large amounts of data are written in a short period of time, as in your exampled use.

[Electronics Wizardy]

33 minutes ago, ThisIsChew said:

3840x2160, 60fps. 56 minute was lower in size at only 174gb, but data rate was 446748kbps, with total bitrate at 447010kbps.

The program I've been using is Vegas pro 19, which wouldn't let me drop in with MKV. 

Thats a pretty high bitrate for that res and codec. Id try something like 100-200mbit and the quality should be about the same 

 

And id use mp4 unless you have issues using mp4

[ThisIsChew]

45 minutes ago, An0maly_76 said:

Some PCIe4s are known to reach 100C if written at full speed for long enough. Like I said, from a longevity standpoint, you're much better off with an actual HARD drive, as opposed to solid state. Solid state is faster in the moment, but does not last as long when large amounts of data are written in a short period of time, as in your exampled use.

I didn't know enterprise drives were a thing. A 20tb WD Gold, 7200rpm, rated for 550tb a year seems perfect for me. And it's only 400 bucks. I'll do a test run or two with a 7,200rpm drive I have lying around to make sure, but this seems like a great deal of help from y'all.

 

Enterprise drive and 20tb for only 400 sems like ridiculously cheap. Here's hoping my friend.

[ThisIsChew]

15 minutes ago, Electronics Wizardy said:

Thats a pretty high bitrate for that res and codec. Id try something like 100-200mbit and the quality should be about the same 

 

And id use mp4 unless you have issues using mp4

I'll definitely need to look at lower quality, because even bumping up from where I'm at, should still have near perfect quality. Especially since the time it takes to create proxy files to actually edit these files. It takes an insane amount of time to create those.

I'll need to do more research on MKV vs MP4 though. It creates double the writing this way, but I'm fine with the cost and time if it gives me an extra layer of protection. So much to learn man. I don't know how y'all do it, knowing all these random tidbits. 

[An0maly_76]

5 minutes ago, ThisIsChew said:

I didn't know enterprise drives were a thing. A 20tb WD Gold, 7200rpm, rated for 550tb a year seems perfect for me. And it's only 400 bucks. I'll do a test run or two with a 7,200rpm drive I have lying around to make sure, but this seems like a great deal of help from y'all.

 

Enterprise drive and 20tb for only 400 sems like ridiculously cheap. Here's hoping my friend.

Keep in mind, two things are important factors with HDDs... RPM, and cache. Larger cache = faster throughput. Think of it as a buffer for what the machine knows it will be reading, that the HDD can be commanded to run ahead and prepare it. Some 7200s do not have a cache at all, or a very large one if they do. 128MB is good, 256MB is better. There are some out there with more, but they are pricey, of course.

 

MTBF (Mean Time Between Failures) is important also, but a foregone conclusion in this day and age. It was a bigger deal when we were dealing with 20MB - 160MB drives, not, 4TB - 20TB.

Edited July 2, 2022 by An0maly_76
Revised, more info