External RAID 1 Write Speeds

[ShadowWolf810]

The Setup: An external hard drive enclosure with 2, WD RE 4TB Drives in RAID 1, and the option for eSATA, Firewire, or USB 3.0 connections. And an Oyen MiniPro 1TB USB 3.0

Links:

http://www.newegg.com/Product/Product.aspx?Item=9SIA54G2PP9628&cm_re=mobius_5_bay-_-9SIA54G2PP9628-_-Product

http://www.newegg.com/Product/Product.aspx?Item=N82E16822236350&cm_re=wd_harddrives-_-22-236-350-_-Product

https://oyendigital.com/hard-drives/store/U32M-HDD-1000-BK.html

 

Brought my RAID to a short film shoot for dumping CFast cards on set, and was getting relatively slow transfer speeds 20-30 MB/s, but this was done with a USB 2 card reader and a laptop so I didn't think anything of it. However, I'm now trying to give the footage out, and whether I used USB 3 connections for both hard drives or eSATA for the RAID and USB 3 for the other drive, I could only get transfer speeds between 15-20 MB/s. I've tried changing the drive policies to be "Better Performance" rather than the default which allows for not needing to safely eject it, and that did initially bring the speed up quite drastically, over a minute or so it quickly fell back down to the same rate.

 

As for the type of files being transferred, there are about 30,000 individual small files, comprising about 200 GB total. The film was shot in RAW, so each clip is a folder containing individual DNG files which represent 1 frame. However the problem is the same when trying to transfer another shoot's footage between the same two drives, only difference being that shoot is made up of clips all between 3-15 GB each. 

 

Any suggestions would be appreciated! 

[Electronics Wizardy]

How big are the DNG files. That can be the problem.

 

Id suggest getting a single 1TB ssd. More reliable and much faster.

[ShadowWolf810]

3 minutes ago, Electronics Wizardy said:

 

They're all about 5 MB each

[Electronics Wizardy]

Just now, ShadowWolf810 said:

They're all about 5 MB each

Get crystal disk mark and see what you get

 

 

[ShadowWolf810]

13 minutes ago, Electronics Wizardy said:

Get crystal disk mark and see what you get

 

 

These were the results when testing the RAID with "Better Performance" off, and with a USB 3 connection. 

https://gyazo.com/39ccbc84e692f0815d41ecd85ba02351

 

The drives are advertised as being capable of 171 MB/s transfer speeds, so it seems like too big of a hit compared to the real world performance for there not to be an issue with it. As far as my knowledge goes, RAID 1 read and write speeds are supposed to be the same as the individual drives.

 

 

[Electronics Wizardy]

Just now, ShadowWolf810 said:

These were the results when testing the RAID with "Better Performance" off, and with a USB 3 connection. 

https://gyazo.com/39ccbc84e692f0815d41ecd85ba02351

 

The drives are advertised as being capable of 171 MB/s transfer speeds, so it seems like too big of a hit compared to the real world performance for there not to be an issue with it. 

 

 

Id try a different raid mode.

 

Its probably a cheap raid chipset and there is nothing your can do. Id return it and get a single 1tb external ssd.

[ShadowWolf810]

4 minutes ago, Electronics Wizardy said:

Id try a different raid mode.

 

Its probably a cheap raid chipset and there is nothing your can do. Id return it and get a single 1tb external ssd.

Well I mostly use it on film sets, so the redundancy is important. I just switched over to the usual eSATA connection and these are the results when testing the RAID. 

https://gyazo.com/8e59a9a520d548eaedd18dea07f2cb14

 

And real world transfer speeds were now up to 120 MB/s as well. But if I'm correct the USB 3.0 max transfer speeds are 5 Gbps, and eSATA is 6 Gbps, so there shouldn't be that big of a difference between the two connections. Its fine when I'm at my desktop and have a convenient eSATA connection but out in the world with a laptop it seems ridiculously slow when it shouldn't be

 

[Electronics Wizardy]

Just now, ShadowWolf810 said:

Well I mostly use it on film sets, so the redundancy is important. I just switched over to the usual eSATA connection and these are the results when testing the RAID. 

https://gyazo.com/8e59a9a520d548eaedd18dea07f2cb14

 

And real world transfer speeds were now up to 120 MB/s as well. But if I'm correct the USB 3.0 max transfer speeds are 5 Gbps, and eSATA is 6 Gbps, so there shouldn't be that big of a difference between the two connections. Its fine when I'm at my desktop and have a convenient eSATA connection but out in the world with a laptop it seems ridiculously slow when it shouldn't be

 

A single ssd would be more reliable than a raid 1 of hdds, esp since it will work fine when dropped.

 

 

[ShadowWolf810]

9 minutes ago, Electronics Wizardy said:

A single ssd would be more reliable than a raid 1 of hdds, esp since it will work fine when dropped.

Its possible it could be worth the investment, but right now I'm trying to figure out why the USB 3.0 transfer speed are 6 times slower than eSATA when they shouldn't be. 

[Electronics Wizardy]

Just now, ShadowWolf810 said:

Its possible it could be worth the investment, but right now I'm trying to figure out why the USB 3.0 transfer speed are 6 times slower than eSATA when they shouldn't be. 

Probably a bad chip

[ShadowWolf810]

Just now, Electronics Wizardy said:

Probably a bad chip

Do you mean in the enclosure?

[Captain_WD]

4 hours ago, ShadowWolf810 said:

~snip~

Hello

 

These results aren't really supposed to be as low for WD Re drives. I'd try connecting each drive to a working PC internally and repeat the tests and see what benchmarks you will see there. It is likely that the enclosure itself is limiting the performance of the storage drives. 

 

Also, I would recommend running some diagnostic tools on those WD Re drives to be sure they are healthy and performing as they should. I'd download and run WD Data Lifeguard Diagnostic tool and see if the drives pass both the quick and the extended tests. 

 

Let me know if you have any questions! 

 

Captain_WD. 

[ShadowWolf810]

On 10/25/2016 at 2:00 AM, Captain_WD said:

Hello

So I downloaded the WD Data Lifegaurd Tool, the drives passed the quick test, haven't gotten around to running the extended test but I have made some changes and I think solved the issue. Since posting this I've flipped my raid over from Raid 1 to Raid 0, and I've also tried connecting with different USB ports, as well as doing a low level format with HDD Low Level Format Tool. After some thinking I thought perhaps it was a bus bandwidth issue with the motherboard since I do have a ton of other USB devices plugged in, so I plugged into a USB 3.0 PCIe card, and these were the results.

 

USB 3.0 PCIe: https://gyazo.com/1db33ada7fc27885ad3754ca631a5da9

 

eSata: https://gyazo.com/e253aefebd1f083268ec8770d21d79ac

 

I'm not really sure how it could cause this much of an issue with bandwidth, only one other port has another hard drive, the rest of my USB ports are just mouse, keyboard, microphone, and some other devices that are off or have nothing plugged into them a lot of the time. Regardless, it seems like that was the issue!

 

Are there any utilities out there that'll show you which devices are taking up that bandwidth? 

 

If I hadn't literally just spent the day migrating all my data back and forth to reconfigure the Raid yesterday I would have done some more testing with it connected internally. Have never tried these drives anywhere other than the enclosure I got for them, is it possible to just take 1 drive from a Raid 0 and plug it in for testing? Seems to me like that wouldn't work/should definitely be avoided with a Raid 0 config. I'm guessing you could probably take both drives out and connect them internally and setup a new Raid 0 in the BIOS with the same settings and it should be able to read all the data, but that's quite a bit of work and I need to get back to using the drives.

 

In doing some more reading however, I'm seeing a few things, one is that eSata is only capable of 300 MB/s, compared to the 750 MB/s of Sata, (which I didn't realize at the start of this post). And USB 3.0 at 5 Gb/s should be capable of 625 MB/s. I do totally understand that these are their theoretical limits, and real world performance will be much much slower, (although when connected with eSata I got speeds faster than even the theoretical limit, assuming I've got all my numbers and specs correct so that's a bit strange.) Anyways, I don't have enough experience to know whether the numbers I'm getting now are much more typical, or if its still drastically slower than what I should be expecting and there's still some other bottleneck or something going on! 

 

 

[Captain_WD]

13 hours ago, ShadowWolf810 said:

~snip~

Thank you for the detailed explanation on your actions and the extensive clarification! 

 

Do check if the drives pass the extended tests as well as check the raw values of the S.M.A.R.T. status (you will need another tool for this).

 

The enclosure may have a problem running RAID1 since you appear to have regular speeds with RAID0 mode on. Try contacting the manufacturer of the enclosure. 

 

You are not likely to get the data off a RAID0 array if you plug it in another controller as those arrays heavily depend on the specific controller, its model, firmware, settings, etc. 

 

Just like regular SATA, eSATA has three revisions with the latest one capable of delivering up to 6Gb/s or 600MB/s. 

 

Let me know if you have any other questions! 

 

Captain_WD.