Will a mechanical hard drive be bottlenecked by Gigabit Ethernet?

[Vadise]

Greetings, I have been looking at two relatively affordable and mainstream solutions for externally holding storage, particularly WD's My Cloud Ex2 (Gigabit Ethernet NAS) and My Book Duo (USB 3.0). Both look to be exactly the same externally, and both handle RAID, along with storage expansion beyond the two included drive bays.

 

Given the simplicity of drive installation (and the fact that you can shove any 2.5 and 3.5 drive in them toolessly), the main use I intend to give them would be raid 0 scratch disk for rendering video projects, using mechanical hard drives. I have read that mechanical hard drives just do not have the read/write speed to utilize the full USB 3.0 bandwidth so that an external enclosure of that kind would not be a bottleneck, so the My Book would be viable.

 

However, having the versatility to access say a USB drive from any machine on my network if I were to use a My Cloud and place a USB drive on one of it's USB 3.0 ports sounds very attractive. So I ask, would Gigabit Ethernet have enough bandwidth so that two WD Blacks in Raid 0 wont be bottlenecked?

[Dayen]

Mechanical hard drives will probably be okay for USB 3.0, but there is no way that you will get Ethernet speeds using gigabit ethernet.  Hard drives simply do not read that fast.

[AshleyAshes]

3 minutes ago, Dayen said:

Mechanical hard drives will probably be okay for USB 3.0, but there is no way that you will get Ethernet speeds using gigabit ethernet.  Hard drives simply do not read that fast.

Yeah, ignore this post, it's incorrect.  1gbit is only 125 MB/s, basically every HDD currently sold hit 125MB/s sustained read or write without issue and those that can't barely come short.

 

Anyone who follows HDDs should know that 125MB/s is a pretty low bar for an HDD to hit.  So yes they 'Simply DO read that fast'.

[The Flying Sloth]

Will a mechanical hard drive be bottlenecked by Gigabit Ethernet?
 

If anything the ethernet will be bottlenecked by the drives (highly unlikely)

[mariushm]

Gigabit is 1000 mbps or 125 MB/s ...  classic hard drives these days can reach or exceed 250 MB/s at their highest but may go down as low as 60-80 MB/s at the other end of the platters.

So when saving to a hard disk installed in your computer, depending on where the operating system chooses to write the download, you may be limited by an individual hard drive or you may not be.  However, keep in mind that it's unlikely you'll have often individual downloads that will come at more than 50-60 MB/s (or about half a gigabit) so it's not really that big of an issue.

I have a 500 mbps internet connection and I basically peak it only with Steam downloads and with small files (< 500 MB .. 1 GB) like drivers from AMD which are served from content delivery networks. 

 

If you download multiple files at the same time or torrents, chances are you're going to use a client which has the option to use computer RAM to cache data before writing it to disk. The gigabytes of computer memory can act as a buffer that can be filled up at however peak your internet connection can reach , while the buffer is being "discharged" to hard drive at the maximum possible speed the drive could achieve.

 

For example, let's say you use a 2000 MB buffer in RAM and that you download with 100 MB/s but the hard drive can sustain only 80 MB/s ... after the first second you had 100 MB in ram, 80 were written to disk, 20 MB are left in memory.  The next second, you had 120 MB in ram, 80 MB are written to disk and 40 MB remain in buffer... keeping this up, for a 2000 MB buffer it would take about 100 seconds for the buffer (or a minute and 40 seconds) for the buffer to fill up.. then the client would have to catch up by spending 2000 MB / 80 MB/s  =  25 seconds to empty the buffer and then repeat the process .. in practice when the buffer is full the client just limits the incoming bandwidth to less than the hard drive speed so that the buffer can slowly be emptied to disk

 

With external drives you have the added latencies and processor usage of usb drivers .. usb 2.0 is limited to 480 mbps or 60 MB/s but in practice you'll see more like around 35-45 MB/s .. usb 3.0 has in theory 5gbps which would mean about 600 MB/s but in practice you probably wouldn't see more than 400-450 MB/s when using a SSD. Some USB 3.0 ports are coming from onboard controllers connected to one pci-e v2.0 lane which has a maximum bandwidth of 500 MB/s, so that's why you can never get the full 5 gbps but more like 400-450 MB/s. 

 

[Vadise]

So USB it is, thank you for the quick responses.

[Guest]

4 minutes ago, AshleyAshes said:

Anyone who follows HDDs should know that 125MB/s is a pretty low bar for an HDD to hit.  So yes they 'Simply DO read that fast'.

Don't forget HDD disks are round. Outer part is faster then inner part of the disk.

 

My Seagate 3 tb drive will hit +/- 200 mb/s on the outer circle, and I will give you an update on the inner circle.

[AshleyAshes]

3 minutes ago, Dutch-stoner said:

Don't forget HDD disks are round. Outer part is faster then inner part of the disk.

 

My Seagate 3 tb drive will hit +/- 200 mb/s on the outer circle, and I will give you an update on the inner circle.

Yes, I know what Constant Angular Velocity is.

[Guest]

So just did a speed test on my 3tb drive. (also to check health)

 

 

This was a 3.5" 7200 rpm HDD. A lower rpm 2.5" HDD will be alot slower than this. Just keep that in mind...

[mariushm]

If it matters, here's 4 of my 5 drives .. the last is a Sandisk X400 the boot drive, and note that the graph is not that representative because the defaults for benchmark is 64 KB blocks of data read from drives, which works fine for regular hard drives but not so much for SSDs.

By raising the block size to 8 MB, i get up to 400 MB/s .. still a bit below than what a Sandisk x400 ssd should do but good enough for me.

 

So a HGST NAS 4 TB with 18k hours on, an older WD green with 23700 hours of operation and an even older  1 TB WD Black with 52340 hours of operation or about 6 hours of 24/7 running and then the sandisk x400 ssd.

 

[Mark77]

Sustained sequential I/O is pretty rare on HDDs unless you're editing video.  So no, Gig-E is rarely the bottleneck unless you have a giant RAID or a significant number of drives and users.

[Phate.exe]

When I was copying my media collection from my gaming pc that was acting as a server to my actual server, I was seeing around 100MB/s write speeds.  Sometimes it would bump up to 110, sometimes drop to 90MB/s.  Both computers were hardwired to a gigabit router.

 

That's 720-880Mb/s, or 72-88% utilization of the network connection.  I have no idea where on the platters it was writing to (didn't care at the time), but it was pretty clear that the network connection was the limiting factor.

 

If a hard disk is capable of 150MB/s write speeds, that's 1200Mb/s, well above what gigabit ethernet is capable of.

[scottyseng]

Yes, a Ethernet connection will bottleneck a decent modern hard drive. Most are capable of 125MB/s and above, which is above what Ethernet can support. I have a large RAID array and I have to live with 80-100MB/s (It's rare to get 125MB/s since I'm also using the normal internet at the same time). I hope to get 10Gb/s some day...

 

It also depends on the type of data, large sequential files will bottleneck, where a lot of random small files will have the hard drive being the bottleneck.

[TRexFreak28]

Just as a reference:

 

I`m using 3x 3TB WD Red HDD`s (5400RPM) in a server running FreeNAS and archive around 110 MB/s via gigabit ethernet. (Read and Write)

The HDD`s are in a raid 5 / raid Z config with ZFS and i`m using the caching features of FreeNAS (L2ARC) so the speed is only limited by the network and not by the HDD`s.

Sadly i am not able to test this with 2x gbe or 10 gbe but when im copying 50 GB of Photos over the Network (5-10MB per File) i am always maxing out at the practical limit of gbe (111mb/s).

So i assume that a USB 3.0 Connection will be faster than gigabit when you are using multiple HDD`s in a raid 0 config or like at least 3 drives in a raid 5.