VERY slow writing speed to external HDD

[mancesco]

Hi everyone,

I have a Seagate Barracuda 2TB 2.5" 5400rpm drive (that I put in a usb 3.0 external enclosure) that behaves oddly. I'm transferring on it 100 GB of small files (~35,000 files, 2.8 MB each) from another external hdd and the writing speed is excruciatingly slow, jumping between 40 and 1.5 MB/s every few seconds. Granted it is a slow drive, but those speeds seem pretty far from normal. Here's a bunch of info and things I've already tried:

 

- the hard drive is brand new, got it directly from Seagate (I RMA'd another one that failed), and was freshly formatted to NTFS when the issue started

- I checked the SMART status with CrystalDiskInfo and it's all good

- I tried the same drive with different USB 3.0 ports and with other enclosures and a SATA to USB 3.0 adapter, always got the same result

- I also tried stopping all unnecessary software running in background, temporarily disabled the antivirus and superfetch

- I cannot enable write-caching on this drive

- Windows, drivers and bios are all up to date, laptop was rebooted many times

- my system is a DELL XPS 15 9560 (Core i7 7700HQ, 16 GB of RAM, GTX 1050, Samsung 500GB nvme SSD, Windows 10 Home)

 

Any help to diagnose/resolve the issue would be much appreciated. Thanks.

[benji_]

writing to a slow hard drive via usb you can expect low speeds. not much you can do really.

[Tabs]

Random read and write speeds are the slowest performing aspect of hard drive performance. If you want better performance, you could add them to archives (.zip, .rar, .7z) so that there are fewer, but larger files to copy.

 

That looks perfectly normal to me. 5400rpm 2.5" drive, plus the overhead of usb signalling, and the fact that there are thousands of small files - each of which needs a corresponding MFT entry produced (making worse the mechanical nature of the drive)… and yeah. Bad times.

 

 

[mancesco]

1 minute ago, benji_ said:

writing to a slow hard drive via usb you can expect low speeds. not much you can do really.

I know that, but all the way down to 1.5 MB/s seems excessive.

[whoisjaya]

This might sound kind of stupid, but are you sure it's getting enough power? (Or would it just not turn on if it didn't receive enough power?)

[benji_]

1 minute ago, mancesco said:

I know that, but all the way down to 1.5 MB/s seems excessive.

not really, its jumping around due to different file sizes and such, tabs explained it much better than i could

[Tabs]

For comparison, here is a 7200RPM desktop (3.5") hard drive performing purely random reads and writes. This is better than you can expect from a fully random workload on your disk, because no MFT seeks are used during this process. This is a 2TB Seagate ST2000DM001 drive.

 

Spoiler

 

This is the performance of an old non-NVME m.2 drive using PCI express.

 

Spoiler

 

And finally, a representative example of a good usb 3 flash drive - this is a Kingston Hyperx Savage.

 

Spoiler

 

Basically, you can expect fully random performance on your 2.5" external hard disk to be roughly the same as my first image, or possibly a little worse. It's just a weakness of spinning drives.

 

So long as you know about this limitation though, you can offset it. If you zip up your files before copying them, you'll get dramatically higher speeds since then the disk will be doing sequential, rather than random, writes. Any good archival program will allow you to extract individual files form an archive, although this isn't useful if you need metadata or search functionality.

[mancesco]

8 minutes ago, Tabs said:

For comparison, here is a 7200RPM desktop (3.5") hard drive performing purely random reads and writes. This is better than you can expect from a fully random workload on your disk, because no MFT seeks are used during this process. This is a 2TB Seagate ST2000DM001 drive.

 

  Reveal hidden contents

 

This is the performance of an old non-NVME m.2 drive using PCI express.

 

  Reveal hidden contents

 

And finally, a representative example of a good usb 3 flash drive - this is a Kingston Hyperx Savage.

 

  Reveal hidden contents

 

Basically, you can expect fully random performance on your 2.5" external hard disk to be roughly the same as my first image, or possibly a little worse. It's just a weakness of spinning drives.

 

So long as you know about this limitation though, you can offset it. If you zip up your files before copying them, you'll get dramatically higher speeds since then the disk will be doing sequential, rather than random, writes. Any good archival program will allow you to extract individual files form an archive, although this isn't useful if you need metadata or search functionality.

Alright, that's a pretty thorough explanation. Thanks for this!
Follow up question: are there any programs or file managers (like alternatives to windows explorer) that allow a simple copy-paste of several files to be done sequentially, thus avoiding the need of zipping them up (if that's at all possible, that is)?

[Tabs]

49 minutes ago, mancesco said:

Alright, that's a pretty thorough explanation. Thanks for this!
Follow up question: are there any programs or file managers (like alternatives to windows explorer) that allow a simple copy-paste of several files to be done sequentially, thus avoiding the need of zipping them up (if that's at all possible, that is)?

 

Unfortunately not, it's an intrinsic way that file systems work. For every file that is copied, an entry needs to be made in the file systems "file allocation table". Non-Windows filesystems use them as well, but they have different names and structures. For NTFS, it's called the Master File Table, or MFT.

 

The basic performance problem with the type of transfer you're doing, is that because each and every file is small and separate, each file needs corresponding MFT entries. Before a new file is copied, the MFT for the previous one needs to be made - and since the MFT is in the centre of the disk, it means that your drive is writing a file, then moving to the middle of the disk to write the MFT entry, and then moving back to copy another. Therefore, each copy involves new seeks, and extra writes, and those are all terrible for hard drive performance - hard drives like writing to one place for a long time, and they can do that kind of work well. But ask them to jump all over the place and the nature of a spinning disk becomes very apparent - it takes a relatively long time to re-seek on a hard drive - 8-15ms - so if you add 16-30ms per file on top of the time taken to actually do the writes, you can see why your average data rate is so low.

 

The reason things are done this way is that it's needed to ensure that when you go to X:\somefolder, you can quickly get a file listing of what is in X:\somefolder (it's pulled from the MFT). The disk itself doesn't care where a file physically is on the drive; only your OS needs to care about that so that when you go to X:\somefolder\somefile.jpg, the MFT points to the physical location of that file on the disk.

 

Sorry if this is a very long-winded explanation. The reason zipping up your files before copying would work, is that the filesystem doesn't need to know whats *inside* a compressed file - only where the compressed file itself is. So if you split your 32000 files into 8 compressed archives of 4000 each, you'd reduce the number of MFT entries (and costly seek/write/re-seek operations) down from 32000 to 8.

Edited August 5, 2018 by Tabs
Corrected minor typographical error

[mancesco]

Awesome, thanks!