Speed Up Large File Count Transfers?

[The Old Myron Siren]

I have been backing up some resources for a Minecraft server and one of the folders has around six million files in it (Dynmap). It's full of small rendered images of the world that are then displayed in an online map. I am running it on a 500GB HDD I pulled from some of my mothers point of sale systems when she got a new system, so I know it's not exactly that greatest drive. Needless to say, however, the transfer speeds are absolutely God awful. I don't have a lot of understanding on how hard drives work, but I think its something along the lines of it need to first open, transfer, close; in that order. Thats where this bottleneck comes from, correct? Is there anyone to speed up this process or am I just at the mercy of the law of hard drives with this one?

[Levent]

Harddrives are consists of mechanical read/write heads that move around the platter, more files and more fragmented they are slower your transfers will be. So yeah, you can either archive your server with winrar/zip/7z and have relatively faster transfer speeds BUT then again you would have to wait awful a lot for that archiving process to complete.

SSD should help with your problem.

[The Old Myron Siren]

Just now, Levent said:

Harddrives are consists of mechanical read/write heads that move around the platter, more files and more fragmented they are slower your transfers will be. So yeah, you can either archive your server with winrar/zip/7z and have relatively faster transfer speeds BUT then again you would have to wait awful a lot for that archiving process to complete.

SSD should help with your problem.

Would adding a second 500GB drive and then running just a basic RAID 0 do anything for me? These drives wouldn't have anything exactly major on them and I could look into setting up a weekly archiving of them to store onto another drive?

[Levent]

Just now, The Old Myron Siren said:

Would adding a second 500GB drive and then running just a basic RAID 0 do anything for me? These drives wouldn't have anything exactly major on them and I could look into setting up a weekly archiving of them to store onto another drive?

RAID0 will increase your transfer speeds but honestly, if your server is relatively small (like less than 120Gs) I would just get a ssd.

[The Old Myron Siren]

1 minute ago, Levent said:

RAID0 will increase your transfer speeds but honestly, if your server is relatively small (like less than 120Gs) I would just get a ssd.

I do believe the actual file size of the entire folder is pretty low, so that actually would be a great idea. I can use these two 500GB drives to store temporary world backups as those are actually getting quite large.

[Mr.Humble]

RAID0 will format the current drive

[The Old Myron Siren]

Just now, Mr.Humble said:

RAID0 will format the current drive

Thats okay, im currently off loading the files from the one drive to another in preparation of installing the second drive!

[mariushm]

28 minutes ago, The Old Myron Siren said:

I have been backing up some resources for a Minecraft server and one of the folders has around six million files in it (Dynmap). It's full of small rendered images of the world that are then displayed in an online map. I am running it on a 500GB HDD I pulled from some of my mothers point of sale systems when she got a new system, so I know it's not exactly that greatest drive. Needless to say, however, the transfer speeds are absolutely God awful. I don't have a lot of understanding on how hard drives work, but I think its something along the lines of it need to first open, transfer, close; in that order. Thats where this bottleneck comes from, correct? Is there anyone to speed up this process or am I just at the mercy of the law of hard drives with this one?

Set up a FTP server on the minecraft server computer. If you run Windows on that PC, Filezilla FTP server is super easy to install and set up ... it literally takes minutes to get it set up.

 

I posted a video tutorial here:

 

 

Filezilla FTP Client can transfer with up to 10 connections from a server. The 10 number is an artificial limit imposed by the developer. But you can go around that by starting multiple instances of Filezilla FTP client and have each instance transfer a different folder (basically you don't want two instances of the client transfer same folder)...  or just use another ftp client. 

 

Another option is to compress the contents before transferring over, if you have the disk space. For example use 7zip with compression level : fastest  and set the number of threads to the highest available (should be about twice the number of cores you have)... on a modern cpu the "fastest" profile gets you over 30-50 MB/s compression speed.

 

Also may want to disable the antivirus if you have any, sometimes antiviruses scan every file not just executables and dll files.

 

Another option I often use is to create a ram hard drive, using ImDisk ram disk , and copy the files into ram and then dealing with them from ram drive... saves with the seek times and constant updating of last access times on files. In your case, it won't help as the files are already on disk.

It can help for example if you have a tool that generates those images... you can configure the tool to generate the images to ram drive which would speed up the process a lot. Then you would copy the files from ram to disk.

 

[mariushm]

don't bother with a second 500 GB drive... if the files are small you won't benefit from raid

With RAID0, the contents of the files is duplicated across both drives and the idea is that one drive reads a chunk of a file and the other drive reads another chunk of the file and the controller combines the chunks ... ex drive1 reads 512 bytes, drive2 reads 512 bytes and you get 1 KB.

For small files, the longest time is taken by the seeking of data on the drive, moving the read heads where the image is .. with raid you don't reduce that, you now have two drives that need to position their read heads so it takes same amount of time. You'd benefit from RAID 0 if your files are relatively large, like let's say 4-10 MB at the minimum.

 

If those maps are used a lot, maybe consider setting up a ram drive.. then create a batch or something that would copy the folder from the disk to your ram drive when the server is started, and have the maps served from ram. also, don't forget to save the ram to disk when shutting down server.

I think imDisk I mentioned above can do that in an automated way, it can save the ram drive to disk as a disk image.

 

But for the minimum stress, I'd say just get a 64-120 GB ssd...