Copying a lot of little files

[Questargon]

Hi. When you're working with node.js projects, you're left with a lot of little files (like under the folder node_modules). When you have to move the project to another drive, you see the transfer speed almost grind to a halt. I know, that this is largely because of filesystem overhead. Has anybody any experience with different filesystems and this issue? Are there filesystems out there that can handle this better than - say - NTFS and ext4?

[emosun]

i imagine its the drives running out of cache if it's a lot of data per transfer , what size are the transfers

[mariushm]

Make a 7zip file with the ultrafast / veryfast , basically no compression , copy the archive there and unpack it.

7zip can use multiple threads to pack the files in parallel and you're copying only one file and then unpacking it.

It's the opening and closing and files and allocating space for the file that takes time, and the copy also waits until the file is confirmed written to disk, so with lots of small files these add up and you get slow speeds.

 

It's not a cache thing... it's just single threaded copy.... create file / open file for writing , transfer data, close file, wait for file system to report file is written, repeat with next file...

[Questargon]

@mariushmHmmm. Okay, gonna try 7zip without compression, but I do not think that this will improve matters by a lot, since all those mini-file-operations are back the moment you unzip your archive. Anyway ... I*ll be back after I fooled around with this for a bit. In any case: Thanks for the suggestion!