how to turn a folder into a single file (instantly)?

[asheenlevrai]

Hey

 

I have a 285GB folder containing a whole bunch of files that I need to move over the network.

For technical reasons, I need to turn this into a single file (like an archive file).

I know that compressing this folder into a zip/7z/rar/whatever archive would take a lot of time and I don't necessarily need to decrease the size (this is not crucial).

 

I assumed (what a fool) that making an archive file from this folder with 0% compression using 7zip would be instantaneous or at least super quick. However it seems to take somewhere between 10-30 minutes and uses very little CPU (although I specified to use all cores, it doesn't seem to do so). Since this is a kind of task that I will have to do more than once, I wonder if there is a faster option, ideally something as quick as renaming my folder archive.zip or something magical like that.

 

I'm on Windows

 

Please let me know what you would recommend.

 

Thanks a lot for your help.

Best,

-a-

[Radium_Angel]

I am unaware of a way to instantly turn a folder into a zip file.

 

[PDifolco]

7 minutes ago, Radium_Angel said:

I am unaware of a way to instantly turn a folder into a zip file.

 

Install 7Zip, then select all files in Explorer, rclick, "7zip" in menu, then "Add to" 

 

[Sauron]

55 minutes ago, asheenlevrai said:

I assumed (what a fool) that making an archive file from this folder with 0% compression using 7zip would be instantaneous or at least super quick. However it seems to take somewhere between 10-30 minutes and uses very little CPU (although I specified to use all cores, it doesn't seem to do so). Since this is a kind of task that I will have to do more than once, I wonder if there is a faster option, ideally something as quick as renaming my folder archive.zip or something magical like that.

This is almost certainly due to your drive speed, at a minimum the program needs to read all your data and write it to the archive. For about 300gb read + 300gb write that's 20 minutes assuming constant peak sata 3 speed. Depending on your drive, the number and size of your files and the data bus you're using it may take less or more but you're still looking at a few minutes even with the fastest SSDs on the market.

[Neroon]

How to Use 7-Zip to Compress and Uncompress Files (hostinger.com)

Apparently, you can go fully uncompressed with 7-Zip, not sure if there 'storage' option is uncompressed, or just very light compression. 

[Avocado Diaboli]

1 hour ago, asheenlevrai said:

For technical reasons

Tell us those technical reasons. Because I have a hunch that this is the actual problem you should look at solving, not turning a folder into a file for a single transfer.

[asheenlevrai]

On 6/7/2023 at 4:42 PM, Sauron said:

This is almost certainly due to your drive speed, at a minimum the program needs to read all your data and write it to the archive. For about 300gb read + 300gb write that's 20 minutes assuming constant peak sata 3 speed. Depending on your drive, the number and size of your files and the data bus you're using it may take less or more but you're still looking at a few minutes even with the fastest SSDs on the market.

I'm running a PCIe3.0 M.2 SSD

 

On 6/7/2023 at 4:43 PM, Neroon said:

How to Use 7-Zip to Compress and Uncompress Files (hostinger.com)

Apparently, you can go fully uncompressed with 7-Zip, not sure if there 'storage' option is uncompressed, or just very light compression. 

AFAIU "storage" should be uncompressed. That's what I used in the settings

 

On 6/7/2023 at 4:45 PM, Avocado Diaboli said:

Tell us those technical reasons. Because I have a hunch that this is the actual problem you should look at solving, not turning a folder into a file for a single transfer.

Those data will be stored on tape. They ask that we provide archives and not folders.

[Electronics Wizardy]

4 minutes ago, asheenlevrai said:

I'm running a PCIe3.0 M.2 SSD

What disk usage do you see in task manager when doing this? Your probably disk limited, maybe writing to a different drive would help.

[Sauron]

27 minutes ago, asheenlevrai said:

I'm running a PCIe3.0 M.2 SSD

assuming an average of 1GB/s (which is generous for accessing a large number of files with varying sizes, even the fastest SSDs only get roughly 200mb/s in random writes) that's still 10 minutes overall, plus the transfer to RAM for processing by 7zip.

[asheenlevrai]

On 6/7/2023 at 5:04 PM, Electronics Wizardy said:

What disk usage do you see in task manager when doing this? Your probably disk limited, maybe writing to a different drive would help.

yup!

85-99% disk usage. Thanks...

 

On 6/7/2023 at 5:37 PM, Sauron said:

assuming an average of 1GB/s (which is generous for accessing a large number of files with varying sizes, even the fastest SSDs only get roughly 200mb/s in random writes) that's still 10 minutes overall, plus the transfer to RAM for processing by 7zip.

yup...

 

7zip reports >750MB/s at the beginning (I guess when using the SLC cache or something ) but then it decreases progressively (I guess 7zip displays a mean) and settles around 200MB/s

 

 

On 6/7/2023 at 5:54 PM, asheenlevrai said:

yup!

85-99% disk usage. Thanks...

I'll try on a machine with 2 SSDs

[Radium_Angel]

1 hour ago, PDifolco said:

Install 7Zip, then select all files in Explorer, rclick, "7zip" in menu, then "Add to" 

Right, but the OP wanted "instantly" which, AFAIK, can't be done

[PDifolco]

11 minutes ago, Radium_Angel said:

Right, but the OP wanted "instantly" which, AFAIK, can't be done

On very fast NVme it's pretty quick  

But instantly don't exist IRL

[asheenlevrai]

18 minutes ago, asheenlevrai said:

I'll try on a machine with 2 SSDs

with a 2nd SSD I still get 85% usage on the source SSD while I get 5-50% (jagged) usage on the target SSD.

7zip reports a higher speed (~950MB/s) and it seems stable.

The whole process now takes a little over 5 minutes.

So... much improvements! Thanks @Electronics Wizardy

[da na]

1 hour ago, asheenlevrai said:

Those data will be stored on tape. They ask that we provide archives and not folders.

if you're storing data on tape why not compress it...?

[mariushm]

Right click on folder, add to 7z archive,

then select  lzma2 (which can do lots of threads),

select fastest compression (because store would use a single core),

select your number of threads (up to 2 per core) and

select a reasonable block size  (I would suggest 128-256 MB but you can go to a bigger size, like 1 GB)

 

7zip will then start to compress your stuff using a super basic compression almost transparent, but will do it using 16 threads or as many you have configured, so it's like 16 copies of 7zip working on 16 separate chunks of those 256 GB of data you have. Being fastest compression, it will only be limited by the drive speeds and IO.

 

 

 

 

anyway, if you can, another option you have is to set up filezilla ftp server on one computer, and filezilla ftp client on the other - doesn't matter which one is the server.

set the ftp client to use 10 simultaneous transfers and set it to treat all file types as binary in options, and you can start transferring the folder with 10 simultaneous transfers.  You could also launch multiple copies of the filezilla ftp client and each will get its own 10 transfers, but you'll have to give each copy of filezilla ftp client to transfer a different subfolder of that 256 GB folder.

Even with folders that had thousands of small files, with 10 simultaneous transfers I could still do 40-50 MB/s

 

(solid block size matters when you want to extract a particular file from the archive, if multiple files are compressed into a block the decompressor has to decompress from the start of a block until it reaches the file contents and then extract the file. Bigger blocks increase compression efficiency so if you want high compression and use big dictionaries you want blocks of 1-4 GB, but NOT when you set the compression level to fastest and only give 7zip a dictionary of 64 KB (like basic-est zip level compression), there's very little benefit for using big blocks.

 

You can select non-solid for block size, but then it's like a zip file, each file its own block, it's better to have small solid blocks instead of no blocks.

 

You could also try and see if bzip2 compression method with 100 KB "dictionary" is fastest than lzma2, I suspect it won't be. ppmd for sure won't be, and that's single threaded, and lzma works with fewer cores and can give higher compression than lzma but it's a bit slower than lzma2 and you don't really care about super compression.

 

[dilpickle]

5 hours ago, asheenlevrai said:

I assumed (what a fool) that making an archive file from this folder with 0% compression using 7zip would be instantaneous or at least super quick.

It can't be instantaneous because it still has to copy all the files. You're limited by the speed of your drive. And obviously using a second drive would make it much faster.

[Needfuldoer]

4 hours ago, asheenlevrai said:

Those data will be stored on tape. They ask that we provide archives and not folders.

That's what .tar files are for. (Tape ARchive.) There's no getting around your drive's read/write speed, but .tar shouldn't have any compression.

 

2 hours ago, da na said:

if you're storing data on tape why not compress it...?

Tape formats like LTO have transparent file compression built into the hardware, which gets you up to 2:1 or 2.5:1 compression depending on generation. That's why an LTO9 cartridge is marketed as 18 TB raw, 45 TB compressed. (Of course that's under ideal circumstances, storing something highly compressible like a text database.)

 

https://www.ibm.com/docs/en/ts4500-tape-library?topic=cartridges-capacity-supported-lto-tape

 

Compressing the files before they get written to tape ties them to one specific archive format, and means you have to wait for your CPU to do work the drive can do by itself.

[asheenlevrai]

16 hours ago, da na said:

if you're storing data on tape why not compress it...?

because of how much time it takes to compress TBs of data

[Mark Kaine]

well, i understand the time concern but you gotta deal with it taking "some" time, thats just part of the task...

 

Above are already lots of suggestions,  id say just try it out which takes the least amount of time because i have a hunch this will also depend on the actual file (types)