Is there an easy way to move 4TB of data from a ZFS Volume to a Btrfs Volume?

[Master Disaster]

Guys, the transfer took about 6 hours in total (I decided not to transfer about 1TB and instead keep it on the USB). It really wasn't that bad.

[leadeater]

1 hour ago, Master Disaster said:

Guys, the transfer took about 6 hours in total (I decided not to transfer about 1TB and instead keep it on the USB). It really wasn't that bad.

Did you recognize the data during the transfer or just move as is? What method did you end up going with to copy the data?

[Master Disaster]

4 hours ago, leadeater said:

Did you recognize the data during the transfer or just move as is? What method did you end up going with to copy the data?

I moved everything directly using File Manager to mount the old SMB Shared Folders then drag and dropping from the old to the new. Everything was done directly on the Synology, once the transfer was started I shut my computer off and went out. Came back around 6 hours later and it was completed.

 

After it was complete I reformatted the USB then copied everything back (I want a backup as I'm using raid 0) then finally once the backup was finished I reorganised everything safe in the knowledge that if I had any issues I had a backup to recover from.

 

I was very anal about the backup because I have about a half TB of music and A LOT of it is pretty rare stuff which if I lost I couldn't ever get again.

[alpha754293]

20 hours ago, leadeater said:

So try and create 10 zip archives using a script then copy and then extract, how exactly does this make it faster. You know you can only go as fast as the HDD on the source computer right?

Actually, since the OP is using a RPi4, he might actually be CPU limited and less so HDD limited.

 

But that's why I suggested that he test it.

 

If his CPU can keep up with it, then it's a toss up between whether he's going to be line limited or HDD limited.

A lot of current HDDs has STRs > GbE speeds, even with just a single drive. (cf. https://www.tomshardware.com/reviews/wd-red-10tb-8tb-nas-hdd,5277-2.html)

 

Again, if you have small files that are dispersed in between the large files, transferring them over the TCP/IP is a vastly slower, high latency process whereby you can't EVER come close to the line rate that the interface is capable of.

 

By storing the files into an archive, to the network interface, it thinks that you're transfer the single large file which can contain all of the small files and therefore; your average transfer rate goes up significantly.

 

Again, you're all welcome to test it out for yourselves as it is quite evident that most of the people who are commenting/replying doesn't appear to have much in the way of experience with this, tape or no tape.

 

Create a GB's worth of files ranging in size from 0.1 kB to 100 kB. Under normal/"ideal" circumstances (which you CAN actually achieve), and using simple math, the amount of wall clock time that it should take for you to transfer 1 GB file over a GbE interface is 10 seconds.

 

Once you've created a GB worth of files ranging in size from 0.1 kB to 100 kB, now try copying them. If the entire process takes you > 10 seconds, then your effect transfer rate over the same GbE interface will start to plummet, e.g. if it takes you 20 seconds to copy those files, you've just gone from an effective transfer rate of 100 MB/s down to 50 MB/s.

 

In fact, the even easier way to test this would be to use iPerf and you can see what your effective transfer rate is over a range of len * num values.

 

21 hours ago, Kilrah said:

Becasuse there is zero doubt that it's not going to help and just be a waste of time.

 

BTW, you DO realise that the OP tells you a little bit about the distribution of the files, right?

 

6 hours ago, Master Disaster said:

I was very anal about the backup because I have about a half TB of music and A LOT of it is pretty rare stuff which if I lost I couldn't ever get again.

 

It's always amazing to me what people assume about the probability density function as a function of size of files that someone might have when there is literally no way that you can assume said PDF(filesize). 

 

Do it. Don't do it. IDGAF.

[Kilrah]

2 hours ago, alpha754293 said:

Create a GB's worth of files ranging in size from 0.1 kB to 100 kB. Under normal/"ideal" circumstances (which you CAN actually achieve), and using simple math, the amount of wall clock time that it should take for you to transfer 1 GB file over a GbE interface is 10 seconds.

I did for the lulz.

76000 files totalling 3.5GB.

 

Direct copy over network: 11min

Uncompressed archive, transfer, unarchive: 10min. Transfer takes 30 secs, but the drives on both ends take ages to gather and create the small files, as predicted.

 

Aka useless. And I'm on RAID0'd recent HDDs on both sides. With just a single older drive like most people will be using for a NAS the drive speed will have even more comparative impact than the network transfer, reducing the difference or tipping it the other way.

[alpha754293]

8 hours ago, Kilrah said:

I did for the lulz.

76000 files totalling 3.5GB.

 

Direct copy over network: 11min

Uncompressed archive, transfer, unarchive: 10min. Transfer takes 30 secs, but the drives on both ends take ages to gather and create the small files, as predicted.

 

Aka useless. And I'm on RAID0'd recent HDDs on both sides. With just a single older drive like most people will be using for a NAS the drive speed will have even more comparative impact than the network transfer, reducing the difference or tipping it the other way.

I tested this as well, here is the methodology that I used:

 

To create 1 GB worth of files, I calculated that for sizes ranging from 100 bytes to 100000 bytes, keeping the number of files consistent across the sizes, I would need 9665 files of each size in order to generate 1073781500 bytes worth of data. (NB: 1 GiB = 1073741824 bytes) so I'm over that by 39676 bytes.

Then I created the files for the different sizes using these commands:
 

time -p for i in `seq -w 1 9665`; do dd if=/dev/random of=100Bfile$i bs=100 count=1; done
time -p for i in `seq -w 1 9665`; do dd if=/dev/random of=1000Bfile$i bs=1000 count=1; done
time -p for i in `seq -w 1 9665`; do dd if=/dev/random of=10000Bfile$i bs=10000 count=1; done
time -p for i in `seq -w 1 9665`; do dd if=/dev/random of=100000Bfile$i bs=100000 count=1; done

The times to create the files, respectively are:

261.42 s
258.22 s
366.94 s
2546.93 s

(These files were created with an HP Z420 Workstation, Intel Xeon E5-2690 (v1, 8-core, 2.9 GHz base, max all core turbo 3.3 GHz, HTT disabled, HGST 3 TB 7200 rpm SATA 6 Gbps HDD connected to the onboard SATA ports, e.g. no HBA, 8x 16 GB (128 GB total) Samsung DDR3-1600 ECC Reg. RAM, OS drive is an Samsung 860 EVO 1 TB SATA 6 Gbps SSD, using Cygwin64, in Windows 7 Pro x64.)

 

The time it took to copy 38660 files from the workstation* to my NAS unit (Qnap TS-832X with eight HGST 6 TB 7200 rpm SATA 6 Gbps drives in RAID5) using the command:

time -p rsync -vr copytest /cygdrive/x/copytest

 

is 1443.66 s.

 

The time it took to store the files into a zip archive using 7-zip with no compression using the command:

time -p 7z a -tzip -mm=Copy copytest.zip copytest

 

is 7.80 s.

 

The time it took to rsync the zip file over to the NAS using the command:

time -p rsync -v copytest.zip /cygdrive/x/copytest

 

is 19.86 s.

 

The time it took to unpack the zip file directly on the NAS, using NAS' file manager -> extract command is 306 s. (See Powerpoint Presentation for the proof/evidence of that.)

 

*I originally tested it with the workstation in order to "simulate" the single drive NAS that the OP had to the new, 2-drive NAS that he has.

 

In testing the NAS-to-NAS transfer, both of my two NAS units (both QNAP TS-832X) have eight drives each, one has eight 6 TB drives, and the other has eight 10 TB drives, all HGST, all 7200 rpm, SATA 6 Gbps HDDs and they can see each other over CIFS. The time it took to run rsync from NAS to NAS using the command:

time -p rsync -rv copytest /share/share/copytest

 

is 614.86 s.

 

Total time: 333.66 s vs. 614.86 s for a NAS-to-NAS direct copy.

 

I've dumped the screenshots of each stage/step into a Powerpoint Presentation.

 

I'm repeating the exercise whereby I am varying the number of files, but keeping the number of files * size of files to be equal between the different file sizes, e.g. 1 GiB / 4. This means that I will need 2685000 100 B files, 268500 1000 B files, 26850 10000 B files, and 2685 100000 B files, for a total of 1074000000 B which means that I am going to be over 1 GiB (technically) by 258176 B.

Generating a grand total of 2983035 files takes a while, even with an 8-core processor, so I'll finish the rest of the test probably tomorrow.

 

 

Presentation1.pptx

[alpha754293]

So....apparently, trying to create 2.9 million 100 byte files is too slow/just takes too damn long (only 533,000-ish files have been created since two-and-a-half days ago), but I think that you get the picture. (I mean, I can still run it with what I've got now as far as number of files and size distribution...)