Is moving a lot of large files a bad idea?

[Tog Driver]

I have a bunch of files I need to move to a different drive, would it be a bad idea to just select all of it and have it moving over night? or should I do it in small batches of maybe 100 Gb? 

any advice would be appreciated! 

[LukeSavenije]

4 minutes ago, TheClueLessMajority said:

I have a bunch of files I need to move to a different drive, would it be a bad idea to just select all of it and have it moving over night? or should I do it in small batches of maybe 100 Gb? 

any advice would be appreciated! 

You'll have some problems with when it goes wrong because of a dll or something. Just use ethernet and do it the way you feel the safest with

[aezakmi]

if the drives or cables aren't damaged and the OS is stable there's no problem on moving files

[Tog Driver]

Just now, LukeSavenije said:

You'll have some problems with when it goes wrong because of a dll or something. Just use ethernet and do it the way you feel the safest with

Well, it's to an External drive via USB. . .

is it more likely to go wrong if there are a bunch of large files?

[Tog Driver]

1 minute ago, aezakmi said:

if the drives or cables aren't damaged and the OS is stable there's no problem on moving files

So, just select the 2Tb of stuff and Cut/Paste and walk away, there isn't some weird instability or something when having a ton of files in queue at once or something?

[LukeSavenije]

Just now, TheClueLessMajority said:

Well, it's to an External drive via USB. . .

is it more likely to go wrong if there are a bunch of large files?

It's in all ways likely. Why not use ethernet to go from drive to drive? Way faster. Or does it need to be stored on the usb drive?

[Tog Driver]

Just now, LukeSavenije said:

It's in all ways likely. Why not use ethernet to go from drive to drive? Way faster. Or does it need to be stored on the usb drive?

Stored on the USB drive.

[LukeSavenije]

Then just copy paste anything you'll need. The risk is in all ways as big as it gets.

[79wjd]

1 minute ago, TheClueLessMajority said:

Well, it's to an External drive via USB. . .

is it more likely to go wrong if there are a bunch of large files?

I would personally do it in groups that way if there is an error then you don't have to restart the entire transfer (technically you could verify files and only re-attempt those that failed, but that's an extra step), but either is technically fine.

[Tog Driver]

1 minute ago, LukeSavenije said:

Then just copy paste anything you'll need. The risk is in all ways as big as it gets.

It's moving so Cut/Paste, does that make any difference?

[Tog Driver]

1 minute ago, 79wjd said:

I would personally do it in groups that way if there is an error then you don't have to restart the entire transfer (technically you could verify files and only re-attempt those that failed, but that's an extra step), but either is technically fine.

Define "verify", as in, check if all the files are still in working order?

[Lady Fitzgerald]

Windows file system has problems with large copy and paste operations. I've had excellent luck using FreeFileSync in Mirror mode to move large amounts of data. There is a hack one can do to force FreeFileSync to verify each file transfer and report any failures at the end of the task. After installing FreeFileSync:

 

1. Go to AppData  in the User folder.

2. Open the Roaming folder.

3. Open the FreeFileSync folder.

4. Click on GlobalSettings.xml

5. Look for the line VerifyCopiedFiles Enabled and change false to true.

6. Save when closing.

It may be necessary to repeat the above steps several times to get the setting to “stick”.

Be aware this hack will double the amount of time needed to run FFS.

 

FreeFileSync in Mirror mode is also excellent for backing up data, with or without the hack (I use the hack).

[Lady Fitzgerald]

5 minutes ago, TheClueLessMajority said:

It's moving so Cut/Paste, does that make any difference?

Copy and paste would be safer.

4 minutes ago, TheClueLessMajority said:

Define "verify", as in, check if all the files are still in working order?

See my previous post.

[Guest]

I'd use robocopy.

 

It's a lot more fail-proof.

 

Lot more options and do it from command line.

 

Here is a link to some features and how to use robocopy:

 

https://en.wikipedia.org/wiki/Robocopy

(pls dont ban, mods).

[79wjd]

8 minutes ago, TheClueLessMajority said:

Define "verify", as in, check if all the files are still in working order?

Technically no, I would do it by checking the hash of files before and after transfer to make sure they're the same, but checking that the files are 'working' is more or less the same (ish). It's really just not worth the effort and it's all around more straight forward to just do the transfer in groups.

[aezakmi]

14 minutes ago, TheClueLessMajority said:

So, just select the 2Tb of stuff and Cut/Paste and walk away, there isn't some weird instability or something when having a ton of files in queue at once or something?

just make sure you have all of the white cis male admin privileges otherwise some errors might appear and you don't want an error to pop up at like 3% of the transfer.

[Tog Driver]

16 minutes ago, Lady Fitzgerald said:

Windows file system has problems with large copy and paste operations. I've had excellent luck using FreeFileSync in Mirror mode to move large amounts of data. There is a hack one can do to force FreeFileSync to verify each file transfer and report any failures at the end of the task. After installing FreeFileSync:

 

1. Go to AppData  in the User folder.

2. Open the Roaming folder.

3. Open the FreeFileSync folder.

4. Click on GlobalSettings.xml

5. Look for the line VerifyCopiedFiles Enabled and change false to true.

6. Save when closing.

It may be necessary to repeat the above steps several times to get the setting to “stick”.

Be aware this hack will double the amount of time needed to run FFS.

 

FreeFileSync in Mirror mode is also excellent for backing up data, with or without the hack (I use the hack).

Thanks, I'll look into that.

[Tog Driver]

14 minutes ago, Netivity said:

I'd use robocopy.

 

It's a lot more fail-proof.

 

Lot more options and do it from command line.

 

Here is a link to some features and how to use robocopy:

 

https://en.wikipedia.org/wiki/Robocopy

(pls dont ban, mods).

Didn't know that was a thing, thanks for the information!

[Tog Driver]

13 minutes ago, 79wjd said:

Technically no, I would do it by checking the hash of files before and after transfer to make sure they're the same

Right, I don't know how to do that. . .

[Tog Driver]

11 minutes ago, aezakmi said:

just make sure you have all of the white cis male admin privileges otherwise some errors might appear and you don't want an error to pop up at like 3% of the transfer.

Would that just be using an admin account?

[corrado33]

Oh for the love of god

 

 

Just freaking copy the files overnight in one big thing. It'll be done in the morning. 

 

Can't believe we have an entire relatively large thread about copying lots of data. 

 

I've copied terrabytes of data at a time, it's fine, just do it. 

[NineEyeRon]

I recommend using Teracopy, works really well to help manage large transfers.

 

You can set it to shut down after completing, I just left it running and came down the next morning to a shut down Pc and all my files moved (after I fixed issues between keyboard and chair but that is a whole other story).

[Lady Fitzgerald]

7 hours ago, corrado33 said:

Oh for the love of god

 

 

Just freaking copy the files overnight in one big thing. It'll be done in the morning. 

 

Can't believe we have an entire relatively large thread about copying lots of data. 

 

I've copied terrabytes of data at a time, it's fine, just do it. 

Apparently, you have been extremely lucky and have not run into problems doing that like some of us here have. Either that, or something didn't copy over correctly and you just didn't catch it.

4 hours ago, NIneEyeRon said:

I recommend using Teracopy, works really well to help manage large transfers.

 

You can set it to shut down after completing, I just left it running and came down the next morning to a shut down Pc and all my files moved (after I fixed issues between keyboard and chair but that is a whole other story).

I used to use Teracopy back in my XP days and it worked great but, when I tried it in Win 7, it would bog down horribly. I found FreeFileSync to be far more reliable (not to mention versatile).

[Tog Driver]

14 hours ago, corrado33 said:

Oh for the love of god

 

 

Just freaking copy the files overnight in one big thing. It'll be done in the morning. 

 

Can't believe we have an entire relatively large thread about copying lots of data. 

 

I've copied terrabytes of data at a time, it's fine, just do it. 

Sometimes while copying, I have had the computer freeze sometimes for a few seconds and then it resets as if I just logged in, and I have to restart everything, was wondering if this and other unfortunate things would be more likely to happen while moving larger amounts of stuff.

[Tog Driver]

10 hours ago, NIneEyeRon said:

I recommend using Teracopy, works really well to help manage large transfers.

 

You can set it to shut down after completing, I just left it running and came down the next morning to a shut down Pc and all my files moved (after I fixed issues between keyboard and chair but that is a whole other story).

Ugh, I am always having problems between the keyboard and chair.