Python Recursive File Copier

[jdfthetech]

I was working on a huge data restore which left me with 500 GB of data in around 5000 directories.  I wrote this little python script to rip all the files into a single directory so I could export to a drive and find files more easily.

Not 100% sure on the win version, so YMMV.

 

anyways, here it is:

 

https://github.com/jdfthetech/recursiveCopy/

 

 

(yes I posted this over on L1 forums too in case anyone cares)

[Alex P]

dumb question, but why do you need python to do that?

 

wouldn't find make it quicker?

 

find ./* -exec cp {} /destination/ \;

[vorticalbox]

You need to put TQDM on that for loop and get a progress bar.

[jdfthetech]

13 minutes ago, Alex P said:

dumb question, but why do you need python to do that?

 

wouldn't find make it quicker?

 

find ./* -exec cp {} /destination/ \;

wouldn't this copy folders as  well?
The idea was to only copy files and recursively go through every folder only copying files

 

 

7 minutes ago, vorticalbox said:

You need to put TQDM on that for loop and get a progress bar.

it prints each file as it copies in the output.  I didn't want to use tqdm as this was quick and dirty

The nice thing is I left the github repo public, so feel free to fork it, modify it and go to town!

[Alex P]

no it would not copy folders, but it will only if you specify "-r" flag after "cp".

[Alex P]

30 minutes ago, Alex P said:

dumb question, but why do you need python to do that?

 

wouldn't find make it quicker?

 

find ./* -exec cp {} /destination/ \;

Also if you want acknowledgement, you can do:

 

find ./* -type f -exec sh -c "cp {}  ~/destination/ ; echo {} copied" \; 

 

[jdfthetech]

1 hour ago, Alex P said:

no it would not copy folders, it would only if you specify "-r" flag after "cp".

if you don't use -r then it won't recursively go into the other folders, no?

[Alex P]

Yes, it ("find") will go into the folders, though "cp" will  not be copying folders themselves, but only the files from these folders (which are the result of "find"). Recursiveness (is there such a word?) is coming form "find" in this case, not from "cp"! )  You can try, you know

It will do exactly what you wanted !

[jdfthetech]

I just tested and that does work.

It also doesn't do what I originally set out to do which was :

 

write a program that recursively goes through directories and copies files over in python

reason for python?  I have them, but glad to see you can do this in a command line as well.

[Alex P]

I tested your python script and can admit that it actually works way faster (like significantly faster) than my "find" approach. Congrats!!!!

 

me@My-HOME-PC:~$ rm -fr destination/*; time find ./test/* -type f -exec cp {}  ~/destination/ \;

real    0m2.384s
user    0m0.063s
sys     0m1.578s

when your python os.walk does:

me@My-HOME-PC:~$ rm -fr destination/*
me@My-HOME-PC:~$ ls -la destination
total 0
drwxrwxrwx 1 me me 4096 Apr 30 21:38 .
drwxr-xr-x 1 me me 4096 Apr 30 21:35 ..
me@My-HOME-PC:~$ time ./recursiveCopy.py

real    0m0.169s
user    0m0.047s
sys     0m0.125s

@My-HOME-PC:~$ ls -l destination/ | wc -l
201

One can see that "find" results in huge system overhead.

[vorticalbox]

1 hour ago, Alex P said:

One can see that "find" results in huge system overhead.

But also required no dependancies other that what's available in the terminal so there is that. 

[Alex P]

1 hour ago, vorticalbox said:

But also required no dependancies other that what's available in the terminal so there is that. 

true, but if you need to sieve through zillions small files, having python ( which any linux distro will have anyway ) is not a big price to pay. I wander if i can use this approach for speeding up nightly backups of my HPC....although there are bottlenecks of different nature, like network throughput..