Which language is faster for file manipulation?

[Neftex]

Hi, id like to make my own program for file searching and comparison. What im thinking of doing is: find all files on system -> check for files with same names -> hash them and compare

 

basically i want to deduplicate files based on name/hash and manually inspect those with different hash

 

Right now im deciding between C# or Java as those are the ones im most comfortable with - which one would be faster?

[r4tch3t]

Assembly would be the fastest. Then c# which is translated almost straight into assembly (simplifying). Java runs on a virtual machine (sort of) so can be much slower, but recent additions let you code more specifically for a platform.

For something like sorting files a c variant would probably be best.

[Dat Guy]

Assembly is not faster than C regarding file manipulation, since you'll basically use the same OS methods in both languages. C# and Java are notably slower due to their levels of abstraction.

[Klemmbrett]

44 minutes ago, r4tch3t said:

For something like sorting files a c variant would probably be best.

I would agree C or maybe C++. Java and C# have both too much overhead so both of them are a bad choice. 

[Mira Yurizaki]

While I haven't looked into the Java side of things, I poked around on the interwebs regarding C#. And at the moment I feel it's disingenuous to claim C# is horribly slower than C/C++:

 

https://stackoverflow.com/questions/686483/c-sharp-vs-c-big-performance-difference/686617#686617

http://journal.stuffwithstuff.com/2009/01/03/debunking-c-vs-c-performance/

https://www.codeproject.com/Tips/860631/Native-Cplusplus-and-Csharp-Which-is-the-Fastest-P

https://www.linkedin.com/pulse/c-faster-than-depends-coding-idom-joe-ellsworth/

 

The short of it is, language doesn't have a speed, its implementation does.

 

My take for the OP: Write the application in a language you're comfortable with and get the application out first. If you've designed the application well and written the code in a manner you can pick up easily after three months of not looking at it, whenever you're not satisfied with the performance and believe there are greener pastures, it'll be quicker to move over.

[Unimportant]

2 hours ago, Neftex said:

hash them and compare

Just to clarify: hash the filenames or the contents ? It makes a significant difference. In the latter case the hashing will be the bulk of the work, not the file manipulation.

[Neftex]

1 hour ago, Unimportant said:

Just to clarify: hash the filenames or the contents ? It makes a significant difference. In the latter case the hashing will be the bulk of the work, not the file manipulation.

hashing filenames wouldnt really make sense, im going to hash the contents

[Blade of Grass]

The differences in speed between languages will be less than the differences in speed between algorithms. Pick a good algorithm, and then go from there.

[reniat]

12 hours ago, Blade of Grass said:

The differences in speed between languages will be less than the differences in speed between algorithms. Pick a good algorithm, and then go from there.

Not to mention that the primary bottleneck will be read times anyway. different languages may have more or less overhead, but reading all files on a system is going to take a ton of disk reading which is not fast regardless of what language is trying to read it. The only difference might be how fast a given language processes a file once its loaded in memory, and if you are doing some crazy processing on it there then maybe language will matter, but I'm willing to bet that the performance here is very much bottlenecked on disk reads.

 

(edit: this does depend on the size of the files though. If you are only searching for a specific file type, and there aren't that many and they aren't that big then loading into memory won't be as big of a deal).

[Neftex]

@reniat afaik theres some kind of file table in the filesystem, so i thought of reading that to get all files (not sure if it contains filenames) so that shouldnt be too slow. slowest things will probably be searching for same filename and hashing

 

thinking about it, i guess i should be looking for language with fastest hashing and string comparison

[Mira Yurizaki]

30 minutes ago, Neftex said:

thinking about it, i guess i should be looking for language with fastest hashing and string comparison

I don't think that's necessary, considering hashing is a well-defined algorithm (also make sure you know which hashing algorithm is being used) and string compares tend to be as basic as possible. So they're likely well optimized enough that between them there's no practical difference.

 

But again, I would suggest just getting the application written first in a language you're most comfortable. If you have issues regarding performance, then we can start looking at other avenues.