A filesystem test

[Poet129]

For the past couple years I have been developing my own filesystem. It is far from perfect and not going to be replacing anything any time soon. This project is for fun and learning. Naturally to know if what I'm trying to do is working I have to test it. Of course said test should also be run against other filesystems for a baseline, which I have done. FS Source

 

The computer and os:

Ubuntu Linux 25.10: 6.17.0-8-generic.

SSD used Intel 180 GB M.2 (GPT, 10 GB partition at end of drive) (This drive was shared with OS, device was not used during testing)

RAM 16 GB SODIMM DDR4 2133 (8 GB x2)

CPU Intel Core i5-6300U

HP ProBook 640 G2

 

Description of test:

This test is quite simple create a file, write to it a consistent random amount between (1, 2097152) of b"\x00" bytes, write same amount but consistent random data, read back and check. All these operations timed independently. Then on to the next file and do the same, as many times as possible and fill the drive. Following that all files are read and double checked correct. Results are labeled Create, Read, Allocation, Write and Files. The first four have two sections, Total seconds and Avg time per file. Files is simply how many files have been created. There is also the possibility for errors to be reported, none were on any filesystem. The goal is to create as many files as possible without errors. Test Source

 

EXT4 4K Cluster size, formatted with command: sudo mkfs.ext4 /dev/sda3

EXT4 2M Cluster size, formatted with command: sudo mkfs.ext4 /dev/sda3 -C 2097152 -O bigalloc

NTFS 4K Cluster size, formatted with command: sudo mkfs.ntfs /dev/sda3 -Q

NTFS 2M Cluster size, formatted with command: sudo mkfs.ntfs /dev/sda3 -Q -c 2097152

FAT 4K Cluster size, formatted with command: sudo mkfs.fat /dev/sda3 -S 4096

KXCSpaceFS 4K Cluster size, formatted with command: sudo mkfs.kxcspacefs /dev/sda3 4096

KXCSpaceFS 2M Cluster size, formatted with command: sudo mkfs.kxcspacefs /dev/sda3 2097152

 

[spacepickle]

My brain is not large enough to comprehend this but that's cool man

[Biohazard777]

I'm no Python expert, but I do know you've got a leak in there. 

c = open("FSTest/" + i, "rb").read()

I don't see a close, or use of context manager.

Putting that aside, I'm not sure you are doing the testing properly...

Quote

flush() does not necessarily write the file’s data to disk.
Use flush() followed by os.fsync() to ensure this behavior.

https://python-reference.readthedocs.io/en/latest/docs/file/flush.html
And I don't see os.fsync() in your python script....

I don’t have time to do a full review (and, as I mentioned, Python isn’t really my forte),
but may I ask: why not use one of the many existing open source filesystem benchmarking tools?
Since you already have your own filesystem to experiment and work on, using a well-tested benchmark would help IMO.

[Poet129]

@Biohazard777 Thank you for your suggestion. It has been reimplemented.

 

As far as the leak was concerned I thought if the context was not saved to a variable it was not stored / leaked.

c = open(FILE, "rb").read()

Being possibly okay, but your correct why take the chance right?

Of course the following was also there...

c = open(FILE, "rb")

And was not closed, so guilty as charged either way.

 

As far as not flushing to disk rather than the fs. The fs was being tested rather then the disk itself. However it is also good to take consideration of how fast it can be written to disk by the fs.

 

As mentioned I have done this for the purpose of fun and learning. I have also tested the Windows version of this filesystem with WinFSP-Tests, Which are quite comprehensive. I will be also testing this fs with other benchmarks before any kind of release. Thanks again.

 

KXCSpaceFS 2M:

[Eigenvektor]

2 hours ago, Poet129 said:

As far as the leak was concerned I thought if the context was not saved to a variable it was not stored / leaked.

That's not how memory/resource leaks work.

 

If you reserve memory, then don't hold a reference to it and never free it, that's the leak. If you open a native resource, then don't hold a reference to it and never release it, that's the leak. So reserving it and effectively forgetting about it by not assigning it to a variable (and calling close on it), that's the leak.

[Sauron]

Python has a safe pattern for dealing with file handles; you can use the with keyword.

with open("FSTest/" + i, "rb") as file:
  if file.read() != data:
    print(f"Error on file: {i}   ")

that way you can be sure the handle will be closed when leaving the with block and you don't have to worry about doing it manually. The same of course goes for the writing section at the beginning.

 

I'm more concerned about the use of randomness. Your check relies on reseeding and rerunning the random generation in the exact same order to get matching results; this makes your code hard to read and easy to inadvertently screw up. Moreover if any of those files is missing from the filesystem you'll get errors on every file after that because you'll be comparing it to the wrong set of random bytes.

 

Save your file names in a list with their size and a checksum of their contents instead. That way you can still check for correctness without having to store or regenerate their contents on the fly and you won't risk comparing a file to the wrong contents.

 

Better yet, since you already read the file back in the first loop, just compare the contents then; this requires no checksum and no storing, as well as no reliance on random to behave as you expect.