Make Cache great again?!

[FancyPants101]

I was running into this fun phenomenon where if I copied a large game folder then stopped it part way and restarted it it would go into burst mode seemingly. It would get about 20gb's of a very fast speed(more in the best case scenario.). Not sure what makes this happen though, but if it's not a pure saturation of the cache and it's from the cache it can only get faster. My normal file operations without this are all 60-135 mb/s.

 

I have a 256 smr drive. It caps this burst mod at around 600mb/s

I have a 6 disk raid with all 64mb caches. It caps out at around 900mb/s

 

If you do the math that is about 150mb/s per drive or per 64mb of cache. Can this be maximized to work all the time. Or Can it be made to atleast provide bust for 20gb of files. Mind you these are game folders like warthunder and not contiguous nice folders. At it's best it would start at around 600mb/s on the srm and drop to around 400mb/s by time it finished. The raid had similar performance at it's best. But the outcome was always random.

 

Not sure what about linux copy jobs did this. It would be awesome if it could be made default. BTW, when it did it's best from 600mb/s to 400mb/s It would get this for the entire folder size. Which was susually 33-50gb/s of size. So it could get very high sustained speeds.

 

IDEA#2

 

My other idea is a software solution to go with this.

 

Can software be made into unique data chunks. Then have those unique data chunks mapped. This would have to made by the program writers as it would be based on unique actual coding and probably made in a way for the computer to organize. These unique code chunks could be mapped to show how many of them there are and in what order. The mapping could be loaded into ram and the data chunks stored as unique chunks in the cache. This could be done differently for different hardware potentially with different tools. But since most things are 64mb's it's probably not that varied. The point is the chunks and the map can be stored with less overall disk space and optimized for both rust disks and possibly higher speed disks for even faster retrieval. The map is then stored in ram and the disks held in cache and pulled into ram as needed. The CPU can handle the rest. If it's loaded in HDD cache can it in essence speed up the machine by keeping all data in the ram and HDD cache and minimal read/write data? It could work on the idea of acceptable volatility. Especially for things like games that are over a network where the server can save the relative data and be checked later for consistency on the local machine.

 

If I'm write this could massively shrink the size of HDD storage per software. And Possibly ram also. Could you efficiently do the same in the ram with the map and data chunks without effecting access times? The map could store data then put it in order if needed. Or mix supplying the cpu cache or gpu with data in the fastest possible way. this could also store multiple version for different machines or applications. Maybe preexisting optimal code for interfacing that never needs to be changed as the code does. Assuming that is possible Or version of the data(different chunks with different sizes of the same data) possible optimized for different uses or parts of the application. Say, physics vs other parts of the code. Redundancy could be used in ram in order to run things more optimally. Assuming you can do it without doubling the HDD cache. This could be done by having all bigger chunks made of smaller chunks. They could then be dealt with in the map so parts of it can be called at different times for different situations. Maybe optimized via a calling program in case calling a larger chunk is better than referencing it in smaller parts.

 

I'm assuming this could also be used to make mixed sized items write to disks similarly to large files.

[Electronics Wizardy]

3 minutes ago, FancyPants101 said:

I have a 256 smr drive. It caps this burst mod at around 600mb/s

I have a 6 disk raid with all 64mb caches. It caps out at around 900mb/s

This is likely the system ram cache you seeing, not the cache in the disks themselves.

 

You can probably change the timeout to force linux to hold things in its ram cache longer, but you don't normally want to incase of system or power faiure

 

4 minutes ago, FancyPants101 said:

 

Can software be made into unique data chunks. Then have those unique data chunks mapped. This would have to made by the program writers as it would be based on unique actual coding and probably made in a way for the computer to organize. These unique code chunks could be mapped to show how many of them there are and in what order. The mapping could be loaded into ram and the data chunks stored as unique chunks in the cache. This could be done differently for different hardware potentially with different tools. But since most things are 64mb's it's probably not that varied. The point is the chunks and the map can be stored with less overall disk space and optimized for both rust disks and possibly higher speed disks for even faster retrieval. The map is then stored in ram and the disks held in cache and pulled into ram as needed. The CPU can handle the rest. If it's loaded in HDD cache can it in essence speed up the machine by keeping all data in the ram and HDD cache and minimal read/write data? It could work on the idea of acceptable volatility. Especially for things like games that are over a network where the server can save the relative data and be checked later for consistency on the local machine.

 

The issue with this is the drive manages its cache, not the software in the os. So the os has no control over the disk cache.

[FancyPants101]

Could they make them with system controlable ones? Or have loadable program areas per software that allow temporary use. Probably in a way that doesn't allow destructive things loaded easily. Maybe temp loaded like you can do with cmos but faster and volatiles like ram so it's loaded and then gone afterwords.

 

I was playing around with something, but I could not get it to do it on the first try of a copy job. It only worked after starting and stopping. And it was hard to do well then. I also tried making it use more ram, which I think got better results. but I don't remember.

[Electronics Wizardy]

Just now, FancyPants101 said:

Could they make them with system controlable ones? Or have loadable program areas per software that allow temporary use. Probably in a way that doesn't allow destructive things loaded easily. Maybe temp loaded like you can do with cmos but faster and volatiles like ram so it's loaded and then gone afterwords.

If you want a volitale write cache, why not just use system ram? You normally have a lot of it already.

 

From what Ive seen the cache is also used heavily already for internal drive use, for things like shuffling data around on SMR drives. Also SSDs normally already have the dram filled with the ftl.

[FancyPants101]

I was trying to use ram. But I couldn't get it to work that way normally.

 

I was playing with this:

 

sudo sysctl -a | grep dirty

vm.dirty_background_bytes = 68719476736
vm.dirty_background_ratio = 0
vm.dirty_bytes = 68719476736
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 0
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200

 

I could not obtain default behavior though. And linux has this weird thing about only using 1gb of data in ram. I found a way to increase this, but I forget how. It might not have been these settings. But I might have been background bytes and ration set to a manual number. I forget though.

[Kilrah]

If you want a reliable cache then you'd want to add an SSD as cache for your volumes. 

 

SMR drives do their own thing so performance can be completely random at any given time.

[Electronics Wizardy]

12 minutes ago, FancyPants101 said:

I was trying to use ram. But I couldn't get it to work that way normally.

 

I was playing with this:

 

sudo sysctl -a | grep dirty

vm.dirty_background_bytes = 68719476736
vm.dirty_background_ratio = 0
vm.dirty_bytes = 68719476736
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 0
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200

 

I could not obtain default behavior though. And linux has this weird thing about only using 1gb of data in ram. I found a way to increase this, but I forget how. It might not have been these settings. But I might have been background bytes and ration set to a manual number. I forget though.

Let me play with these settings, I forget which ones let you have a gain write cache.

 

But Id probably suggest not having a huge ram write cache as the chance of something going wrong goes up. If you want a huge write cache, Id just get a ssd for caching.

[LogicalDrm]

-> Moved to Storage Devices