FreeNAS: Why are my files taking up so much space???

[grimreeper132]

This is a pretty simple problem to explain, I have a NAS running the FreeNAS OS, and when ever I put a file on it, it uses a large space on the disc, two examples below. How do I fix it so 24.3GB doesn't take up 117GB of space or 475GB taking up 2.13TB cause it means that my size is effectively quartered to fifthed (depending). It was much worse at one point, as it was 400-500GB taking up about 7.5TB before, which was fixed by changing the compression ratio, by changing it to gzip-9 which videos and music were transferred under. I have also turned compression off for Games which also has this problem. I also changed allocation size to 2K (from 512 (I think/imagine bytes)). How do I fix this as having to use a large amount of space for small files makes the NAS almost pointless. Any questions just ask.

 

[orangecat]

If I'm not mistaken doesn't freeNAS have file redundancy built in? Could be making backups automatically.

[grimreeper132]

1 minute ago, Kilobytez95 said:

If I'm not mistaken doesn't freeNAS have file redundancy built in? Could be making backups automatically.

then
A) why isn't it a constant whole number for the diffrence, and not somehwere between 4.4 and 5.4

B) 3-4 copies of the same file, that doesn't sound right to me, redundant files are normally if they exist only 1 other copy

C) why did changing the compression type change the amount of space everything takes up so dramatically

[grimreeper132]

1 minute ago, Kilobytez95 said:

If I'm not mistaken doesn't freeNAS have file redundancy built in? Could be making backups automatically.

then
A) why isn't it a constant whole number for the diffrence, and not somehwere between 4.4 and 5.4

B) 3-4 copies of the same file, that doesn't sound right to me, redundant files are normally if they exist only 1 other copy

C) why did changing the compression type change the amount of space everything takes up so dramatically

[jde3]

FreeNAS doesn't do that, you have something else going on.

 

Go on the server and do a du -h in the folder.

 

[grimreeper132]

2 minutes ago, jde3 said:

FreeNAS doesn't do that, you have something else going on.

 

Go on the server and do a du -h in the folder.

 

Was this what you were meaning???

[jde3]

Yes, but thats not the right folder. I don't want to see your files but that will show you the files and what is being used.

 

ZFS should allocate LESS space on disk than filesize but that isn't reported to Windows. Are you using shadowcopy?

[jde3]

You can do this also to look at your snapshot size

zfs get all | grep usedbysnapshots

 

But... Windows won't be able to see your snapshots so it wouldn't report that as used space. Something in Windows, some sort of archiving or cloning is doing that.

 

Your referenced space should be less than your allocated space.

[orangecat]

11 minutes ago, grimreeper132 said:

then
A) why isn't it a constant whole number for the diffrence, and not somehwere between 4.4 and 5.4

B) 3-4 copies of the same file, that doesn't sound right to me, redundant files are normally if they exist only 1 other copy

C) why did changing the compression type change the amount of space everything takes up so dramatically

Check your settings maybe you have something mis-configured. Just because you have file desundancy doesn't mean the sace usage will be exactly 2x for example.

[grimreeper132]

2 minutes ago, jde3 said:

You can do this also to look at your snapshot size

zfs get all | grep usedbysnapshots

 

But... Windows won't be able to see your snapshots so it wouldn't report that as used space. Something in Windows, some sort of archiving or cloning is doing that.

 

Your referenced space should be less than your allocated space.

Please note that I have been using FreeNAS for only a day or two at the moment so yea no idea how to do any of that

[jde3]

17 minutes ago, grimreeper132 said:

then
A) why isn't it a constant whole number for the diffrence, and not somehwere between 4.4 and 5.4

B) 3-4 copies of the same file, that doesn't sound right to me, redundant files are normally if they exist only 1 other copy

C) why did changing the compression type change the amount of space everything takes up so dramatically

Changing the compression of the pool doesn't do anything for files already in the pool. Only for NEW files created after. You have to move the files and rewite them to change their own compress value. (just use LZ4.. seriously.. it's good)

[jde3]

2 minutes ago, grimreeper132 said:

Please note that I have been using FreeNAS for only a day or two at the moment so yea no idea how to do any of that

Ok. Let's figure this out..

 

Do you have a file explorer that can show you a size graph? Something like Baobab on Linux?

[grimreeper132]

1 minute ago, jde3 said:

Changing the compression of the pool doesn't do anything for files already in the pool. Only for NEW files created after. You have to move the files and rewite them to change their own compress value. (use use LZ4.. seriously.. it's good)

I know I transferred everything between the first and second one, and between the second and third I did transfer a new file across (Games) which has the same  issue

[jde3]

I'm going to take a guess and say you have shadow copy on.

[grimreeper132]

2 minutes ago, jde3 said:

I'm going to take a guess and say you have shadow copy on.

possibly, where do I check/disable???

[jde3]

5 minutes ago, grimreeper132 said:

I know I transferred everything between the first and second one, and between the second and third I did transfer a new file across (Games) which has the same  issue

This isn't your problem. But LZ4 is special in that it won't try to compress files that don't compress. Meaning it's great for general purpose.

[jde3]

Got me.. Who knows.. some windows bullshit feature.

 

https://www.techrepublic.com/blog/windows-and-office/how-do-i-configure-and-use-shadow-copy-in-microsoft-windows/

 

You won't really need to use that with ZFS snapshots.. if you want this feature use one.

[grimreeper132]

4 minutes ago, jde3 said:

Got me.. Who knows.. some windows bullshit feature.

 

https://www.techrepublic.com/blog/windows-and-office/how-do-i-configure-and-use-shadow-copy-in-microsoft-windows/

 

You won't really need to use that with ZFS snapshots.. if you want this feature use one.

I doubt that I have it on as you need to enable that which I have not done

[jde3]

So.. anyhow you got a couple options there.

 

You can use a file size explorer like the one I showed there to look at the files

You can do this from the command line but you'll have to sort the text output to find whats eating the space..

Maybe like du /folder -h | sort

(care with that not sure if that will do what we want, it's safe but the output my be large and slow)

Check your ZFS snapshot size. (if your using them)

zfs get all | grep usedbysnapshots

 

That's all I can think of off hand. I find it easier to use the command line to figure this stuff out but there are other ways.

[jde3]

If you have a snapshot referencing a file. And you delete that file ZFS still keeps that data.. so if you delete it and re-compress it ZFS will keep two copies if there is a snapshot for the old one.

 

The solution is just wipe the snapshots out then the deleted files will return to the free space pool.

[grimreeper132]

10 minutes ago, jde3 said:

So.. anyhow you got a couple options there.

 

You can use a file size explorer like the one I showed there to look at the files

You can do this from the command line like I showed you but you'll have to sort the text output to find whats eating the space.

Check your ZFS snapshot size. (if your using them)

 

That's all I can think of off hand.

Ok I used spacesniffer to get this. I haven't set up snapshots far as I know either

 

 

[jde3]

Ok well you know what directories to look in now. Will that program show you the actual files, or just directories?

 

I don't know... you have something really suspect and wrong here.

Maybe... you have a really weird block size? That is the old reason this sort of thing happened. It's called Filesystem Slack but pretty much everything uses 4k blocks now. That is tuneable on ZFS when you create the pool.. but it's an advanced option.

[grimreeper132]

2 minutes ago, jde3 said:

Ok well you know what directories to look in now. Will that program show you the actual files, or just directories?

 

both, if it's blue its a file if it's orange it's a folder. And it says the total amount of used space for Shared is 3.5GB but doesn't state it's size on disk, but properties says that's 19.4GBish, and the FreeNAS says it is 16.4GiB or 17.6GB.

[jde3]

13 minutes ago, grimreeper132 said:

both, if it's blue its a file if it's orange it's a folder. And it says the total amount of used space for Shared is 3.5GB but doesn't state it's size on disk, but properties says that's 19.4GBish, and the FreeNAS says it is 16.4GiB or 17.6GB.

Ok verify what is written on the server is the same that Windows reports.

 

go to the folder. cd /folder/path

then look at the files ls -lah

and you can du -h in that folder to get it's disk usage.

 

This long command here will show you the difference in size between what ZFS's file size is vs what is referenced (size on disk).

This is per your datasets in ZFS.

zfs get all | grep " used " && zfs get all | grep " referenced"

No matter what windows says.. that is what is actually there. ZFS sometimes... umm.. works differently on "free space" because it is a pooled storage so it doss not have a fixed number for that.. and some programs don't understand that.

 

Keep in mind, it's supposed to lie. the difference between them is called compressratio

 

Lastly you can just do zfs list

To see what is actually reported... if thats right you can ignore what windows reports.

[grimreeper132]

9 minutes ago, jde3 said:

Ok verify what is written on the server is the same that Windows reports.

 

go to the folder. cd /folder/path

then look at the files ls -lah

and you can du -h in that folder to get it's disk usage.

 

This long command here will show you the difference in size between what ZFS's file size is vs what is referenced (size on disk).

This is per your datasets in ZFS.

zfs get all | grep " used " && zfs get all | grep " referenced"

 

Keep in mind, it's supposed to lie. the difference between them is called compressratio

ls -lah output

 

long zfs command output