Files are removed from Truenas pool but free space changes are not reflected until a restart.

[Levent]

I am not sure how much information I can provide here but gist of it is that I have an application that downloads to a mirrored 640GB pool in TrueNAS. Once downloads are complete, another application copies them out of there to another pool and removes source files.

 

Here is what happens when I ncdu.

Any ideas? Scrubbing doesnt change anything.

[Needfuldoer]

33 minutes ago, Levent said:

Any ideas? Scrubbing doesnt change anything.

Is that share inheriting a snapshot policy?

[Levent]

1 hour ago, Needfuldoer said:

Is that share inheriting a snapshot policy?

There are no snapshot policies on this pool (or the target pool) anywhere.

 

This is my ZFS get all Array-640G result, if its anything to go by.

Spoiler

 

NAME        PROPERTY              VALUE                  SOURCE
Array-640G  type                  filesystem             -
Array-640G  creation              Sat Nov 11 16:54 2023  -
Array-640G  used                  219G                   -
Array-640G  available             355G                   -
Array-640G  referenced            96K                    -
Array-640G  compressratio         1.00x                  -
Array-640G  mounted               yes                    -
Array-640G  quota                 none                   default
Array-640G  reservation           none                   default
Array-640G  recordsize            128K                   default
Array-640G  mountpoint            /mnt/Array-640G        default
Array-640G  sharenfs              off                    default
Array-640G  checksum              on                     default
Array-640G  compression           off                    local
Array-640G  atime                 off                    local
Array-640G  devices               on                     default
Array-640G  exec                  on                     default
Array-640G  setuid                on                     default
Array-640G  readonly              off                    default
Array-640G  jailed                off                    default
Array-640G  snapdir               hidden                 default
Array-640G  aclmode               passthrough            local
Array-640G  aclinherit            passthrough            local
Array-640G  createtxg             1                      -
Array-640G  canmount              on                     default
Array-640G  xattr                 on                     default
Array-640G  copies                1                      local
Array-640G  version               5                      -
Array-640G  utf8only              off                    -
Array-640G  normalization         none                   -
Array-640G  casesensitivity       sensitive              -
Array-640G  vscan                 off                    default
Array-640G  nbmand                off                    default
Array-640G  sharesmb              off                    default
Array-640G  refquota              none                   default
Array-640G  refreservation        none                   default
Array-640G  guid                  14424326156488930116   -
Array-640G  primarycache          all                    default
Array-640G  secondarycache        all                    default
Array-640G  usedbysnapshots       0B                     -
Array-640G  usedbydataset         96K                    -
Array-640G  usedbychildren        219G                   -
Array-640G  usedbyrefreservation  0B                     -
Array-640G  logbias               latency                default
Array-640G  objsetid              54                     -
Array-640G  dedup                 off                    default
Array-640G  mlslabel              none                   default
Array-640G  sync                  always                 local
Array-640G  dnodesize             legacy                 default
Array-640G  refcompressratio      1.00x                  -
Array-640G  written               96K                    -
Array-640G  logicalused           219G                   -
Array-640G  logicalreferenced     42.5K                  -
Array-640G  volmode               default                default
Array-640G  filesystem_limit      none                   default
Array-640G  snapshot_limit        none                   default
Array-640G  filesystem_count      none                   default
Array-640G  snapshot_count        none                   default
Array-640G  snapdev               hidden                 default
Array-640G  acltype               nfsv4                  default
Array-640G  context               none                   default
Array-640G  fscontext             none                   default
Array-640G  defcontext            none                   default
Array-640G  rootcontext           none                   default
Array-640G  relatime              off                    default
Array-640G  redundant_metadata    all                    default
Array-640G  overlay               on                     default
Array-640G  encryption            off                    default
Array-640G  keylocation           none                   default
Array-640G  keyformat             none                   default
Array-640G  pbkdf2iters           0                      default
Array-640G  special_small_blocks  0                      local

 

 

[Needfuldoer]

Sounds like either the download application isn't letting go of the space, or maybe the copy/move step isn't cleaning up after itself properly.

 

If you kill the download application process at the end of a job instead of rebooting, does the space free up? Does it stake a claim to the amount of space it's going to need before the download commences?

[Levent]

9 minutes ago, Needfuldoer said:

Sounds like either the download application isn't letting go of the space, or maybe the copy/move step isn't cleaning up after itself properly.

 

If you kill the download application process at the end of a job instead of rebooting, does the space free up? Does it stake a claim to the amount of space it's going to need before the download commences?

Holy shit that was the case. Turning off the Jails that were using the said pool immediately reduced my disk usage. Now question is why this even happens to begin with.