Uses for optane drives - 16GB

[ianm_ozzy]

Hi all

 

For context.

A sata ssd failed in my proxmox server about 6 months ago  - boot drive. It was  just out of warranty.

No great problem as I  bought another cheap one to do the same & replaced with a cheap 256GB  for about AU$20.

It failed recently.

It iwas not worth  the trouble of replacing/seeking refund/replacement as I would need to take it back, then maybe go for the replacement if available.

 

Modern Sata SSDs just suck. Typically the have 4 bits/cell and no RAM on them as  cache. This destroys  performance and  longevity. Also  when the SLC cache is full, write speeds tank to worse than hard drives.

I am sick of them & wish to use them as little as practical.

 

Presently in my main server have 3 x sata 512GB in RAIDZ1 for redundancy at least.

It seems their price  has increased substantially recently, so  have more reason to avoid them.

 

I just ordered a few pcie nvme adapers. They are very cheap.

 

I am about to order some 16GB optane drives. They are also very cheap.

 

I am presently using  2 of them in my server as special deduplication devices  in truenas for a games storage.

With  1MB block sizes,  does really well. They are mirrored  and are both about 6GB used.

There are about 6TB multiple game ISCSI shares to different machines.

The point here is I find them cheap and useful.

 

Other uses I can think of, meta data devices or zlog devices in truenas scale.  That probably will not be much use to me.

Boot drives as a mirror  for a home server, such as proxmox or truenas - definitely.

 

 

 

I am definitely buying more. I am unsure how many  I need.

So a mirrored boot drive in main server proxmox.

 

Also an old machine with truenas on there, periodically turned on and  backs up my data.

1 for boot drive and maybe 2 for   deduplication devices, which saves a lot of cash on hard drives if configured well (1MB block/record sizes).

 

 

1 or 2 spare possibly.

 

Which other uses so they have , that is in the context of a home server?

 

 

So to reiterate - sata SSDS - just awful and increasing in price & wish to use as little as possible.

Intel optane 16GB - very cheap. Will maybe last 'forever', where  the 16GB capacity  eventually makes them useless - but do not fail.

 

Which other uses so they have , that is in the context of a home server?

I am trying to estimate how many to buy.

 

Useful and informed information is  appreciated.

 

 

Thanks

 

 

 

 

 

 

 

 

[Bersella AI]

I actually have 3 Optane SSDs utilized on my TrueNAS server, as shown below:

 

As you can spot, one of them is hosting the TrueNAS system itself, while others are assigned to the Cache pool and arranged as a "stripe" array. They gave me a good place to host a light-weight Linux-based system & some additional services, as well as caching.

As written in my blog, assigning them as L2ARC or SLOG is not ideal in my scenarios: both my family's traffic and access to the server are very limited, and such configurations, intended for concurrent access, would not seem to be useful.

 

Also, be aware that any electronic stuff is deemed to fail, including Optane, especially when operating under uncontrolled environments such as high temperature. It's also worth noting that not all SATA SSDs are equipped with QLC (as you said, 4 bits/cell) Flash chips; this is more likely the case in cheaper models like WD Green SN350, but is not in premium ones such as Crucial MX500 (some also have dedicated cache equipped).

[ianm_ozzy]

10 hours ago, Bersella AI said:

I actually have 3 Optane SSDs utilized on my TrueNAS server, as shown below:

 

As you can spot, one of them is hosting the TrueNAS system itself, while others are assigned to the Cache pool and arranged as a "stripe" array. They gave me a good place to host a light-weight Linux-based system & some additional services, as well as caching.

As written in my blog, assigning them as L2ARC or SLOG is not ideal in my scenarios: both my family's traffic and access to the server are very limited, and such configurations, intended for concurrent access, would not seem to be useful.

 

Also, be aware that any electronic stuff is deemed to fail, including Optane, especially when operating under uncontrolled environments such as high temperature. It's also worth noting that not all SATA SSDs are equipped with QLC (as you said, 4 bits/cell) Flash chips; this is more likely the case in cheaper models like WD Green SN350, but is not in premium ones such as Crucial MX500 (some also have dedicated cache equipped).

 

Interesting.

I am planning to use an L2ARC, but probably not an optane drive.

All electronic storage devices  do fail, but those optane drives are extremely  resilient for their size. I am really comparing them to most sata drives. My server(s) are reasonably well ventilated. They are  just standard microATX/ATX cases  with ventilation and fans.  No big GPUs generating heat. No pegged CPUs.

I notice the boot pool is  a single 16GB optane drive.  

Are you concerned if that fails, then  you cannot boot the machine? I expect you would do a reinstall and upload the configuration.

 

The question I am really asking, would a 16GB optane drive probably last 'forever' as a server boot drive, where its usefulness is dictated by its size?

I wish to use them as boot drives. One is a proxmox machine, the other truenas scale.

 

I am just unsure if a mirrored boot setup is needed?

 

I also use 2 already as deduplication devices in a mirror.    They get hit hard by I/O, but that is what they are designed for?

 

 

Thanks

 

 

 

 

[Bersella AI]

On 3/30/2024 at 7:46 AM, ianm_ozzy said:

Are you concerned if that fails, then  you cannot boot the machine? I expect you would do a reinstall and upload the configuration.

Yes I indeed have concerns about post-install configurations. I utilized TrueNAS Scale 23.10.1 earlier this year, which dropped native Docker support and I found challenging to get k8s-based Charts working due to poor Internet connections. I then installed Docker manually via command lines, which did not persist after a system upgrade... This means I have to reconfigure Docker and its apps after another clean installation of Scale, making me annoyed. But then, I decided to stay on that initial Scale version, leave it for months or years, until that little Optane runs into trouble. That's it! Nothing to worry about!

 

On 3/30/2024 at 7:46 AM, ianm_ozzy said:

The question I am really asking, would a 16GB optane drive probably last 'forever' as a server boot drive, where its usefulness is dictated by its size?

I wish to use them as boot drives. One is a proxmox machine, the other truenas scale.

As for the endurance, a product brief of 1st gen Optane can be found on Intel's official website, guarantying a daily writing endurance of 100GB, and a warranty of 5 years, which theoretically make it more likely to last for a long time. However, as said above, there are other factors that can potentially kill Optane including high temperature, power fluctuation or failure, a current surge, etc., and we therefore cannot expect it to persist "forever".

As read from an article by Jason Rose, deduplication is discouraged in regular setups as it consumes too much on processors, RAM and/or storage. I did, though, attempt to assign an Optane as SLOG in the Main Storage pool, but only negligible throughput was observed in the monitoring section. Two Optanes are now assigned to a regular storage pool, to provide cache for apps, including transcoded streams by Jellyfin. That's what I found these Optanes are usable for besides being a boot drive.

[ianm_ozzy]

On 4/1/2024 at 1:18 AM, Bersella AI said:

Yes I indeed have concerns about post-install configurations. I utilized TrueNAS Scale 23.10.1 earlier this year, which dropped native Docker support and I found challenging to get k8s-based Charts working due to poor Internet connections. I then installed Docker manually via command lines, which did not persist after a system upgrade... This means I have to reconfigure Docker and its apps after another clean installation of Scale, making me annoyed. But then, I decided to stay on that initial Scale version, leave it for months or years, until that little Optane runs into trouble. That's it! Nothing to worry about!

 

As for the endurance, a product brief of 1st gen Optane can be found on Intel's official website, guarantying a daily writing endurance of 100GB, and a warranty of 5 years, which theoretically make it more likely to last for a long time. However, as said above, there are other factors that can potentially kill Optane including high temperature, power fluctuation or failure, a current surge, etc., and we therefore cannot expect it to persist "forever".

As read from an article by Jason Rose, deduplication is discouraged in regular setups as it consumes too much on processors, RAM and/or storage. I did, though, attempt to assign an Optane as SLOG in the Main Storage pool, but only negligible throughput was observed in the monitoring section. Two Optanes are now assigned to a regular storage pool, to provide cache for apps, including transcoded streams by Jellyfin. That's what I found these Optanes are usable for besides being a boot drive.

 

The intel optane 16GB I have and just ordered have  far higher endurance than 100GB.

the once I uses for deduplicton have about 365TB endurance.

 

The ones I just ordered about 180TB.

Their main benefit as I see is their cost.

I intend to use them only in mirrored configurations if practical, with extras to replace upon a failure.

So mirrored dedup devices, mirroed boot drives, caches.

Maybe useful for swap drives in linux  or pagefile in windows.

 

16GB is enough to hold linux, and also  usable for a router like opnsesne of pfsense.

 

Also as they are 16GB, replacing a failed one, then using a linux boot USB, or netboot  then something like:

cat backupfile | dd of=/dev/sd?  - could be much quicker than a reinstall.

 

Your situation with truenas scale seems messy.

I try to use the right tool for the job.

So proxmox as a hypervisor, truenas as NAS only.

 I have a virtual machine in proxmox, ubuntu server.

I use docker there, with the data on the NAS.

The VM is easily backup up/restored through proxmox.

It is for jellyfin, audio streaming & netboot.

 

 

Also an LXC container using docker, which lancache.

 

 

 

 

 

 

 

 

 

[Nick7]

On 3/29/2024 at 4:57 AM, ianm_ozzy said:

Modern Sata SSDs just suck. Typically the have 4 bits/cell and no RAM on them as  cache. This destroys  performance and  longevity. Also  when the SLC cache is full, write speeds tank to worse than hard drives.

I am sick of them & wish to use them as little as practical.

You get what you pay for.

Cheapest SSD's are like this. More expensive ones are fast, even under sustained write loads.

[ianm_ozzy]

On 4/5/2024 at 6:25 PM, Nick7 said:

You get what you pay for.

Cheapest SSD's are like this. More expensive ones are fast, even under sustained write loads.

 

Yes. They are terrible.

Also they are increasing in price.

The workaround is the very cheap 16GB optane.

They have much higher endurance than the more expensive sata SSDs.

The downside is their size though.

I have been using a couple  in truenas as dedup devices & working well for a year.

Apparently deduplication is  hated by a lot of truenas users, but do not care.

A massive 6GB used on each.

 

They are hit hard with I/O.

 

So useful boot drives for a home server, or dedup/metadata devices in truenas are an option.

Also they can be used as a cache drive as initially designed.

Apparently helpful as  swap/pagefile drives also, due to the very low latency.

 

So I ordered more.  I will use them in mirrored configs, and  have spare if any fail.

 

 

 

 

[cmndr]

As hinted at, you probably do NOT want to do de-dup, it creates more points of failure and complicates things overall. 

There are edge cases like running a system where the exact same file gets thrown around a bunch but that's not the norm. It's usually smarter just to buy more drives and to use ZSTD. 

Larger optane drives do work well for dedup tables but they also work well as a `special vdev` though you'd likely either want to mirror this or to have a second system where the data is backed up just in case. 

[ianm_ozzy]

On 4/13/2024 at 1:06 AM, cmndr said:

As hinted at, you probably do NOT want to do de-dup, it creates more points of failure and complicates things overall. 

There are edge cases like running a system where the exact same file gets thrown around a bunch but that's not the norm. It's usually smarter just to buy more drives and to use ZSTD. 

Larger optane drives do work well for dedup tables but they also work well as a `special vdev` though you'd likely either want to mirror this or to have a second system where the data is backed up just in case. 

Oh so amusing to me.

 

Firstly I have no use for truenas without deduplication. If not, then Standard shares like cifs/nfs are easily created through turnkey containers in proxmox.

 

 

Some investigation testing & tweaks, and guess what - It is working great for me, despite all the negativity due to the deduplication is bad narrative.

 

I suggest you watch this: 

 

Also look at this.

I doubt my truenas VM will kick the bucket before this is available:

https://www.truenas.com/blog/fast-dedup-is-a-valentines-gift-to-the-openzfs-and-truenas-communities/

 

Maybe those 16GB optane will no longer be needed,  after I spent so very very much cash on them.

 

 

My data is backed up frequently

 

 

 

 

 

[cmndr]

17 hours ago, ianm_ozzy said:

Oh so amusing to me.

 

Firstly I have no use for truenas without deduplication. If not, then Standard shares like cifs/nfs are easily created through turnkey containers in proxmox.

There's file compression and read acceleration. 
It's also fairly robust against data corruption.
What use case do you have? 

 

17 hours ago, ianm_ozzy said:

Maybe those 16GB optane will no longer be needed,  after I spent so very very much cash on them.

I think those are like... $5 on ebay. They've been $5-10 for a a few years now. 

[ianm_ozzy]

8 hours ago, cmndr said:

There's file compression and read acceleration. 
It's also fairly robust against data corruption.
What use case do you have? 

 

I think those are like... $5 on ebay. They've been $5-10 for a a few years now. 

So my use case:

2 x 16GB intel optane as dedup devices in truenas scale.

2 x iscsi 12TB sparse drives - most importantly 1MB block sizes.

A virtual machine on the server connected to one ISCSI.

My main gaming machine connected to another.

Both ISCSI drives in both machines formatted with 1MB block sizes.

 

Presently 6TB of games - 2x  sharing the same space.

 

Deduplication data is only for those ISCSI drives - all with 1MB block sizes.

All for games storage - nearly all large files.

 

No deduplication of any shares using cifs on nfs.

 

Deduplication data is a massive 6GB on the mirrored drives and 2GB of memory.

It is hardly an issue.

If it fills up to the size of the optane drives, will be an issue.

That is not happening. It is staying constant, unless I install more games that is.

 

Also a lancache server in a container - 200GB space.

 

Typical scenario:

The windows virtual machine updates my games overnight, writing to one of the ISCSI.

The lancache  is populated also.

 

 

When I wish to game, well boot into the gaming machine.

 

The 1/2TB nvme as a cache drive using primocache ensures  fast game updated also - on my gaming machine.

 

Game updates -  oh so fast downloads from the lancache.

There are lots of concurrent reads & writes  to & from the cache disk and the NAS. It is easy to monitor in windows.

 

The concurrent reads/writes to & from the NAS are interesting.

So reads when needed with random 1MB blocks.  Not very taxing.

Writes to the NAS - just taking up network bandwidth.  Deduplication means mainly  changes to the dedup data.

 

So  less space  needed to store games.

Very fast game updates.

All games are updated fast and available to play when I want - the main motivation for me.

I do not want to install/reinstall them.

 

 

Ongoing saving of cash with my 100Mb internet speed.  Gigabit speeds are much  much more expensive where I live.

 

I could just leave the gaming machine on 24/7instead , but not a smart idea.

 

A lowish power server is a better option, as it does the gaming related updates as well as much more.

 

 

 

 

 

 

 

 

 

 

 

 

 

[cmndr]

You're likely better off going RAID 5 (err Z1) and getting doubling your storage that way. It's a lot less finicky, increases read throughput (instead of decreasing it) and you don't have to worry about the array going wonky. Also as an FYI, dedup works best with smaller block sizes, if you have large blocks, the blocks will disproportionately be unique - though this increases the amount of space needed for dedup tables (going to 128 means 8x the space needed). The exception is if it's the exact same file 10 times (which sometimes happens with game textures). 

There might be an argument for setting up one pool with dedup and trying. I would NOT setup anything with VMs in a dedup setting due to the performance hit. 

ZFS can also do compression which is nice. ZSTD and LZ4 work well and often get you around 10% more space for basically free. 

With that said, it's VERY easy for a low power server's running cost to roughly match the cost of a faster internet connection. Also the benefit of the server will go down with time as internet plans tend to get better. 20 years ago I had a high speed 8Mbps connection at my mother's house. I think it's 500ish now at a lower price. 

-----

With that said, my views in de-dup might change in a few months. 
Fast Dedup is being worked on and it sounds promising. 
https://www.ixsystems.com/blog/fast-dedup-is-a-valentines-gift-to-the-openzfs-and-truenas-communities/