Intel Optane on AMD base TrueNAS???

[wingyiulam]

Hi all,

 

Does Intel Optane work as cache disk on an AMD base system TrueNAS?

I can't tell there is any differences using an optane ssd vs a normal consumer ssd?

it seems like the Optane SSD simply just works as a normal SSD instead of a caching feature enable SSD on AMD environment.

 

spec

Intel SSDPEL1K100GA01

AMD 3700x

x570s Aorus Master

 

Thanks,

Wing

[jaslion]

15 minutes ago, wingyiulam said:

Hi all,

 

Does Intel Optane work as cache disk on an AMD base system TrueNAS?

I can't tell there is any differences using an optane ssd vs a normal consumer ssd?

it seems like the Optane SSD simply works as a normal SSD on my AMD based system.

 

spec

Intel SSDPEL1K100GA01

AMD 3700x

x570s Aorus Master

 

Thanks,

Wing

It does work but just get a bigger ssd. Optane is fast but hard drives stay slow and a small cache is useless basically. Might as well just get a cheap good 250gb ssd like a mx500 or wd blue one and call it a day.

 

Optane only had one real purpose and that was being an extremely fast scratch drive other than that it never really took of for anything. In very niche cases it was used as a cache for a sata based ssd array but other than that it was always just a scratch disk.

 

Unless you have hundreds of drives working as one forget about it and just use a normal ssd for cache.

[porina]

Optane basically can be considered like any other SSD in the generic sense.

 

The differences vs flash based SSD mainly lie in two areas: random read speed and endurance.

 

Random read of Optane is many times faster than top tier flash SSDs, however when used as a cache, obviously it can only work if that random data is in the cache. Does this apply to the expected workloads?

 

Endurance is much higher per capacity than flash. On the flip side, due to the cost of Optane, you can get a LOT more flash for the same price, and endurance scales with capacity. The linked 100GB Optane drive is rated at 10.9 PBW. For comparison, a 980 Pro 2TB is rated at 1.2 PBW. If you're expecting to do a ton of writes to the cache, then there might be some argument for Optane. In realistic workloads, you're not going to wear out the 980 Pro I used as comparison example. You'd have to write 360GB to it every day for 5 years to reach its endurance rating.

[Soapy1234]

Not to hijack too much, but Optane would be preferred for a SLOG, correct?

[Electronics Wizardy]

9 minutes ago, Soapy1234 said:

Not to hijack too much, but Optane would be preferred for a SLOG, correct?

Yea they make a great slog. Here is a good article about that https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/

 

You probably want a 900p for a slog if possible, as the write speeds on the 16 +32gb modules is pretty low.

[wingyiulam]

17 hours ago, Electronics Wizardy said:

Yea they make a great slog. Here is a good article about that https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/

 

You probably want a 900p for a slog if possible, as the write speeds on the 16 +32gb modules is pretty low.

does it work on AMD based system? I tried an Intel SSDPEL1K100GA01 and an Adata SX8200 Pro on my system; there is no difference using any of them.

[Electronics Wizardy]

6 minutes ago, wingyiulam said:

does it work on AMD based system? I tried an Intel SSDPEL1K100GA01 and an Adata SX8200 Pro on my system; there is no difference using any of them.

Yea it works on a amd system, optane drives are just anouther nvme drive connection wise.

 

What are you using the drive for in ZFS? If your using it as a slow, are you really limits by sync writes, Id argue most  home servers don't need a slog, and you can get away with just turning sync off.

 

Optane has one other big advantage, all the drives have power loss protection, while most consumer grade ssds don't.

[Soapy1234]

9 minutes ago, Electronics Wizardy said:

Yea it works on a amd system, optane drives are just anouther nvme drive connection wise.

 

What are you using the drive for in ZFS? If your using it as a slow, are you really limits by sync writes, Id argue most  home servers don't need a slog, and you can get away with just turning sync off.

 

Optane has one other big advantage, all the drives have power loss protection, while most consumer grade ssds don't.

At what point would you recommend turning on synchronous writes and adding a slog?  For what kind of workflow?  Again, not to hijack too much, but I do audio recording and editing, and I'd like to be able to work off of the NAS.

[Electronics Wizardy]

Just now, Soapy1234 said:

At what point would you recommend turning on synchronous writes and adding a slog?  For what kind of workflow?  Again, not to hijack too much, but I do audio recording and editing, and I'd like to be able to work off of the NAS.

When you do lots of writes that you can't afford to lose, and need good performance. 

 

Normally ZFS can take upto 5 seconds to write data, so if your ZFS nas lost power when recording you lost up to 5 seconds of audio(but test how well your program handles storage failure too). With a Slog, its safe on disk as soon as the sync write command has finished. 

 

Id probably use it for things like customers vms to lower the risk of customers losing data, databases where you don't want to lose any data. 

 

 

[Soapy1234]

1 minute ago, Electronics Wizardy said:

When you do lots of writes that you can't afford to lose, and need good performance. 

 

Normally ZFS can take upto 5 seconds to write data, so if your ZFS nas lost power when recording you lost up to 5 seconds of audio(but test how well your program handles storage failure too). With a Slog, its safe on disk as soon as the sync write command has finished. 

 

Id probably use it for things like customers vms to lower the risk of customers losing data, databases where you don't want to lose any data. 

 

 

I have all my stuff on a UPS, so I imagine it would be less an issue--but feel free to correct me if I'm wrong.  Thanks!

[Electronics Wizardy]

Just now, Soapy1234 said:

I have all my stuff on a UPS, so I imagine it would be less an issue--but feel free to correct me if I'm wrong.  Thanks!

Yea a UPS removes a good amount of the risk, but the system can still crash, have a psu failure, be unplugged or have many other issues that cause it to turn off without notice.

[Soapy1234]

7 minutes ago, Electronics Wizardy said:

Yea a UPS removes a good amount of the risk, but the system can still crash, have a psu failure, be unplugged or have many other issues that cause it to turn off without notice.

I see.  But it's only 5 seconds of data?  I'm a bit confused.

[Electronics Wizardy]

2 minutes ago, Soapy1234 said:

I see.  But it's only 5 seconds of data?  I'm a bit confused.

There is a setting called vfs.zfs.txg.timeout(name may vary depending on exact os). I think its 5-30 seconds by default. There is a also a size limit

 

But there are many companies and uses where losing that much data, esp after ZFS says its safe on disk its a big issue, so they will pay the extra for a fast slog to make sure that the data is actually safe.

[Soapy1234]

1 hour ago, Electronics Wizardy said:

There is a setting called vfs.zfs.txg.timeout(name may vary depending on exact os). I think its 5-30 seconds by default. There is a also a size limit

 

But there are many companies and uses where losing that much data, esp after ZFS says its safe on disk its a big issue, so they will pay the extra for a fast slog to make sure that the data is actually safe.

Ok, so a small bunch of 100kb-40MB files written every once in a while should transfer to the drives without many issues, right?  Cause I'm hearing if the 5-30 seconds of data are in the GBs it makes sense.  I'm just trying to figure out my particular use case, cause there ain't much I can find online about it.

[Electronics Wizardy]

3 minutes ago, Soapy1234 said:

Ok, so a small bunch of 100kb-40MB files written every once in a while should transfer to the drives without many issues, right?  Cause I'm hearing if the 5-30 seconds of data are in the GBs it makes sense.  I'm just trying to figure out my particular use case, cause there ain't much I can find online about it.

What program is writing these files? Is this over a network share?

 

Have you played with sync settings?

 

By default program determins if sync if used. samba doesn't use sync by default, But I think iscsi and nfs do. ZFS can override this though.

 

Its 5-30 seconds or a few gb of data, whichever is less. So if your writing 1mB/s and its set to 5 seconds, the max you can lose is the last 5mB of data.

 

 

Also what happens if you lose the last 5 seconds of data in your application? Can you recreate the data? Can you ignore the issue?

[jde3]

Uhh... hmm..

So, I wouldn't really do this. You can but.. using a NVME for slog and cache is fine. (the same device partitioned 80% cache 20% slog works pretty well) Optaine is prob unnecessary.

 

You don't need a UPS with ZFS and you don't need to worry about disk consistency with ZFS because it's always consistent to it's last state.

in other words don't worry about the 5 second bit, if ZFS is in a write at the time, it will discard it on mount and roll itself back to the last consistent state. it's not really possible for it to do a partial write. (yes you lose the data in flight, the data in ram at the time) but no you will not lose the file or anything nor will your disk become corrupt.

 

 I think you may be overly worried about it. If you lose power you're going to lose the thing making the audio too so.. how often is power going out at your house? The true solution might be to move

[Soapy1234]

5 hours ago, Electronics Wizardy said:

What program is writing these files? Is this over a network share?

 

Have you played with sync settings?

 

By default program determins if sync if used. samba doesn't use sync by default, But I think iscsi and nfs do. ZFS can override this though.

 

Its 5-30 seconds or a few gb of data, whichever is less. So if your writing 1mB/s and its set to 5 seconds, the max you can lose is the last 5mB of data.

 

 

Also what happens if you lose the last 5 seconds of data in your application? Can you recreate the data? Can you ignore the issue?

The program is Pro Tools, and the idea is over a network share.  I'm on Unraid right now, but am planning to switch to ZFS for read caching purposes.  Considering ZFS on Unraid is a thing now and they'll likely add ZFS functionality officially soon, I'm going that route.

 

Haven't set it up yet, so I haven't played with sync settings.  Basically doing research so I don't screw it up the first time.

 

Good to know about how much data.  If it's a couple takes that get lost that's usually fine worst case.  I can just do another take if everything goes kaput.

 

 

1 hour ago, jde3 said:

Uhh... hmm..

So, I wouldn't really do this. You can but.. using a NVME for slog and cache is fine. (the same device partitioned 80% cache 20% slog works pretty well) Optaine is prob unnecessary.

 

You don't need a UPS with ZFS and you don't need to worry about disk consistency with ZFS because it's always consistent to it's last state.

in other words don't worry about the 5 second bit, if ZFS is in a write at the time, it will discard it on mount and roll itself back to the last consistent state. it's not really possible for it to do a partial write. (yes you lose the data in flight, the data in ram at the time) but no you will not lose the file or anything nor will your disk become corrupt.

 

 I think you may be overly worried about it. If you lose power you're going to lose the thing making the audio too so.. how often is power going out at your house? The true solution might be to move

 

I'm already on a UPS for both my current NAS and main computer with 25 minutes of possible runtime, so there's that.  And thanks for letting me know about how that works.  I wasn't sure if file corruption was a thing to worry about in that case or something.  Again, mostly just doing research and teaching myself how to use ZFS.  Thanks!

[Electronics Wizardy]

9 minutes ago, Soapy1234 said:

The program is Pro Tools, and the idea is over a network share.  I'm on Unraid right now, but am planning to switch to ZFS for read caching purposes.  Considering ZFS on Unraid is a thing now and they'll likely add ZFS functionality officially soon, I'm going that route.

Wht network protocol? Your probably use smb, so sync is off by default, so slog won't help at all unless you set sync=always.

 

Id just switch to truenas or anouther os, hanles zfs much better and is offically supported. ZFS on unraid doesn't seem like a great idea to me.

 

[jde3]

33 minutes ago, Soapy1234 said:

The program is Pro Tools, and the idea is over a network share.  I'm on Unraid right now, but am planning to switch to ZFS for read caching purposes.  Considering ZFS on Unraid is a thing now and they'll likely add ZFS functionality officially soon, I'm going that route.

 

Haven't set it up yet, so I haven't played with sync settings.  Basically doing research so I don't screw it up the first time.

 

Good to know about how much data.  If it's a couple takes that get lost that's usually fine worst case.  I can just do another take if everything goes kaput.

 

 

 

I'm already on a UPS for both my current NAS and main computer with 25 minutes of possible runtime, so there's that.  And thanks for letting me know about how that works.  I wasn't sure if file corruption was a thing to worry about in that case or something.  Again, mostly just doing research and teaching myself how to use ZFS.  Thanks!

I've also done professional music recording.

You know really in 2021 I'd do it like this personally.. MacOS -> project file APFS internal (or ZFS on MacOS on mirrored external drives if you are really paranoid) as a temporary location then rsync that over to the NAS when finished.

I don't think I'd try to record a project file direct to the NAS but their is nothing really wrong with doing so. For post you can work directly off the NAS. ZFS snapshots on all the things sounds like a really great idea here.

 

Agreed on Unraid. Linus is like the only person in the world that likes that. I've worked in the industry for 20 years and never even heard of it till he started talking about it.

Me being an expert with it I'd just do FreeBSD vanilla or Ubuntu+ZFS.. but if you want one with a web frontend.. might check Xigma NAS if you don't like FreeNAS/TrueNAS. In my opinion storage should be boring.. you don't need a lot of features or VM's or jails or nonesense that can go wrong. ZFS I trust.. everything else not so much lol

 

 

[Soapy1234]

11 hours ago, Electronics Wizardy said:

Wht network protocol? Your probably use smb, so sync is off by default, so slog won't help at all unless you set sync=always.

 

Id just switch to truenas or anouther os, hanles zfs much better and is offically supported. ZFS on unraid doesn't seem like a great idea to me.

 

It's all SMB at the moment, so good to know.  That's what I figured as far as synchronous writes.

 

11 hours ago, jde3 said:

I've also done professional music recording.

You know really in 2021 I'd do it like this personally.. MacOS -> project file APFS internal (or ZFS on MacOS on mirrored external drives if you are really paranoid) as a temporary location then rsync that over to the NAS when finished.

I don't think I'd try to record a project file direct to the NAS but their is nothing really wrong with doing so. For post you can work directly off the NAS. ZFS snapshots on all the things sounds like a really great idea here.

 

Agreed on Unraid. Linus is like the only person in the world that likes that. I've worked in the industry for 20 years and never even heard of it till he started talking about it.

Me being an expert with it I'd just do FreeBSD vanilla or Ubuntu+ZFS.. but if you want one with a web frontend.. might check Xigma NAS if you don't like FreeNAS/TrueNAS. In my opinion storage should be boring.. you don't need a lot of features or VM's or jails or nonesense that can go wrong. ZFS I trust.. everything else not so much lol

 

 

 

It's currently a plugin, but planned to get official release for Unraid in a few months. It;s just not officially supported yet.  I agree it should be boring, but I also use it for media streaming and whatnot, and I like how Unraid uses Docker containers and makes things relatively painless--and the community is just overall nicer than FreeNAS.  I actually got the idea from Wendell and Steve expanding his storage, and THEN Linus did the video.  I'll look into Xigma NAS, haven't heard of it before.

I've been doing that process (I'm on PC BTW) with using an internal drive and copying it over later, but I haven't tried with rsync, just doing it manually with Krusader.  I'll have to try that.

That said, I've ran into issues with syncing because of overwriting the main PT file every time I sync to the NAS--I don't tend to "Save As" different versions, and I'd like to keep things relatively organized without having multiple disparate versions that I have to worry about overwriting doing the copy whenever I do the sync.  And some of my projects go on for months at a time, which I've ran into these kinds of issues.

[jde3]

Rsync is a Unix thing not really a windows thing. (Windows world hasn't discovered delta copy / checksum on copy yet)

 

You can do it but it's clunky.. and far as I know there are no good GUI's for it.

 

Xigma NAS's former names were: FreeNAS -> NAS4Free -> Xigma NAS. It is the original FreeNAS UI and distro before IX Systems forked it. Essentially it's FreeBSD with a WebUI. (Where as TrueNAS is TrueOS with a WebUI.. Beta Hacked up FreeBSD)

 

I think Rsync really would solve a lot of your problems tho because you could verify the data was the same at the source and destination.. Perhaps just do it in the console with MobaXTerm (Cygwin environment)?

 

I will tell you the windows file copy dialogue has destroyed more of my audio files than any other thing.

[Soapy1234]

On 10/11/2021 at 10:30 AM, jde3 said:

Rsync is a Unix thing not really a windows thing. (Windows world hasn't discovered delta copy / checksum on copy yet)

 

You can do it but it's clunky.. and far as I know there are no good GUI's for it.

 

Xigma NAS's former names were: FreeNAS -> NAS4Free -> Xigma NAS. It is the original FreeNAS UI and distro before IX Systems forked it. Essentially it's FreeBSD with a WebUI. (Where as TrueNAS is TrueOS with a WebUI.. Beta Hacked up FreeBSD)

 

I think Rsync really would solve a lot of your problems tho because you could verify the data was the same at the source and destination.. Perhaps just do it in the console with MobaXTerm (Cygwin environment)?

 

I will tell you the windows file copy dialogue has destroyed more of my audio files than any other thing.

I figured. I'll give that a shot. Thanks!