These Servers are TOO EXPENSIVE

[Marvin_Nor]

2 minutes ago, leadeater said:

Even 2 tier is still tiering, both tiers contribute to total capacity of the provisioned virtual disk unlike caching where the cache does not contribute. It's not really 'caching' either, it acts like caching because writes happen in the fastest tier but you want that, difference is new data will stay in the fast tier until tier high water mark or aged where caching will write the data to the underlying array as soon as possible and that data lives in both.

 

Storage Spaces only stores the data chunks in a single tier, it's migrated between them so you won't have same data chunk in 2 tiers at the same time. The exception to that is modifying data (which is technically new data) because it's written to the fast tier then replaces the expired chunk.

 

Most of the new fancy features are ReFS only though.

The default is to make faster disks "cache" aka, it doesn't count towards capacity (for 2016 and 2019). You can make it true two tier through PowerShell, which will make it reserve some capacity on the fast tier for "cache" (1GB or something you set). 

 

Yeah, most fancy features are ReFS only, like resiliency tiering and so on.

[leadeater]

3 minutes ago, Marvin_Nor said:

The default is to make faster disks "cache" aka, it doesn't count towards capacity (for 2016 and 2019).

It does, I'm using it right now and both tiers contribute to provisioned virtual disk size. Windows Server 2016. Edit: WB cache is still set but if you enable tiering you have two actual tiers.

[Marvin_Nor]

Just now, leadeater said:

It does, I'm using it right now and both tiers contribute to provisioned virtual disk size. Windows Server 2016.

Is that a Storage Spaces or Storage Spaces Direct setup?

The GUI will also always show how much of the cache is dedicated to the virtual disk, as if it was capacity.

[leadeater]

13 minutes ago, Marvin_Nor said:

Is that a Storage Spaces or Storage Spaces Direct setup?

The GUI will also always show how much of the cache is dedicated to the virtual disk, as if it was capacity.

Standard not S2D, though the same happens for both last I checked (long time ago). You will also see a cache amount, default size changes on resiliency chosen, but it's not used unless the fast tier fills up and it caches the slow tier. If you have NVMe and are using S2D then you get a bit nicer caching but that's actually exclusive to S2D, standalone SS doesn't get that feature and NVMe is just another storage device type.

 

It's been ages since I've done anything with S2D though and I never actually got to do a configuration with NVMe + SSD + HDD.

 

GUI example of stand alone SS with SSD + HDD:

Spoiler

 

 

 

 

 

[Marvin_Nor]

Just now, leadeater said:

Standard not S2D, though the same happens for both last I checked (long time ago). You will also see a cache amount, default size changes on resiliency chosen, but it's not used unless the fast tier fills up and it caches the slow tier. If you have NVMe and are using S2D then you get a bit nicer caching but that's actually exclusive to S2D, standalone SS doesn't get that feature and NVMe is just another storage device type.

 

It's been ages since I've done anything with S2D though and I never actually got to do a configuration with NVMe + SSD + HDD.

 

GUI example of stand alone SSD + HDD:

  Reveal hidden contents

 

 

 

 

 

Ah, right, I come from a primarily S2D Environment with SSD + HDD or NVMe + SSD + HDD, where it won't ever give you the option to define a fast tier unless it's a three tiered solution, or you define costume tiers and make a vDisk based on your own tiers. So it fully commits all fast disk as cache.

Yeah, the cache you're seeing is just a dedicated space on the fast tier, as opposed to S2D where all the fast drives will be cache, again, unless you define it otherwise.

 

Personally had bad experience with standard SS, reaching 500k IOPS is no issue, but when I've put workloads on it, the latency really starts growing (used it for backup). So, switched over to S2D for everything, 1-2M IOPS and still below 1ms latency, happy with that. 

[leadeater]

15 minutes ago, Marvin_Nor said:

Personally had bad experience with standard SS, reaching 500k IOPS is no issue, but when I've put workloads on it, the latency really starts growing (used it for backup).

I've never used it for anything actually at work for production, trialed it on some dev hardware but we just don't have a use for it because we have Netapp and Nutanix HCI so it's firmly in the cool but why category. I use it on my servers at home because it's simple and cheap compared to other options, not having to install extra software or deploy a virtual appliance or dedicate a server just to storage is nice for a home lab.

 

It's also nice having dual 10Gb SMB multichannel to my desktop to store all my games, because why not.

 

Side note it's nice to see someone not auto bitching about Windows and it being used for storage, with a mind set of Windows Server 2003 era. Microsoft has gotten 13 million IOPs and 535GB/s seq out of S2D using 12 servers. Wish S2D was in standard edition though.

[Marvin_Nor]

2 hours ago, leadeater said:

I've never used it for anything actually at work for production, trialed it on some dev hardware but we just don't have a use for it because we have Netapp and Nutanix HCI so it's firmly in the cool but why category. I use it on my servers at home because it's simple and cheap compared to other options, not having to install extra software or deploy a virtual appliance or dedicate a server just to storage is nice for a home lab.

 

It's also nice having dual 10Gb SMB multichannel to my desktop to store all my games, because why not.

 

Side note it's nice to see someone not auto bitching about Windows and it being used for storage, with a mind set of Windows Server 2003 era. Microsoft has gotten 13 million IOPs and 535GB/s seq out of S2D using 12 servers. Wish S2D was in standard edition though.

Yeah, we considered a lot of solutions, including Nutanix. vSAN and Starwind. But ended up on S2D due familiarity, simplicity, performance and already having it licensed due to our licensing model.

 

SMB Multichannel is great! We're using SMB Direct (RDMA), it's working wonders to be honest.

 

Yeah, Windows has come a long way with 2016 and 2019 regarding storage to be fair. And it's pushing some great performance numbers, especially with 2019 and the utilization of persistent memory as cache.

 

With that said though, just came to think of it. Linus could force the fast disk tier to behave like cache (as it would in S2D) instead of fast tier, if desired. Would loose out on the capacity, but could improve the overall performance, also shuffles data quicker in and out of the fast disks.

 

After creating the pool, you could run this little command to force the SSDs into Journal mode (cache mode).:

Get-PhysicalDisk -FriendlyName *part of SSD name* | Set-PhysicalDisk -Usage Journal

 

Then create a virtual disk with a volume:

New-Volume -StoragePoolName *Pool* -FriendlyName "Volume01" -Size 250GB -ResiliencySettingName "Mirror" -FileSystem NTFS -AccessPath "E: "-ProvisioningType Fixed

 

Should also set the column back to default (he had 6 HDDs if I am not mistaken, so 1 column per disk pair) before creating the volume.

Think this would work.

[VanBantam]

Dunno why it took me so long to think of this but why not use a distributed file system?  Each editing PC would be a storage node.  Data would be replicated across the different editing workstations.  If you feel like burning cash you could use something like Spectrum Scale, aka GPFS, which works quite well.  I've had less experience with Gluster but for the price it's a much more reasonable solution.  With either solution you could use data affinity to colocate data with where the data will be processed. 

 

Anyways just an idea.

[unijab]

Does anyone else see the irony here?

 

Linus talks about ZFS needing Linux experts and paying those people.

 

He proceeds to talk to an expert on the forum then still has trouble setting it up.

 

 

[leadeater]

6 hours ago, Marvin_Nor said:

With that said though, just came to think of it. Linus could force the fast disk tier to behave like cache (as it would in S2D) instead of fast tier, if desired. Would loose out on the capacity, but could improve the overall performance, also shuffles data quicker in and out of the fast disks.

I was thinking about that as well, another one I was wondering was having 2 SSDs set Journal mode and 2+ set as Auto for Fast tier. Wonder how that would perform or if the Journal and Tiering would just sort of fight/negate each other.

[Marvin_Nor]

10 minutes ago, leadeater said:

I was thinking about that as well, another one I was wondering was having 2 SSDs set Journal mode and 2+ set as Auto for Fast tier. Wonder how that would perform or if the Journal and Tiering would just sort of fight/negate each other.

In that case the Journal would be write and read cache for HDD, but write only cache for the two SSDs in Fast Tier. And it would shuffle data between slow and fast tier if necessary, while also caching the data. The benefit would be more capacity, and data would be retained on SSDs for a longer period of time due to the fast tier, but would be moved to SSDs faster due to the Journal (while moving data from slow to fast tier). So, could be a gain, if the file usage isn't consistent, but more sporadic. 

 

Really depends on usage I would say.

[leadeater]

2 hours ago, unijab said:

Does anyone else see the irony here?

 

Linus talks about ZFS needing Linux experts and paying those people.

 

He proceeds to talk to an expert on the forum then still has trouble setting it up.

 

 

Don't worry, it's not like Linus actually tried most of what I said anyway . From the bits that were shown in the video I'd rather have just set it up, but I would have also used different hardware given the choice as well.

 

From memory Linus is using that Windows only software to copy files to an offsite server so I think that was also a factor in wanting to stick with Windows, don't actually know if that's still being used or not though.

 

Not matter what though high performance storage requires expertise to setup, how much varies on what platform you pick but there's no true easy foolproof one.

[LinusTech]

1 hour ago, leadeater said:

Don't worry, it's not like Linus actually tried most of what I said anyway . From the bits that were shown in the video I'd rather have just set it up, but I would have also used different hardware given the choice as well.

 

From memory Linus is using that Windows only software to copy files to an offsite server so I think that was also a factor in wanting to stick with Windows, don't actually know if that's still being used or not though.

 

Not matter what though high performance storage requires expertise to setup, how much varies on what platform you pick but there's no true easy foolproof one.

While filming I only had the initial message from you. 

 

And yeah, it's very clear that to get the best results with tiering it would absolutely be necessary to have a TON of experience with this stuff and a pretty well-equipped lab full of test gear. 

 

I think we are just going to stick to our stone-age tiering for the time being (separate servers) and we will cross the tiering bridge when we come to it again... 

[leadeater]

24 minutes ago, LinusTech said:

While filming I only had the initial message from you. 

It's all good, I enjoyed watching it fail more, as a viewer .

 

24 minutes ago, LinusTech said:

I think we are just going to stick to our stone-age tiering for the time being (separate servers) and we will cross the tiering bridge when we come to it again... 

That's still my pick for you guys anyway, as much as I like tiering in principle I prefer the guarantee of all SSD performance. 

[LinusTech]

1 minute ago, leadeater said:

It's all good, I enjoyed watching it fail more, as a viewer .

 

That's still my pick for you guys anyway, as much as I like tiering in principle I prefer the guarantee of all SSD performance. 

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

[imreloadin]

44 minutes ago, LinusTech said:

I think we are just going to stick to our stone-age tiering for the time being (separate servers)

Tiered servers sound more fun anyways

[unijab]

44 minutes ago, LinusTech said:

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

 

Anything other than cost keeping you from going with the 1.5TB 905p ? (Im assuming they are compatible with your server backplane)

[lil_killa]

2 hours ago, LinusTech said:

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

Agreed with @leadeater all flash is still the best option. But, be aware the All NVMe Supermicro Chassis you have oversubscribes the PCIe lanes(PCIe 3.0 x 16) to the Front 24 drives(Back Plane) and another(PCIe 3.0 x 16) to the mid 24 Drives(Mid Plane).

 

You're going to only see a Max of 16 GB/s Read or Write (Best case scenario but will probably be less) for each set of 24 Drives. So if you want to utilize all the performance one day its going to be time for clustering.

[Mark77]

2 hours ago, lil_killa said:

You're going to only see a Max of 16 GB/s Read or Write (Best case scenario but will probably be less) for each set of 24 Drives. So if you want to utilize all the performance one day its going to be time for clustering.

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

[leadeater]

34 minutes ago, Mark77 said:

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

Don't we all wish we were 'limited' to 16GB/s storage system performance though

[lil_killa]

2 hours ago, Mark77 said:

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

Yeah for sure, but you can find a Mellanox 56G switches on ebay for around $2000 and the 56G ConnectX3 cards for around $200 ish. 

[Rudolf Meier]

That wasn't a joke... I'm serious about this... try Intel CAS... I think that's providing what you want... or at least it would be worth a try and maybe you can create another video from this experiment... and it's not that expensive (even free if you use Intel ssds)

And by the way, it also answers another question you had on a board... "who is ever using 4 U.2 drives in one system??" ... well, the guy that's using those ssds as a cache

[nukedeath]

This video was very interesting as I did something similar sometime ago with Linux.

I had a slow RAID6 array of HDDs and a very fast RAID10 of SSDs.

By setting up everything with LVM I could use the fast RAID10 array as an LVM cache in Writeback mode.

With that, data I moved inn goes straight to the SSDs first, LVM then syncs the data down to the slower array.

This was very nice as I could transfer very large DNXHD encodes over 10gbit super fast and know they would stay on the SSD cache as I worked on some silly vacation video.

My SSDs were cheap TLC crap so the write speeds went to shit after a while, please don't use TLC drives for this .

 

LVM cache also has a Writethrough mode that writes to both slow and fast device at once, and is safer.

In either caching modes, all new data written enters the cache.

 

Red Hat started to support this back in 2016 so it's quite battle tested.

[RedlightSparks]

@LinusTech Hi, thought I'd offer some insight into the other potential side of the issue - the one I believe is actually more likely.

 

I believe from previous videos that LMG are running a 10GbE network, are the adapters on your servers teamed in Server (2012R2/2016/2019) and trunked correctly on the switch?  That's assuming you're running multiple connections from each server - ideally 4?  Also is the switch/switches suitably spec'd to permit data flow, do you have a core switch or an edge switch doing the job of a core?  If you have multiple are they correctly stacked to take full advantage of the backplane speed? I also assume that all of your cabling is cat6/7 between workstations and switches, and either twinax or fibre with SFP adapters between the servers and the switches?

 

I also believe that you have a number (3?) import stations elsewhere in the building that are used purely to ingest footage to I assume your hot server.  I'm guessing due to the RED cameras that this is all at 8K so hundreds of GB at a time.  Which means you have the potential for a whole lot of writes happening to your storage that's unrelated to the workloads of your editors. 

 

So before you have any workloads impact from your editors, you have a potentially high number of writes using are storage servers adapters and switch bandwidth.

 

Add the editors doing likely numerous read/writes to the same server across the same switches, it wouldn't surprise me if you're seeing slowdowns/latency due to network flooding, collisions, package drops.  It would be interesting to see a Solarwinds (or similar) network monitor report for your switches over a couple of weeks and to see if the slow downs experienced by your editors coincides with times of high usage.  If so, you may see a marked improvement by simply ingesting footage to local storage and then setting up a scheduled move to hot server nightly.

 

If this is a network issue, this will only get worse if the 6 editors one PC VM experiment completes even if you've configured multiple adapters with teaming within the hypervisor. But that presents its own issues.

[0x6A7232]

Right, so, from the YouTube comments, which illustrate my thought, which is, why not have the combined storage capacity of the mechanical and NVMe, and connect the two servers with a dedicated optical networking link using [url=https://fibertronics-store.com/NIC-PCI-PCI-Express-Cards_c938.htm]Fiber Optic NICs[/url] (no affiliation: please feel free to refer to any source for FONICs, I just did a quick Duck search).  Files not accessed in a certain $time get moved to slow storage. Accessing a file gets it moved to fast storage until it's not accessed for $T.  Or if you've got the space in one server you could do it locally.

 

But you need the network share to see both $slow and $fast tier as 1 drive.  So network share ONIC (sp?) = $slow + $fast.  Internally, however, the server is configured so that files arrive in $fast, move to $slow after specified $time of no access, and get moved to $fast on access in $slow.

 

djdannypachy:

Quote

Hey Linus, what you mention is possible to do! You can use your existing servers (or file shares). Basically, you would have two seperate file volumes / file shares, one would be using NVMe SSD drives and the other would use hard disk drives (your archive). You can create a PowerShell script (or find one online) that would move the files (i.e. project folder) from the fast to slow storage based on your criteria. For example, if a project folder hasn't been accessed, modified or moved within a certain period of time it would be moved to the slower storage. This can also work the other way too, if a folder has been accessed or modified in the slow storage (your archive) it would be moved to the fast storage. You can have the Powershell script(s) run automatically using Task Scheduler. For instance, you can have one script that checks your fast storage on a daily basis, and another check your slow storage more frequently (every hour, 30 minutes, 5 minutes, etc).

 

HoudWarZ:

Quote

@djdannypachy's Scheduled Task idea (or file copy triggered task even ?) is an interesting solution, but this can be simplified to a single File Share, using Tiered Storage and Tier Pinning (ps> Set-FileStorageTier) based on files last access times, combined with forcing an immediate Tier Optimisation to apply the pinning changes aka move files from tier to tier (ps> Optimize-Volume -DriveLetter X -TierOptimize). Your Virtual Drives would also probably greatly benefit to be setup with a large Write Cache, which can only be customised using PowerShell commands. Lastly, to be honest, learn PowerShell if you want to play with Storage Spaces ! You can build yourself a script to easily destroy and rebuild the whole Storage layers and File Shares in a minute using different parameters that you can then benchmark for your specific use cases. Several parameters are also only available to the Powershell commands.

Nikolaos Videos:

Quote

djdannypachy, Linus might want to correct me if I am wrong but what I got from the above video is that he has around of 24TB of superfast NVme SSD storage, where they put all the 8K Video files and the editors work on them videos from then on. But after they are done with a project it goes to the "Archive Server" where the high cqpacity HDD lay around. The superfast SDD server is needed for the Active Projects and for the "We-Might-Need-These-In-The-Near-Future Projects" and his editors complain there isn't enough space available. If they archive on an SSD server, they throw away money.