Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
nicklmg

These Servers are TOO EXPENSIVE

Recommended Posts

Many ways to skin a cat, but archiving with breadcrumbs was a solution I used at a previous job. Leaving the folder structures in tact, a user would get their project archived and the project folder itself flips to a shortcut. I feel like today you would instead use DFS+PS or maybe even symbolic links to make it even more seemless. Was a solution I had to figure out because engineers were too "busy" to archive their own projects. Was fun in the end though.

Link to post
Share on other sites
39 minutes ago, Marvin_Nor said:

Storage Spaces / S2D doesn't really tier data either, unless you have a three tier setup; NVMe, SSD and HDD. With only two "tiers" one will be cache and the other will be storage.

Even 2 tier is still tiering, both tiers contribute to total capacity of the provisioned virtual disk unlike caching where the cache does not contribute. It's not really 'caching' either, it acts like caching because writes happen in the fastest tier but you want that, difference is new data will stay in the fast tier until tier high water mark or aged where caching will write the data to the underlying array as soon as possible and that data lives in both.

 

Storage Spaces only stores the data chunks in a single tier, it's migrated between them so you won't have same data chunk in 2 tiers at the same time. The exception to that is modifying data (which is technically new data) because it's written to the fast tier then replaces the expired chunk.

 

Most of the new fancy features are ReFS only though.

Link to post
Share on other sites
2 minutes ago, leadeater said:

Even 2 tier is still tiering, both tiers contribute to total capacity of the provisioned virtual disk unlike caching where the cache does not contribute. It's not really 'caching' either, it acts like caching because writes happen in the fastest tier but you want that, difference is new data will stay in the fast tier until tier high water mark or aged where caching will write the data to the underlying array as soon as possible and that data lives in both.

 

Storage Spaces only stores the data chunks in a single tier, it's migrated between them so you won't have same data chunk in 2 tiers at the same time. The exception to that is modifying data (which is technically new data) because it's written to the fast tier then replaces the expired chunk.

 

Most of the new fancy features are ReFS only though.

The default is to make faster disks "cache" aka, it doesn't count towards capacity (for 2016 and 2019). You can make it true two tier through PowerShell, which will make it reserve some capacity on the fast tier for "cache" (1GB or something you set). 

 

Yeah, most fancy features are ReFS only, like resiliency tiering and so on.

Link to post
Share on other sites
3 minutes ago, Marvin_Nor said:

The default is to make faster disks "cache" aka, it doesn't count towards capacity (for 2016 and 2019).

It does, I'm using it right now and both tiers contribute to provisioned virtual disk size. Windows Server 2016. Edit: WB cache is still set but if you enable tiering you have two actual tiers.

Link to post
Share on other sites
Just now, leadeater said:

It does, I'm using it right now and both tiers contribute to provisioned virtual disk size. Windows Server 2016.

Is that a Storage Spaces or Storage Spaces Direct setup?

The GUI will also always show how much of the cache is dedicated to the virtual disk, as if it was capacity.

Link to post
Share on other sites
13 minutes ago, Marvin_Nor said:

Is that a Storage Spaces or Storage Spaces Direct setup?

The GUI will also always show how much of the cache is dedicated to the virtual disk, as if it was capacity.

Standard not S2D, though the same happens for both last I checked (long time ago). You will also see a cache amount, default size changes on resiliency chosen, but it's not used unless the fast tier fills up and it caches the slow tier. If you have NVMe and are using S2D then you get a bit nicer caching but that's actually exclusive to S2D, standalone SS doesn't get that feature and NVMe is just another storage device type.

 

It's been ages since I've done anything with S2D though and I never actually got to do a configuration with NVMe + SSD + HDD.

 

GUI example of stand alone SS with SSD + HDD:

Spoiler

 

wdOiPm.jpg

 

yFXa8t.jpg

 

8tcUO9.jpg

 

 

Link to post
Share on other sites
Just now, leadeater said:

Standard not S2D, though the same happens for both last I checked (long time ago). You will also see a cache amount, default size changes on resiliency chosen, but it's not used unless the fast tier fills up and it caches the slow tier. If you have NVMe and are using S2D then you get a bit nicer caching but that's actually exclusive to S2D, standalone SS doesn't get that feature and NVMe is just another storage device type.

 

It's been ages since I've done anything with S2D though and I never actually got to do a configuration with NVMe + SSD + HDD.

 

GUI example of stand alone SSD + HDD:

  Reveal hidden contents

 

wdOiPm.jpg

 

yFXa8t.jpg

 

8tcUO9.jpg

 

 

Ah, right, I come from a primarily S2D Environment with SSD + HDD or NVMe + SSD + HDD, where it won't ever give you the option to define a fast tier unless it's a three tiered solution, or you define costume tiers and make a vDisk based on your own tiers. So it fully commits all fast disk as cache.

Yeah, the cache you're seeing is just a dedicated space on the fast tier, as opposed to S2D where all the fast drives will be cache, again, unless you define it otherwise.

 

Personally had bad experience with standard SS, reaching 500k IOPS is no issue, but when I've put workloads on it, the latency really starts growing (used it for backup). So, switched over to S2D for everything, 1-2M IOPS and still below 1ms latency, happy with that. 

Link to post
Share on other sites
15 minutes ago, Marvin_Nor said:

Personally had bad experience with standard SS, reaching 500k IOPS is no issue, but when I've put workloads on it, the latency really starts growing (used it for backup).

I've never used it for anything actually at work for production, trialed it on some dev hardware but we just don't have a use for it because we have Netapp and Nutanix HCI so it's firmly in the cool but why category. I use it on my servers at home because it's simple and cheap compared to other options, not having to install extra software or deploy a virtual appliance or dedicate a server just to storage is nice for a home lab.

 

It's also nice having dual 10Gb SMB multichannel to my desktop to store all my games, because why not.

 

Side note it's nice to see someone not auto bitching about Windows and it being used for storage, with a mind set of Windows Server 2003 era. Microsoft has gotten 13 million IOPs and 535GB/s seq out of S2D using 12 servers. Wish S2D was in standard edition though.

Link to post
Share on other sites
2 hours ago, leadeater said:

I've never used it for anything actually at work for production, trialed it on some dev hardware but we just don't have a use for it because we have Netapp and Nutanix HCI so it's firmly in the cool but why category. I use it on my servers at home because it's simple and cheap compared to other options, not having to install extra software or deploy a virtual appliance or dedicate a server just to storage is nice for a home lab.

 

It's also nice having dual 10Gb SMB multichannel to my desktop to store all my games, because why not.

 

Side note it's nice to see someone not auto bitching about Windows and it being used for storage, with a mind set of Windows Server 2003 era. Microsoft has gotten 13 million IOPs and 535GB/s seq out of S2D using 12 servers. Wish S2D was in standard edition though.

Yeah, we considered a lot of solutions, including Nutanix. vSAN and Starwind. But ended up on S2D due familiarity, simplicity, performance and already having it licensed due to our licensing model.

 

SMB Multichannel is great! We're using SMB Direct (RDMA), it's working wonders to be honest.

 

Yeah, Windows has come a long way with 2016 and 2019 regarding storage to be fair. And it's pushing some great performance numbers, especially with 2019 and the utilization of persistent memory as cache.

 

With that said though, just came to think of it. Linus could force the fast disk tier to behave like cache (as it would in S2D) instead of fast tier, if desired. Would loose out on the capacity, but could improve the overall performance, also shuffles data quicker in and out of the fast disks.

 

After creating the pool, you could run this little command to force the SSDs into Journal mode (cache mode).:

Get-PhysicalDisk -FriendlyName *part of SSD name* | Set-PhysicalDisk -Usage Journal

 

Then create a virtual disk with a volume:

New-Volume -StoragePoolName *Pool* -FriendlyName "Volume01" -Size 250GB -ResiliencySettingName "Mirror" -FileSystem NTFS -AccessPath "E: "-ProvisioningType Fixed

 

Should also set the column back to default (he had 6 HDDs if I am not mistaken, so 1 column per disk pair) before creating the volume.

Think this would work.

Link to post
Share on other sites

Dunno why it took me so long to think of this but why not use a distributed file system?  Each editing PC would be a storage node.  Data would be replicated across the different editing workstations.  If you feel like burning cash you could use something like Spectrum Scale, aka GPFS, which works quite well.  I've had less experience with Gluster but for the price it's a much more reasonable solution.  With either solution you could use data affinity to colocate data with where the data will be processed. 

 

Anyways just an idea.

Link to post
Share on other sites

Does anyone else see the irony here?

 

Linus talks about ZFS needing Linux experts and paying those people.

 

He proceeds to talk to an expert on the forum then still has trouble setting it up.

 

 


Can Anybody Link A Virtual Machine while I go download some RAM?

 

Link to post
Share on other sites
6 hours ago, Marvin_Nor said:

With that said though, just came to think of it. Linus could force the fast disk tier to behave like cache (as it would in S2D) instead of fast tier, if desired. Would loose out on the capacity, but could improve the overall performance, also shuffles data quicker in and out of the fast disks.

I was thinking about that as well, another one I was wondering was having 2 SSDs set Journal mode and 2+ set as Auto for Fast tier. Wonder how that would perform or if the Journal and Tiering would just sort of fight/negate each other.

Link to post
Share on other sites
10 minutes ago, leadeater said:

I was thinking about that as well, another one I was wondering was having 2 SSDs set Journal mode and 2+ set as Auto for Fast tier. Wonder how that would perform or if the Journal and Tiering would just sort of fight/negate each other.

In that case the Journal would be write and read cache for HDD, but write only cache for the two SSDs in Fast Tier. And it would shuffle data between slow and fast tier if necessary, while also caching the data. The benefit would be more capacity, and data would be retained on SSDs for a longer period of time due to the fast tier, but would be moved to SSDs faster due to the Journal (while moving data from slow to fast tier). So, could be a gain, if the file usage isn't consistent, but more sporadic. 

 

Really depends on usage I would say.

Link to post
Share on other sites
2 hours ago, unijab said:

Does anyone else see the irony here?

 

Linus talks about ZFS needing Linux experts and paying those people.

 

He proceeds to talk to an expert on the forum then still has trouble setting it up.

 

 

Don't worry, it's not like Linus actually tried most of what I said anyway ?. From the bits that were shown in the video I'd rather have just set it up, but I would have also used different hardware given the choice as well.

 

From memory Linus is using that Windows only software to copy files to an offsite server so I think that was also a factor in wanting to stick with Windows, don't actually know if that's still being used or not though.

 

Not matter what though high performance storage requires expertise to setup, how much varies on what platform you pick but there's no true easy foolproof one.

Link to post
Share on other sites
1 hour ago, leadeater said:

Don't worry, it's not like Linus actually tried most of what I said anyway ?. From the bits that were shown in the video I'd rather have just set it up, but I would have also used different hardware given the choice as well.

 

From memory Linus is using that Windows only software to copy files to an offsite server so I think that was also a factor in wanting to stick with Windows, don't actually know if that's still being used or not though.

 

Not matter what though high performance storage requires expertise to setup, how much varies on what platform you pick but there's no true easy foolproof one.

While filming I only had the initial message from you. 

 

And yeah, it's very clear that to get the best results with tiering it would absolutely be necessary to have a TON of experience with this stuff and a pretty well-equipped lab full of test gear. 

 

I think we are just going to stick to our stone-age tiering for the time being (separate servers) and we will cross the tiering bridge when we come to it again... 

Link to post
Share on other sites
24 minutes ago, LinusTech said:

While filming I only had the initial message from you. 

It's all good, I enjoyed watching it fail more, as a viewer ?.

 

24 minutes ago, LinusTech said:

I think we are just going to stick to our stone-age tiering for the time being (separate servers) and we will cross the tiering bridge when we come to it again..

That's still my pick for you guys anyway, as much as I like tiering in principle I prefer the guarantee of all SSD performance. 

Link to post
Share on other sites
1 minute ago, leadeater said:

It's all good, I enjoyed watching it fail more, as a viewer ?.

 

That's still my pick for you guys anyway, as much as I like tiering in principle I prefer the guarantee of all SSD performance. 

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

Link to post
Share on other sites
44 minutes ago, LinusTech said:

I think we are just going to stick to our stone-age tiering for the time being (separate servers)

Tiered servers sound more fun anyways ?

Link to post
Share on other sites
44 minutes ago, LinusTech said:

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

 

Anything other than cost keeping you from going with the 1.5TB 905p ? (Im assuming they are compatible with your server backplane)


Can Anybody Link A Virtual Machine while I go download some RAM?

 

Link to post
Share on other sites
2 hours ago, LinusTech said:

Yeah I just ordered some more 750 series 1.2TBs... That'll give us 40TB accessible on the fast machine. Should be enough. 

Agreed with @leadeater all flash is still the best option. But, be aware the All NVMe Supermicro Chassis you have oversubscribes the PCIe lanes(PCIe 3.0 x 16) to the Front 24 drives(Back Plane) and another(PCIe 3.0 x 16) to the mid 24 Drives(Mid Plane).

 

You're going to only see a Max of 16 GB/s Read or Write (Best case scenario but will probably be less) for each set of 24 Drives. So if you want to utilize all the performance one day its going to be time for clustering.

Link to post
Share on other sites
2 hours ago, lil_killa said:

You're going to only see a Max of 16 GB/s Read or Write (Best case scenario but will probably be less) for each set of 24 Drives. So if you want to utilize all the performance one day its going to be time for clustering.

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

Link to post
Share on other sites
34 minutes ago, Mark77 said:

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

Don't we all wish we were 'limited' to 16GB/s storage system performance though

Link to post
Share on other sites
2 hours ago, Mark77 said:

4x40gig-E or 2x100Gig-E networking gear is crazy $$$$ though to actually run into those limits...  

Yeah for sure, but you can find a Mellanox 56G switches on ebay for around $2000 and the 56G ConnectX3 cards for around $200 ish. 

Link to post
Share on other sites

That wasn't a joke... I'm serious about this... try Intel CAS... I think that's providing what you want... or at least it would be worth a try and maybe you can create another video from this experiment... and it's not that expensive (even free if you use Intel ssds)

And by the way, it also answers another question you had on a board... "who is ever using 4 U.2 drives in one system??" ... well, the guy that's using those ssds as a cache

Link to post
Share on other sites

This video was very interesting as I did something similar sometime ago with Linux.

I had a slow RAID6 array of HDDs and a very fast RAID10 of SSDs.

By setting up everything with LVM I could use the fast RAID10 array as an LVM cache in Writeback mode.

With that, data I moved inn goes straight to the SSDs first, LVM then syncs the data down to the slower array.

This was very nice as I could transfer very large DNXHD encodes over 10gbit super fast and know they would stay on the SSD cache as I worked on some silly vacation video.

My SSDs were cheap TLC crap so the write speeds went to shit after a while, please don't use TLC drives for this ?.

 

LVM cache also has a Writethrough mode that writes to both slow and fast device at once, and is safer.

In either caching modes, all new data written enters the cache.

 

Red Hat started to support this back in 2016 so it's quite battle tested.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×