Fire Simulation Rig : Permanently on, rarely restarted.

[GeekFixUK]

So here it goes. good evening guys from sunny England. i wanted to see what other peoples thoughts are on this particular rig my client uses. They currently have 4 rigs, year on year getting a further one, generally more and more powerful each time. My clients use of the system is quite unique. They are generally not switched off much and this year they are starting to increase the budget into workstation/server boards. Therefor they accept a certain risk of hardware failure and are happy to work with manufacture warranties but i want better than that, im hoping to give them more reliability at the core count they ask or more. Each simulation year on year with their newer systems has managed to reduce the length of time per run however it is still at around 5 to 7 days. During the run the cpu % is pegged at a certain level for the whole run, set by the user. With water-cooling this temp on the cpu stays around 50 degrees c or lower, never higher. Aside from that ram usage depends on the size of data set and is easily covered with 32gbs of ddr 3 or 4 ram depending on the system.

 

So a little background, the client uses the rigs for fire simulation on architectural drawings digitally recreated, and with the recent London fires they are very very busy and cant risk motherboards burning out or else such hardware. Last week an asus x99 board stopped booting and is being RMA'ed as we speak being less than a year old. The software is quite unique and specific in its hardware requirements but the biggest hurdle is that it cannot utilize hyper-threading, therefor hyper-threading is an accepted waste SO in short the computers currently used are super high end gaming rigs with no graphics card lol. Well i lie a little, they have pathetic little cards (£30 asus single slot cards) to provide monitor connection if and when the remote software has issues as most of the Intel cpu's they use have no dedicated graphics. But understandably they want and need more cores and have come back with the newest request whilst i deal with the RMA. They want to use the intel i9 but with workstation/server grade motherboards but i cant seem to find any decent Asus x299 workstation/server boards so here is my idea:

 

 Using an Intel Core i9-7900X 3.3GHz (Core 10/20) (Skylake X / Basin Falls) Socket LGA2066 Processor with an Asus TUF X299 Mark1 (Socket 2066) Intel X299 ATX Motherboard. Cost of cpu around £950 and mobo around £280.

 

The other choice, if the client wants server/workstation boards is to use the Z10PE-D8 WS Xeon Series S2011 v3 Asus Motherboard with Intel Xeon E5-1650v4 or Intel Xeon E5-2630v4. Mobo cost around £450-500 and cpu cost £630 or £680 depending on the Dual Xeons used. This would give them either 12 or 20 physical cores to use and not much more of a price. 

 

Either way the system will have 32gb of corsair ram, 240gb Kingston Savage SSD Drive with a cold storage WD Red 2TB and corsair psu.

 

What are peoples thoughts on longevity and reliability with the above plan? The clients systems are generally decommissioned and upgraded every 3-4 years but in that time they want as little downtime as possible without the use of redundant hardware as the budget wont go that far yet.

 

[AshleyAshes]

What kind of software is multi-threaded but can't use hyperthreading?

[Lurick]

You should either go with server grade components and Xeon processors or just order some Dell R630 or R730 servers.

 

Can the software at least use multiple processors?

[PeachyUwUSenpai]

why not just make 2 or 3 ryzen setups at close to the same cost then if one fails you still have 1-2 others

[tlink]

why do they prefer i9 over threadripper? is it better optimized for intel and single core performance?

[GeekFixUK]

3 minutes ago, Lurick said:

You should either go with server grade components and Xeon processors or just order some Dell R630 or R730 servers.

 

Can the software at least use multiple processors?

Yes, i believe the software can utilize multi CPU's. The client just want to achieve a decent physical core count without costing the earth that lasts in a system a decent amount of time. Im just really running ideas and seeing what peoples thoughts are.

[GeekFixUK]

2 minutes ago, tlink said:

why do they prefer i9 over threadripper? is it better optimized for intel and single core performance?

What is the heat output like on decent load? Client prefers Intel over AMD after years of use and therefor until Ryzen never considered AMD, also the rigs are kept in their office so they must run fairly quiet.

[pyrojoe34]

If you're looking for a system that runs 24/7, is reliable, and is low maintenance, you should go with server hardware. Ditch the water cooling (not a good idea on a 24/7 system and a terrible idea if you want low maintenance and no risk of water damage. Good airflow and a heatsink should be plenty of cooling and basically zero maintenance or risk). Get a single or dual Xeon system, ECC RAM, and a redundant RAID array (I'm not super familiar with RAID but from what I understand RAID 5 is the way to go). You can get good prices on hardware that's just a few years old. You should be able to get a 12-24core server that's a few years old for <$1000, for example someone here is selling an R610 with dual quad core X5560 Xeons (8 cores, 16 threads total) with 16GB of ECC RAM and 3x600GB SAS drives in a 2U rack for $150 on CL. Threadripper is also a good choice as it's many cores and ECC compatible for a much cheaper price than Intel.

[tlink]

5 minutes ago, GeekFixUK said:

What is the heat output like on decent load? Client prefers Intel over AMD after years of use and therefor until Ryzen never considered AMD, also the rigs are kept in their office so they must run fairly quiet.

we're not sure yet since it isn't released yet. its just around the corner, august i believe is the launch date. even if the heat output is much worse, it has much  much more cores for a lower price. ryzen is the go-to platform for core's, threadripper will be too. 16 core cpu will cost 1000$ vs 16 cores i9 for 1700$. it is probably a little behind in clocks per instructions but the vast price difference should make it worth it.

[porina]

14 minutes ago, AshleyAshes said:

What kind of software is multi-threaded but can't use hyperthreading?

Anything strongly using AVX2. HT does not magic up any new resources if you can max out the performance on a single thread, which is not difficult to do. If this applies here, Ryzen family suck at AVX2. Intel all the way if that is the case.

[tlink]

1 minute ago, porina said:

Anything strongly using AVX2. HT does not magic up any new resources if you can max out the performance on a single thread, which is not difficult to do. If this applies here, Ryzen family suck at AVX2. Intel all the way if that is the case.

yea, this. if it uses instruction sets that are much better on intel than _probably fuck threadripper.

[GeekFixUK]

4 minutes ago, pyrojoe34 said:

If you're looking for a system that runs 24/7, is reliable, and is low maintenance, you should go with server hardware. Ditch the water cooling (not a good idea on a 24/7 system and a terrible idea if you want low maintenance and no risk of water damage. Good airflow and a heatsink should be plenty of cooling and basically zero maintenance or risk). Get a single or dual Xeon system, ECC RAM, and a redundant RAID array (I'm not super familiar with RAID but from what I understand RAID 5 is the way to go). You can get good prices on hardware that's just a few years old (you should be able to get a 12-24core server that's a few years old for <$1000. Threadripper is also a good choice as it's many cores and ECC compatible for a much cheaper price.

 

It seems strange that you can use multiple cores but not hyperthreading. I feel like most software can't tell the difference between a HT and a core from a resource allocation perspective.

Thanks, i wholeheartedly agree that its bizarre from my seat why the software cannot utilize hyperthreading. RAID wouldn't be needed as they don't store any data on it aside from preprocessing, after that data is pulled off to their own systems for analysis. During processing there is barely any hard drive activity, aside from results being transferred from RAM where the majority of the data set lives. That data transfer is minute and doesnt exceed the ssd bandwidth.

[GeekFixUK]

5 minutes ago, porina said:

Anything strongly using AVX2. HT does not magic up any new resources if you can max out the performance on a single thread, which is not difficult to do. If this applies here, Ryzen family suck at AVX2. Intel all the way if that is the case.

 

3 minutes ago, tlink said:

yea, this. if it uses instruction sets that are much better on intel than _probably fuck threadripper.

 Thank you both for that. Would of been a costly error and will bear it in mind.

[porina]

Just now, GeekFixUK said:

 Thank you both for that. Would of been a costly error and will bear it in mind.

We don't know the software but it would be worth testing it to find out. Conversely if it isn't the case, Ryzen might offer a lot of performance for not a lot of money. Testing only way to be sure.

[pyrojoe34]

5 minutes ago, GeekFixUK said:

Thanks, i wholeheartedly agree that its bizarre from my seat why the software cannot utilize hyperthreading. RAID wouldn't be needed as they don't store any data on it aside from preprocessing, after that data is pulled off to their own systems for analysis. During processing there is barely any hard drive activity, aside from results being transferred from RAM where the majority of the data set lives. That data transfer is minute and doesnt exceed the ssd bandwidth.

Even if you don't need the extra speed of a striped RAID it would be a good idea to have some sort of redundant mirrored RAID within the system to protect the (presumably) valuable data against hardware failure. Maybe even just a simple dual disk RAID 1. In addition to that make sure they have a secondary backup drive (like an external drive or backup NAS) that they can regularly backup their data to.

[GeekFixUK]

1 minute ago, pyrojoe34 said:

Even if you don't need the extra speed it would be a good idea to have some sort of redundant RAID within the system to protect the (presumably) valuable data against hardware failure. Maybe even just a simple dual disk RAID 1. In addition to that make sure they have a secondary backup drive (like an external) that they can regularly backup their data to.

I see what you mean, sorry i should of explained (not my best skill lol, i fix and build sh*t). The WD 2tb Red in the systems are to hold data as the result comes off through the 5-7 day run. If the process is interrupted the whole data set fails and needs to be started anyway, so redundancy for data "in system" is not necessary. Once the simulation is done they pull the data off onto a Synology 4 disk Raid NAS.

[tlink]

1 minute ago, porina said:

We don't know the software but it would be worth testing it to find out. Conversely if it isn't the case, Ryzen might offer a lot of performance for not a lot of money. Testing only way to be sure.

since the software is so specialized they should also consult with the client and ask if they still have a contact with the developer of the software and ask the dev what instructionset it uses, i would still test it on the actual hardware though. just an extra layer of security and dedication that the client notices.

[GeekFixUK]

1 minute ago, tlink said:

since the software is so specialized they should also consult with the client and ask if they still have a contact with the developer of the software and ask the dev what instructionset it uses, i would still test it on the actual hardware though. just an extra layer of security and dedication that the client notices.

Just sent him a message asking for the software name, in hind sight it would of helped lol

Once i have it ill update you guys but the post is generally more about getting max reliability without having spare rigs or redundant components, they accept a certain failure rate over a given time period, i just want that time period to be as long as possible.

[GeekFixUK]

The software is FDS (Fire Dynamics Simulator) from the US National Institute of Science and Technology. 

 

Link:

 

https://pages.nist.gov/fds-smv/

[Yoinkerman]

Uh, have you considered just running the tests redundantly?

 

 

[Lurick]

40 minutes ago, GeekFixUK said:

Yes, i believe the software can utilize multi CPU's. The client just want to achieve a decent physical core count without costing the earth that lasts in a system a decent amount of time. Im just really running ideas and seeing what peoples thoughts are.

Good point, if they're okay with somewhat used but still good shape hardware then I would suggest looking at Dell R620 boxes on ebay or the likes. If they need advanced replacement and the best warranty then I would suggest going brand new from Dell themselves. Most of the cost will come down to the Xeons and memory, most everything else is just extra. I think brand new an R630 with 14C/28T Xeons and 32GB of RAM is looking round $4,000 which is a tad pricey but it does come with all the warranties and whatnot. By comparison I see a few used 2x 8-Core R620s with 64GB of RAM (give or take) for around $800. 

[GeekFixUK]

2 minutes ago, Lurick said:

Good point, if they're okay with somewhat used but still good shape hardware then I would suggest looking at Dell R620 boxes on ebay or the likes. If they need advanced replacement and the best warranty then I would suggest going brand new from Dell themselves. Most of the cost will come down to the Xeons and memory, most everything else is just extra. I think brand new an R630 with 14C/28T Xeons and 32GB of RAM is looking round $4,000 which is a tad pricey but it does come with all the warranties and whatnot. By comparison I see a few used 2x 8-Core R620s with 64GB of RAM (give or take) for around $800. 

OK, thanks great info. Now with 4k Dollars is budget worthy. They do prefer new as each new rig they get they want to go more and more. The last rig was 6 cores, theyve asked for 10 core or more but dont want to spend 4k just on a cpu. the main reason i believe they havent gone full rig from a manufacturer is speed of service. IF any issues arise, having the warranty on each part allows me to get them up and running 24-36hrs. Dell or any big manufacturer would be a week or longer with hurdles of shipping, losing data etc etc. Me, im onsite same day

[porina]

42 minutes ago, GeekFixUK said:

The software is FDS (Fire Dynamics Simulator) from the US National Institute of Science and Technology. 

 

Link:

 

https://pages.nist.gov/fds-smv/

Just skimmed through the docs. No clues as to optimal CPU choice but it looks like it can support clustering, using more than one system on the same job. It is beyond me, but may be something to look at.

[GeekFixUK]

18 minutes ago, porina said:

Just skimmed through the docs. No clues as to optimal CPU choice but it looks like it can support clustering, using more than one system on the same job. It is beyond me, but may be something to look at.

Yeah not found much so far on optimal cpus however i have come across this:

 

http://www.sabalcore.com/vertical-markets/engineering/fds-smv/?gclid=CjwKCAjw47bLBRBkEiwABh-PkeHFKQR9VVg7m8wc-ucuKRIcurzVCS1iKZ_32yjD71_2sdt76w18RhoCKoYQAvD_BwE

 

who are using xeon x5 cores. The clusters is beyond me too and im pretty damn certain that its beyond the companys business costing and planning.