Fold8 - 2-slot Folding Rig

[Gorgon]

I've been building 2-slot (GPU) folding rigs in Fractal Design R4 and S chassis as I like these voluminous chassis with their discreet looks. I'm not a fan of windowed cases. I can monitor my rigs from afar and would rather have a rigid side than a piece of plexi to get scratched. I'm old school that way and as long as the chassis is easy to work in I don't care if the cables are "ketchup and mustard"

 

The Standard S or R Fractal chassis also consume a lot of real estate so I'm planning a new rig and looking at using either the Meshify or C.

 

My current rigs use Noctua 140mm iPPC3000 PWM fans at the front to get better static pressure through the Define faceplate as well as one mounted in the bottom to feed intake air into the bottom GPU. I have found testing that this bottom fan provides a significant decrease in the lower cards GPU temperature allowing me to run at higher boost speeds and yielding more Points per Day (PPD). For the exhaust fans I use 140mm Noctua NF-A14s in the rear and the three top positions.

 

Typically I've set the motherboard fan controls to manage the upper front fan and rear top fan by CPU temperature just to provide some airflow when not folding. The rest of the fans are controlled using a custom curve off the temperature sensor on the Gigabyte motherboard's upper PCIe3x16 slot. In my current rig I've found a steep slope between 30% at 42C and 70% at 48C slot temperatures give me a quite system when folding at low-power setting (running the GPUs at their minimum power limit) and adequate airflow when folding full-tilt (running the GPUs at their maximum power limit).

 

Tech Jesus Steve at Gamers Nexus in a recent post about noise normalized thermals got me thinking about moving from the Define R or S series to a more open front design to see if I could achieve similar noise results from slower running, smaller fans in the Define C or Meshify chassis.

 

The kicker, as I mentioned earlier, is both these cases have power supply shrouds which will impeded forcing air up from the bottom of the chassis into the intake of the lower GPU.

 

Comparing the Meshify and the C one significant difference which I'm not keen on is the foam dust filter on the Meshify. That looks like it could be a major PITA to clean. The C, on the other hand, has the standard Define front plate but a nice mesh grill over top of the fans.

 

So my intention is to acquire a Define C when they next are on sale and run it with the cover on and off observing the temperatures while folding. The next step would be to drill out the rivets and remove the power supply shroud and see what impact that has on cooling.

 

Here is the current PC Partpicker build list. You may notice that with the front cover removed I'm moving from the High Static Pressure iPPC3000 PWM input fans to 3 120mm NF-P12 1700rpm PWM Redux on the front and a single 120mm NF-F12 iPPC 2000 PWM on the bottom. The rear exhaust will also be a 120mm NF-P12 1700rpm PWM Redux and the two top chassis exhaust fans will be 140mm NF-P14s 1500rpm PWM Redux.

 

The motherboard & CPU are not yet set in stone. The principal requirements are 2 PCIex16 slots spaced 3 slots apart with the last slot not obscuring i/o connectors and both slots running using 8 CPU lanes. The CPU requirements are a single core/thread for each GPU and one for the OS yielding either a 4C/4T or 2C/4T requirement. The 3.7GHz Intel Pentium Gold G5400 in my last build has proven more than capable of keeping it's GTX 1070 Ti and GTX 1060 well fed with work.

 

In this build I am looking at the slightly less expensive Gigabyte Z370XP SLI over the Gigabyte Aorus Z370 Gaming 5 I currently use. I went with the gaming 5 based on the excellent review it had a Tweak Town and the Z370XP SLI Likewise has been favorably reviewed there. One of the main reasons I went with the Gaming 5 was it's inclusion of two external 10K Thermistors for measuring temperature. I used one to measure Ambient and the other measured the exhaust of the upper GPU. After testing extensively I confirmed that the on-board PCIe3_x16 temperature sensor was capable of driving the cooling for the GPUs.

 

The only thing I had to struggle with on the Gigabyte boards was due to their use of ITE Super I/O controller I had to build an experimental module in Linux to be able to access the Fan speeds, Voltages and Temperatures for system management.

 

I am open to an AMD build. I'm just not as familiar with the "Gotcha's" on the AMD AM4 platform specifically the CPU PCIe lane limitations on Bristol, Athlon and Zen architectures. The main requirement is to get a 2.8 - 3.5GHz 4C/T system with 2 PCIe3 x16 slots running 8 lanes each

from the CPU so any advise would be appreciated.

 

As for memory, Folding requires only a gig or two and these systems will be running Ubuntu 18.4.1 LTS Server so 2x 2GB DDR4 would be ideal if such kits existed.

 

 

 

[palespartan]

Not sure on what motherboards will be best spacing wise for the GPU's nor am I sure of what budget you are looking to spend on that part of the setup but if this is going to dedicated to folding and remote and completely headless maybe a board with a drac would be a good thing to look for. The asus WS boards usually have one but that might be out of budget.

 

Edit: mining boards also have the same thing I'm personally running a biostar TB85 right now I think it has Drac functionality but haven't taken the time to try to set it up yet. (it only has one 16x slot though so keep that in mind if looking for mining boards)

[Gorgon]

2 minutes ago, palespartan said:

Not sure on what motherboards will be best spacing wise for the GPU's nor am I sure of what budget you are looking to spend on that part of the setup but if this is going to dedicated to folding and remote and completely headless maybe a board with a drac would be a good thing to look for. The asus WS boards usually have one but that might be out of budget.

Here's the list for Z370:

Spoiler

ASRock    Z370 Killer SLI
ASRock    Z370 Extreme4
ASRock    Fatal1ty Z370
ASRock    Z370 Taichi
ASRock    Z370 Professional Gaming i7
Asus        TUF Z370 Pro Gaming
Asus        Strix Z370-H
Asus        Prime Z370-A
Asus        ROG Strix Z370-F Gaming
Asus        ROG Strix Z370-E
Asus        ROG Maximus X Hero (WiFi AC)
Asus        ROG Maximus X Hero
Asus        ROG Maximus X Code
Asus        Maximus X Formula
BioStar    Z370GT6
EVGA      Z370 FTW
EVGA     Classified K
Gigabyte    Z370XP SLI (rev. 1.0)
Gigabyte    Z370 Aorus Ultra Gaming
Gigabyte    Z370 Aorus  Gaming WiFI
Gigabyte    Z370 Aorus Gaming 5
Gigabyte    Z370 Aorus Gaming 7
MSI    Z370 SLI PLUS
MSI    Z370 Krait Gaming
MSI    Z370 Gaming Pro Carbon
MSI    Z370 Gaming M5
MSI    Z370 Gaming Pro Carbon AC
SuperMicro    MBD-C7Z370-CG-L-O

Out-of-Band is not as much of an issue. I can and do most of the management in-band using SSH. Most of the monitoring I can do from the FAHControl and I have HFM.net set up for Folding production and Zabbix for OS & Hardware Health monitoring.

 

I would prefer to use SuperMicro boards with their IPMI as I'm very familiar with it from work and home but their current lineup doesn't have a good dual-slot board in Socket 1151 in the X10 or X11 series so I'm pretty much stuck with Socket 1151 or AM4 to keep the costs down.

 

I've done some cost analysis and for the price of 2 3-slot X99 systems I can easily build 3 2-slot systems using Socket 1151 or AM4 and have cash left over to spend on GPUs.

[palespartan]

11 minutes ago, Gorgon said:

Here's the list for Z370:

  Hide contents

ASRock    Z370 Killer SLI
ASRock    Z370 Extreme4
ASRock    Fatal1ty Z370
ASRock    Z370 Taichi
ASRock    Z370 Professional Gaming i7
Asus        TUF Z370 Pro Gaming
Asus        Strix Z370-H
Asus        Prime Z370-A
Asus        ROG Strix Z370-F Gaming
Asus        ROG Strix Z370-E
Asus        ROG Maximus X Hero (WiFi AC)
Asus        ROG Maximus X Hero
Asus        ROG Maximus X Code
Asus        Maximus X Formula
BioStar    Z370GT6
EVGA      Z370 FTW
EVGA     Classified K
Gigabyte    Z370XP SLI (rev. 1.0)
Gigabyte    Z370 Aorus Ultra Gaming
Gigabyte    Z370 Aorus  Gaming WiFI
Gigabyte    Z370 Aorus Gaming 5
Gigabyte    Z370 Aorus Gaming 7
MSI    Z370 SLI PLUS
MSI    Z370 Krait Gaming
MSI    Z370 Gaming Pro Carbon
MSI    Z370 Gaming M5
MSI    Z370 Gaming Pro Carbon AC
SuperMicro    MBD-C7Z370-CG-L-O

Out-of-Band is not as much of an issue. I can and do most of the management in-band using SSH. Most of the monitoring I can do from the FAHControl and I have HFM.net set up for Folding production and Zabbix for OS & Hardware Health monitoring.

 

I would prefer to use SuperMicro boards with their IPMI as I'm very familiar with it from work and home but their current lineup doesn't have a good dual-slot board in Socket 1151 in the X10 or X11 series so I'm pretty much stuck with Socket 1151 or AM4 to keep the costs down.

 

I've done some cost analysis and for the price of 2 3-slot X99 systems I can easily build 3 2-slot systems using Socket 1151 or AM4 and have cash left over to spend on GPUs.

If you do want supermicro look back at their atx boards for the z270 gen and look into 7th gen Pentium. That should work pretty well.

 

http://www.supermicro.com/products/motherboard/Core/index.cfm?mlg=0

[TVwazhere]

3 hours ago, Gorgon said:

So my intention is to acquire a Define C when they next are on sale and run it with the cover on and off observing the temperatures while folding. The next step would be to drill out the rivets and remove the power supply shroud and see what impact that has on cooling.

This should almost definitely help the cooling of the lower GPU. Cases like the NZXT H700 have ventilation holes through the top of the PSU shroud which GN showed to have an improvement on GPU temps even with only a single card over a closed off (taped up) shroud. 

 

The foam isnt terrible to clean out, but since it's "permanently" in the front panel (It's possible to disassemble the front panel entirely), it is a bit easier to clean the mesh filter on the Define C, and theoretically with similar fans, temps shouldnt be much higher in the define C over the Meshify C. 

 

According to PCPPer there are no 2x2GB DDR4 kits on the market (unless you're talking SODIMM)

[Gorgon]

1 hour ago, palespartan said:

If you do want supermicro look back at their atx boards for the z270 gen and look into 7th gen Pentium. That should work pretty well.

 

http://www.supermicro.com/products/motherboard/Core/index.cfm?mlg=0

Yeah,

 

I haven't tried the SuperMicro "Enthusiast" Boards yet but I'd like too. Though reviews suggest their Fan Control needs work so that is a big concern as is the location of the thermal sensors on the board. I almost pulled the trigger on their Z370 board but went with the Gigabyte instead. The Z270 and even the Z370 SuperMicro boards are getting hard to find now that their Z390 has been announced.

 

The Supermicro Z370 does have a header for an external temperature probe that you can use as a control for fans but the board doesn't come with a 10K NTC thermistor included and those can be a pain to source in North America at a reasonable price.

 

What I would like is a Socket 1151 Supermicro Server Grade board with an Intel Server chipset, IPMI and 2 PCIe3 x16 slots 3-slots apart. That I would gladly pay $50 - $100 over the current mid-tier gaming boards as I have much better faith in SuperMicro to design something to run 7x24x365.

 

It just kind of irritates me to be getting WiFi, Dual NVMe, Smokin' sound and all these other features that I'll never use in these systems and have to incur the added complexity they bring.

[Gorgon]

1 hour ago, TVwazhere said:

This should almost definitely help the cooling of the lower GPU. Cases like the NZXT H700 have ventilation holes through the top of the PSU shroud which GN showed to have an improvement on GPU temps even with only a single card over a closed off (taped up) shroud. 

 

The foam isnt terrible to clean out, but since it's "permanently" in the front panel (It's possible to disassemble the front panel entirely), it is a bit easier to clean the mesh filter on the Define C, and theoretically with similar fans, temps shouldnt be much higher in the define C over the Meshify C. 

 

According to PCPPer there are no 2x2GB DDR4 kits on the market (unless you're talking SODIMM)

I'm planning on running the "C" without the front grill but with the dust filter (These systems do, after all, live in the basement) The "C" and "Meshify", aside from the front grill are essentially the same chassis. I like that they have lots of air holes at the back because there will be lots of positive pressure.

 

Its just the damned power supply shrouds.

 

I've done a lot of looking at the pictures at various sites and it looks like if I remove 7 or 8 rivets I can get the shroud out and hopefully the bottom middle of the motherboard tray wont be too wobbly as a result.  Worst case I'll bribe  one of the machinists at work to bend me a bracket.

 

Yeah, I know about the 2GB kits.

[palespartan]

1 hour ago, Gorgon said:

Yeah,

 

I haven't tried the SuperMicro "Enthusiast" Boards yet but I'd like too. Though reviews suggest their Fan Control needs work so that is a big concern as is the location of the thermal sensors on the board. I almost pulled the trigger on their Z370 board but went with the Gigabyte instead. The Z270 and even the Z370 SuperMicro boards are getting hard to find now that their Z390 has been announced.

 

The Supermicro Z370 does have a header for an external temperature probe that you can use as a control for fans but the board doesn't come with a 10K NTC thermistor included and those can be a pain to source in North America at a reasonable price.

 

What I would like is a Socket 1151 Supermicro Server Grade board with an Intel Server chipset, IPMI and 2 PCIe3 x16 slots 3-slots apart. That I would gladly pay $50 - $100 over the current mid-tier gaming boards as I have much better faith in SuperMicro to design something to run 7x24x365.

 

It just kind of irritates me to be getting WiFi, Dual NVMe, Smokin' sound and all these other features that I'll never use in these systems and have to incur the added complexity they bring.

I'm sure they have some bare bones 4u server chassis that would be compatible with 1151 xeon chips but that would probably be same budget as going x99 anyways. If you don't care about all the gaming features then I would drop to a lower chipset like the H series or something. Its not like Intel lets you overclock a celeron anyways.

1 hour ago, Gorgon said:

I'm planning on running the "C" without the front grill but with the dust filter (These systems do, after all, live in the basement) The "C" and "Meshify", aside from the front grill are essentially the same chassis. I like that they have lots of air holes at the back because there will be lots of positive pressure.

 

Its just the damned power supply shrouds.

 

I've done a lot of looking at the pictures at various sites and it looks like if I remove 7 or 8 rivets I can get the shroud out and hopefully the bottom middle of the motherboard tray wont be too wobbly as a result.  Worst case I'll bribe  one of the machinists at work to bend me a bracket.

 

Yeah, I know about the 2GB kits.

This entire month Ive been running my define s without its front cover on for the added airflow. It works really well. If you want any other specs to help you out I don't mind taking measurements or anything. Another really easy thing to do would be to take the front cover for the define S and cut the front out of it and replace it with a mesh screen or something. Would be a really easy mod since its all plastic and a completely open front with no 5 1/4" bays.

[Gorgon]

Found and ordered an Gigabyte Aorus Z370 Ultra Gaming 1.0 for $132Cdn

[Gorgon]

On 11/19/2018 at 10:35 PM, palespartan said:

I'm sure they have some bare bones 4u server chassis that would be compatible with 1151 xeon chips but that would probably be same budget as going x99 anyways. If you don't care about all the gaming features then I would drop to a lower chipset like the H series or something. Its not like Intel lets you overclock a celeron anyways.

This entire month Ive been running my define s without its front cover on for the added airflow. It works really well. If you want any other specs to help you out I don't mind taking measurements or anything. Another really easy thing to do would be to take the front cover for the define S and cut the front out of it and replace it with a mesh screen or something. Would be a really easy mod since its all plastic and a completely open front with no 5 1/4" bays.

I'm running my main rig in a define S and find that works well but the fans are quite loud when running both cards full-blast. My 2nd folding system is in a Define R4 but it's only a single card system as it has an old ITX motherboard

 

I'm going to be picking up a CPU, ATX SLI-capable motherboard and some RAM over the next couple of weeks. Once I have all the parts I'm going to install them in the R4 for now until I can find a C on sale at a reasonable price. They're currently around $105Cdn but are often on sale for $70-$75.

 

I'll also have to buy a metric ton of 120mm Noctua Fans. I have many 140mm iPPC 3000 PWM fans but I want to try out some slower 120mm fans, specifically the NF-A12x25 PWM model which seems to have a good balance between a pressure and airflow optimized design.

 

Tentative placement is:

• 3 x NF-A12x25 PWM Front Intake SYS_FAN3 via 3-way splitter
• 1 x  NF-F12 iPPC 2000 PWM Bottom Intake SYS_FAN2A
• 1 x NF-F12 iPPC 2000 PWM Rear Exhaust SYS_FAN1
• 2 x NF-A14 PWM Top Exhaust CPU_OPT via 2-way splitter

All fans will be controlled by the temperature sensor beside the PCIEX16 Slot which is the upper slot. Like fans are grouped on the same fan headers and will have custom fan curves. I'm going with the iPPC-2000 PWM rather than the 3000 rpm variants to keep noise down and only using one on the bottom intake where I'll need more static pressure to get through the honeycomb case holes and also at the rear exhaust where I'll want more airflow.

 

Only the CPU fan will be controlled by the CPU temperature sensor as the CPU will only be generating heat when folding so the airflow called for by the PCIEX16 sensor will be adequate for keeping the CPU cool.

[Gorgon]

On 11/27/2018 at 8:40 PM, Gorgon said:

Found and ordered an Gigabyte Aorus Z370 Ultra Gaming 1.0 for $132Cdn

Turned out that the vendor (an on-line retailer in Canada) showed the item in Stock they did not have stock. They offered to sell me a version Aorus Z370 Ultra Gaming 2.0 for $100 more. I declined.

If I was suspicious I would suspect that they were engaged in bait-and-switch.

Curiously they didnt mention that also had stock on the Gigabyte - Z370XP SLI (rev. 1.0) ATX LGA1151 which is what I ended up ultimately ordering but from another vendor that was slightly more expensive but I've had good luck with in the past.

[Gorgon]

Well it only took five days for Purolator to get the motherboard from NYC to Toronto so hopefully it will arrive tomorrow.

 

Still no luck sourcing a Pentium G5400 at a reasonable cost ($90Cdn) in Canada. Most site show no stock due to the shortage of 14nm Intel chips so it is looking like I’ll be forced to go with a G5500 at $120.

 

The Celeron G3930 is on sale currently at $40Cdn but with only 2C/2T at 2.9GHz won’t have enough Jam to drive two Pascal or Turing cards even under Linux. It would, however, be up to the task in a B or H series board for driving a single Pascal or Turing GPU under Linux but I expect the utilization would be over 50% due to its lower clock speed and the OS overhead.

 

As a test I loaded up two threads on my G5400 with and infinite loop and immediately saw the CPU go from 54 to 100% utilization and a slight decrease in PPD.

[Gorgon]

Picked up a Pentium Gold G5500 and a 128GB Adata NVMe SSD.

 

System assembled using 1 stick of GSkill RipJaws V DDR4 2400 from my other folding Rig.

 

Installed Windows 10 then Ubuntu and compared folding out of a chassis and found no difference in performance using a EVGA GTX 1060 6GB Short-Board.

 

Components installed in Fractal Design R4 Chassis with a EVGA RTX 2070 XC until I can source a Define C on sale.

 

2 Noctua 140mm iPPC 3000 PWM Fans as Front Intake (SYS2)

1 Cougar 120mm Vortex CF-V12HPB PWM Fan as Bottom Intake (SYS3)

2 CoolMaster 140mm Masterfan Pro Fans as Top Exhaust (CPU_OPT)

1 Noctua 140mm iPPC 3000 PWM Fans as Rear Exhaust (SYS1)

 

All Fans tied to PCIex16 motherboard sensor

 

Fan curves adjusted for positive case pressure. Intake speeds increased over those of Exhaust speeds.