Bifrost a 6 GPU Folding Machine

[Gorgon]

On 11/27/2019 at 9:19 AM, Mbowen said:

What single slot card were you using in the first picture?

Sorry, I seem to have notifications disabled. That was a Radeon HD5670

[j1philli]

This is some great info @Gorgon thank you. I will most likely be using this as my template for my next build!

[pducharme]

I have 2 Mining Frame Veddah, but Sold all my 1070s and 1060s that I was using for mining in 2017-18   I only Fold on a 1080 ti and a 1060 6GB i kept... looking at that is tempting to get back into this kind of Rigs, but for F@H instead of shitcoins mining   Only issue, is i'll keep the $$ available in case the covid crisis requires me to buy some stuff  

[Nine Tailed Fox]

840k ppd seems kinda low...

[Gorgon]

On 11/18/2019 at 11:52 AM, Gorgon said:

Had an issue where 8192MB RAM was displayed in the BIOS but the OS only detected 4GB. Had to install both sticks on one channel to get it recognized. Further troubleshooting will be required to determine if this is an issue with the memory controller on the CPU or if there is a bad slot on the motherboard or an issue with the memory timings/terminations/voltages. It was working well under Linux and appears to have become an issue when the CPU was swapped. ...

Update.

 

Swapped out the 2700 today for a 3900x to get more threads for folding and BOINC. It was a tight squeeze getting the stock Wraith cooler installed with the PCIe riser cable coming off the upper m.2 adapter but it fit and the extra bit of bend in the cable doesn’t seem to be impacting the attached GPU.

 

The bad news is, however, that it appears there is an issue with the motherboard and even with a new CPU the second memory bank slots still aren’t working.

 

So I’ll have to wait until I replace the motherboard for a x570 Gaming X and then RMA the motherboard with GigaByte.

[Gorgon]

Update: It appears there are cleaner solutions for breaking-out a Bifurcated PCIe3 x16 slot to 4 PCIe3x16 slots at x4 now. These will likely be less expensive than a Quad-port m.2 NVMe adapter and 4 m.2 to PCIe3 x16 adapters.

 

Kudos to @LazyDev for finding these.

 

 

[LazyDev]

2 hours ago, Gorgon said:

Update: It appears there are cleaner solutions for breaking-out a Bifurcated PCIe3 x16 slot to 4 PCIe3x16 slots at x4 now. These will likely be less expensive than a Quad-port m.2 NVMe adapter and 4 m.2 to PCIe3 x16 adapters.

Just to give an update. Ironicaly, I saw your build on how to bifrucate the PCIE to 4x's, but after looking around, It came apparent that it's expensive going down the M.2 to PCIe conversion, and in some cases, hard to get a hold of the ADT adapters to do so. I had to look outside of the box to think of cheaper, reliable solutions. Even looking into enterprise stuff for inspiration. The MaxCloudOn has not failed me in the months of using, that with 24/7 use. I've not had any dropped GPU's or bandwidth issues. 

[Gorgon]

If I was to do this again I’d likely use a Spotswood Wide Open-frame Mining Tray for Two PSUs rather than the Veda mining frame as the Spotswood has bars for each side of the riser PCBs and would support the GPUs better.

 

Additionally I’d use the Maxoncloud riser to replace the 4 x4 NVMe adapter and 4 of the ADT NVMe to PCIe3 x16 adapters but two of the ADT NVMe adapters would still be required to break out the two PCIe x4 NVMe slots on the motherboard to PCIe3 x16 slots.

 

Gigabyte Aorus x570 Gaming X Motherboard

AMD Ryzen 3 3100 4c/8t CPU (minimum 1t per GPU)

2 x 8GB DDR4 DRAM (2400MHz or faster) (1.5GB per GPU)

60-120GB SATA SSD for OS(es)

2 x 850 or 1000W Power Supplies

Thermaltake Dual PSU Adapter

Additional PCIe Power Cables for GPUs and Maxoncloud Riser PCBs

2 x Molex to 4-pin Floppy Power Adapters for ADT Riser PCBs

Spotswood Wide Open-Frame Mining Frame for 2 PSUs

5 x 120 or 140mm Airflow Fans (may need mounting kit)

Fan extensions and splitters as required

Maxoncloud Riser Kit for 2nd through 5th GPUs
ADT 25cm R43MR adapter for the 1st GPU

ADT 50cm R43ML adapter for the 6th GPU

6 x GPUs

[Shlouski]

@Gorgon

 

I see you were using 2 psu's, normally this means that the one powering the motherboard is also connected by the 12v rail to all the card through the bifurcated Asus card and up through the risers, if a second psu is connected to one of those card then you have two psu's on the same 12v circuit, which from what I understand is a bad idea. 

 

How did you solve the issue of load balancing a multi psu system?

[Gorgon]

1 minute ago, Shlouski said:

@Gorgon

 

I see you were using 2 psu's, normally this means that the one powering the motherboard is also connected by the 12v rail to all the card through the bifurcated Asus card and up through the risers, if a second psu is connected to one of those card then you have two psu's on the same 12v circuit, which from what I understand is a bad idea. 

 

How did you solve the issue of load balancing a multi psu system?

The risers I used are powered. So the risers are fed from the same PSU as the PCIe 6 & 8-pins for the GPUs.

 

1 PS Powers the Motherboard & the 3 lower power GPUs risers and PCIe 6 & 8-pin plugs

1 PS Powers the 3 higher power GPUs risers and PCIe 6 & 8-pin plugs.

 

For the risers, it's a good idea NOT to use SATA power connectors but Molex or 6-pin (if available)

[Shlouski]

33 minutes ago, Gorgon said:

The risers I used are powered

 

Even so they are likely still wired for power from the pcie slot. 

 

Those first few pins before the key are power pins, the Asus card will be using that to run and to power devices attached to it, of course it can't deliver 4x 75watts so this is why your riser are powered, but I bet your risers are still wired up to those vcc 12v pins. 

 

I'm also using powered risers, using a multimeter I have found continuity from the motherboard 12v pins all the way to the pins on the card. 

 

 

[Shlouski]

I tested my maxcloudon powered risers and these do not send power from the slot 12v pins, its also clear to see there are no traces connected to the 12v pins on the bifurcation card.

 

This is not true of the other powered risers I have, when one psu is only connected to the risers and gpu's and the other to the pc components, my multimeter shows there's a connection between the 12v rail of the psu's.

 

As your asus bifurcation card was designed to run m.2 devices such as ssd's and nvme drives so it's also most certainly using power from the pcie slot, and the ribbon cable going up the ADT riser states 30V 3A (90w) on the side of it, which suggests it may also be wired to use 12v power from the slot.

 

[Shlouski]

I'm using powered risers also, one psu is only connected to the riser and gpu, and the other only to the motherboard. 

 

I connected my multimeter to the 12v molex pins on each of the psu's:

 

 

 

 

There is a connection there, unfortunately powered risers don't mean they aren't also wired to the slot. When I do the same test with maxcloudon risers the multimeter does not detect anything, just the loading screen waiting for a connection. 

 

It might be a good idea to check and make sure this is not the case in your rig.