LTT Folding Team's Emergency Response to Covid-19

[Zberg]

Honest question(for the mercenary kings) how much does it end up costing per day to run those high point totals.  Is it reasonable or expensive?

[Xsilent(X)]

3 minutes ago, Favebook said:

What do you mean? I am not talking about software modifications, you are bottlenecking them by using PCIe risers (as far as I can see). Also, it is not true that AMD cards have low PPD on beta WUs, I had up to 20% or more when I used it for short period.

 

Please, humor me, I would love to see a screenshot with all of those cards having 2.4M PPD.

 

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16X lanes...

[marknd59]

What are the + / -'s of setting beta ?

 

[Zberg]

1 minute ago, marknd59 said:

What are the + / -'s of setting beta ?

 

Im curious of this too.  I had beta running for the competition, and honestly I got WU after WU for a while (most of the time while running it), and I had none fail.

 

I dont know if they normally fail?  But mine all went through, no issues, and kept feeding me WU once I started running it

[leadeater]

2 minutes ago, Xsilent(X) said:

 

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16 lanes...

I think the concern is more what the actual bandwidth the cards are getting, not all risers and riser setups are all that good, for things like FAH. If they are only getting x4 2.0 then yea you're probably losing a bit of performance but x4 3.0 probably not.

[Kilrah]

6 minutes ago, Xsilent(X) said:

 

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16X lanes...

Talking of bandwidth, not power supply. On my 2080S bus utilization is 37% of a PCIe3 x16 link. So x8 would be OK, but less and it would be choking. Your cards are less powerful so they might be OKish at x4.

So what riser configuration did you use?

[Ayren]

8 minutes ago, Xsilent(X) said:

 

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16X lanes...

there are motherboards like that (even ones with 10) but they utilize PLX chips n so on so overall throughput is limited

[Xsilent(X)]

2 minutes ago, leadeater said:

I think the concern is more what the actual bandwidth the cards are getting, not all risers and riser setups are all that good, for things like FAH. If they are only getting x4 2.0 then yea you're probably losing a bit of performance but x4 3.0 probably not.

True but i don't have Thousands of dollars or Euro's to buy the equipment needed to set it up properly... That rig i have bought for like 300 Euro's in total and was second handed...

[Kilrah]

6 minutes ago, Ayren said:

there are motherboards like that (even ones with 10) but they utilize PLX chips n so on so overall throughput is limited

If you want to make a decent multi GPU folding rig you'd go intel HEDT (my old 5960X has 40 lanes, on mobo split as one x16 slot, 2 x8 slots, one x4 slot and 4 for the NVMe) or Threadripper.

 

  

6 minutes ago, Xsilent(X) said:

True but i don't have Thousands of dollars or Euro's to buy the equipment needed to set it up properly... That rig i have bought for like 300 Euro's in total and was second handed...

Sure, but check how they all work together - maybe you can get as much with 2 as with all of them, with a lot lower power consumption...

[Gorgon]

15 minutes ago, Xsilent(X) said:

 

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16X lanes...

No but you can get close with some of the Asus WS boards but they cheat using PLX Switches

[Xsilent(X)]

1 minute ago, Kilrah said:

Sure, but check how they all work together - maybe you can get as much with 2 as with all of them, with a lot lower power consumption...

 

I just fired up the rig as off today, i am going to let it fold for at least 4 days to see the average... And about power consumption each card draws no more then 180Watt per card...  as they are all Radeon RX 470's cards with only one on the pci-e 16 lane to be a Radeon rx 590 card for video output.

[leadeater]

2 minutes ago, Gorgon said:

No but you can get close with some of the Asus WS boards but they cheat using PLX Switches

[Gorgon]

9 minutes ago, Kilrah said:

If you want to make a decent multi GPU folding rig you'd go intel HEDT (my old 5960X has 40 lanes, on mobo split as one x16 slot, 2 x8 slots, one x4 slot and 4 for the NVMe) or Threadripper.

Only if you wanted to overspend. x570 + 3900x/3950x + PCIe bifurcation + m.2 to PCIe x16 adapters is far less expensive

but I've been getting a lot of really low PPD WUs last few days

[GOTSpectrum]

1 minute ago, Gorgon said:

Only if you wanted to overspend. x570 + 3900x/3950x + PCIe bifurcation + m.2 to PCIe x16 adapters is far less expensive

But, if all you care for is PCIe lanes then you can grab a quad core x79 xeon with 40 lanes for 20GBP 

[leadeater]

1 minute ago, Gorgon said:

Only if you wanted to overspend. x570 + 3900x/3950x + PCIe bifurcation + m.2 to PCIe x16 adapters is far less expensive

If X399 motherboards weren't so expensive I would think used 1900X might be price competitive.

[GOTSpectrum]

Hey guys, quick update, the ceremony is coming, just not feeling all to well today! 

 

Out of all the days to feel unwell, it is today!

 

Spoiler

It's not COVID or anything respiratory, nothing series like that!

 

[lewdicrous]

4 minutes ago, GOTSpectrum said:

Hey guys, quick update, the ceremony is coming, just not feeling all to well today! 

 

Out of all the days to feel unwell, it is today!

 

  Reveal hidden contents

It's not COVID or anything respiratory, nothing series like that!

 

Get well soon dude, all the best. 

[Xsilent(X)]

8 minutes ago, Gorgon said:

Only if you wanted to overspend. x570 + 3900x/3950x + PCIe bifurcation + m.2 to PCIe x16 adapters is far less expensive

 

Dude how do you have the money to buy 8 rigs, and so much GPU's?

[Kilrah]

8 minutes ago, Gorgon said:

Only if you wanted to overspend. x570 + 3900x/3950x + PCIe bifurcation + m.2 to PCIe x16 adapters is far less expensive

 

6 minutes ago, GOTSpectrum said:

But, if all you care for is PCIe lanes then you can grab a quad core x79 xeon with 40 lanes for 20GBP 

Yeah, cheap used server equipment. They've got bunch of PCIes and cores even if they're relatively slow ones that's good for the job

[Dalmahr]

Kinda disappointed in myself for not getting higher place! Combination of work and running F@H on windows i think that caused me to not be able to run 24/7 to accumulate points. I kept failing to get WUs without restarting windows or restarting F@H totally. Hope i can get a better situation/rig going for the next official event.

[Tomatodude95]

32 minutes ago, Zberg said:

Honest question(for the mercenary kings) how much does it end up costing per day to run those high point totals.  Is it reasonable or expensive?

I have looked at it during the competition and tried it out a bit, so I'm no expert. Also I only tested Google Cloud so this might differ significantly from other providers.

If you are willing to put in the work to deal with preemptible instances in Google Cloud you can setup a Node with 8 vCores and 4 Nvidia T4s for about 0.5 USD an hour, which makes it about 12 USD a day. If you go non-preemptible it becomes quite a bit more expensive, it'll be something like 1.18 - 1.70 USD per hour depending on how long you run it (the longer you do the more discount you get, ie. the closer to the 1.18 USD you get). You might be able to go with lower configs for the CPU, if you aren't using the CPU. I tried the 4 vCore version as well, but if I folded on the CPU as well I had an issue where the system ran out of RAM. (You get about 1GB of RAM per core.) However, the difference between 4 and 8 vCores is less than 3 cents per hour (About 22 USD for 24/7 preemptible usage per month).

AFAIK on AWS it's a bit different because you can more or less decide at what price you want you instances to run and as long as prices are lower than you predefined price and AWS doesn't need the machines otherwise they will run. Anyone who is more informed about Spot instances on AWS can correct me if I'm wrong please!!

 

Auzure has similar offers and possibilities, but I'm not informed about those.

 

So it really depends on your definition of reasonable and expensive. Might make sense if you need some power in the short term or have some free credit to try it but it's still quite a chunk of money, I mean the cheapest option I presented costs 12 USD/day or 360 USD/month.

For me it was fun to play with and good to get some experience with but I wouldn't consider running it in the long term. Anyhow if you want to run hardware over a longer period of time it is usually cheaper to buy the hardware and run it at home/locally.  (Yes, I know not always, there are benefits to having stuff in the cloud.)

 

Additional Info about Performance:

So if you don't know what the performance is like of a Nvidia T4, it's about 800k PPD, so one of the machines with 8 cores and 4x T4s got about 3.2M PPD (I measured values from 2.8M - 3.8M PPD depending on the units and specific workload.)

 

Hope that answers your question, don't hesitate to ask if you want more details on something.

Edited April 11, 2020 by Tomatodude95
Added some info on Azure

[Ayren]

14 minutes ago, Gorgon said:

No but you can get close with some of the Asus WS boards but they cheat using PLX Switches

Ive been wondering: could one use a PCIE 4.0 PLX switch to split 4.0 into 2x3.0 lanes... since AFAIK the speed is negotiated between the PLX switch and the pcie card

[Xsilent(X)]

Guys I did the math if I am not getting beta projects I would be averaging around 2.2M PPD

[Favebook]

37 minutes ago, Xsilent(X) said:

PCI risers do not cause to bottleneck as long as you power the PCI lines properly. There is not a motherboard with 6 PCI-E 16X lanes...

I am sorry, but I will not even bother explaining you anything as you are rather stubborn.

33 minutes ago, leadeater said:

I think the concern is more what the actual bandwidth the cards are getting, not all risers and riser setups are all that good, for things like FAH. If they are only getting x4 2.0 then yea you're probably losing a bit of performance but x4 3.0 probably not.

I'll just say: THIS! 

He is 100% using x1 riser. Bottleneck is at least 40% per card.

[Xsilent(X)]

2 minutes ago, Xsilent(X) said:

Guys I did the math if I am not getting beta projects I would be averaging around 2.2M PPD

 

 

The MB is this brand https://www.asrock.com/MB/Intel/H110 Pro BTC+/#Support