time for quad titan memesand other things

but nice idea!

Quote

Part of my reason to want to go EYPC is just because I want an excuse to use an AMD server platform

Why not 

Quote

I can see with a project of this scale I'd be liable to run into issues such as attempting to connect 7+ GPUs and having the server even POST. I'd probably run into the same issue Linus did when he was building the 6 editor workstation where the motherboard just wasn't having it. What I would probably do is start with a "safe" 4 GPUs. Most systems seem fine with 4, going beyond that things get sketchy. Beyond this I would probably only buy one at a time and see how she handles.

Yours is going to be away easier and at least more supported. You're not also trying to pass through USB cards and not using PCIe expander backplanes. I'd ask for the PCIe eletrcial layout diagram from ASRock in advance, then you can check for problems like PCIe switch chips that might interfere with IOMMU groups.

@Gegger Actually no, I'd be going team Red here. NVIDIA really doesn't want people using their GPUs in VMs unless you buy their GPUs they specify for said usage (and pay licensing fees). Basically the card detects weather or not it's in a VM and if it does it goes NOPE and slaps you in the face with a code 43 error. There's software hacks to tell the GPU that it's not in a VM but from what I've heard they don't always work. AMD on the other hand is like "ppph, do whatever you want I don't care. Throw two or three in a VM."

@Leadeater Crap I forgot about the possibility of some going though a Chipset. Weird thing is on the product page I wasn't able to find a chipset. I'll have to read though the manual and see what it says. If it doesn't help I'll shoot ASRock an email.

another person fails at tagging me or spelling my handle wrong

ofc, novideo does weird things

@Gegger I don't @ people properly in status updates because they'll get two notifications. I'm just saving people the headache.

Mis-spelling your name on the other hand was a mistake. Where I'm from Gregger is just kind of a derogatory or slang term for addressing someone who's name is Greg or Gregory. You picked an easily misconstrued name.

/shrug

thx for saving me the spam though

@Gegger If you noticed ol'colonel_mortis is asking for suggestions to improve the Status Updates. If he ends up including an option to opt-out of Status Update notifications I'll have to start using the @ properly.

I hope he does. Sometimes all I want to do is comment once but then two people will start a longass unrelated conversation and I get notified of all their posts.

Like right now Leadeater is probably getting all of these...sorry man. Do mods have any adv power to opt-out?

*looks at den-fi*

*quad AMD Radeon Vega 64 build incoming*

@Gegger Den-Fi operates in creeper mode. Sneaking around the forum without letting people see.

@TopHatProductions115 As great as quad Vegas would be the cost per card isn't worth the performance. Really taking a page out of the crypto miners book here.

I can get cards with 2048 Streaming Processors for <$150

Vega 64's have 4096 but cost >$500

Financially it's really not worth it. 8 GPUs will have the compute performance of ~4 Vega 64's but at about 1/2 the cost. What I'll pay in electricity will take a long time to regret by which time the GPU's will either die or need to be upgraded anyways. Also it will allow me to crunch twice as many projects. I'm quite certain these cards don't support SR-IOV as we have discussed before. If they did I could split them all in 1/2 and run 8 projects on 4 cards.

@Windows7ge I meant it as a joke :3

@TopHatProductions115 I'm OP you don't have to @ me .

I mean, I'd love to. If I was rich you bet your ass I'd grab 8 Radeon VII's (although according to wendell over on the Level1Tech forums apparently the VII has some issues passing though to VMs).

What I will probably go with (if they're still for sale by the time I make up my mind to build this - they'll probably be long EOL) would be the RX570's. They have 2048 Streaming Processors & only require 1x8-pin. Ideal for the application.

You can run more than one WU per GPU.  BOINC manager even allows configuration per project for GPUs.  In the past when I had OG Titans, I had them set to run two Einstein WUs each.  I even seen folks configure newer Titans to run up to four WUs at once.

Offset, you do have longer run times per WU, but depending on the project, can be very minimal depending how well the GPU can handle multiple WUs.

On the issue of going beyond four GPUs, yes, that does get into funky territory.  Though, I seen folks run five+ setups for F@H and BOINC.  But, such setups do require a good bit of work to get everything smooth.

Still, I would not mind reading a write up on how to do pass through with QEMU virtualization.  I'm still very newbie level to VMs (most of my experience been with VMWare setups that are handle with scripts by our programmers - at most troubleshooting I done with those is re-covering the main controller when its file system shot itself - it is a SEUS distro setup and it crap itself on the beginning day of a military exercise).  And, those don't have any dGPUs in them.

BOINC may allow you to run more than 1 WU of the same project on one GPU but what I mean isn't the same.

Now while talking to TopHatProductions115 I learned about SR-IOV and how its use on AMD Firepro cards allows the creation of multiple vGPUs on one physical GPU. In this sense we could force BOINC to run completely different projects on the same GPU at the same time. Unfortunately support never became available on the consumer cards so that's not an option.

The VFIO guide for QEMU/KVM I followed on Debian Linux although tedious showed that disconnecting a GPU from the host and passing it though isn't terribly difficult. At its core all you have to do is tell the Linux Kernel to not assign it a driver at startup. From here it's isolated from the OS which it can then be picked up via it's IOMMU group and passed-though to a VM. You can plug a monitor into the GPU even or setup Looking Glass so you can use the VM GPU on the host machine at very near native performance.

I'll start with four then just +1 until I can't get it working or I run out of slots.

Giving you a QEMU write up on GPU pass-though is something I think I can do. Writing tutorials has kind of become my niche.

Ah, so the goal is to run multiple different projects per GPU.

Hmmm, I don't recall anyone doing such, so that definitely be interesting to see.

I have around a total six GPUs myself.  Though, trying to sell two to three off so I can refresh to some newer ones.  May smack two of them into the 2nd rig and look into following along.  I'm always down to learn something new.

Uuuhh still no, lol. Ok, yes the whole SR-IOV thing, vGPUs, VM's sharing a single GPU, yes very cool. That's not what I'm looking to do. The GPU's I'd be buying don't support SR-IOV.

I mean independent GPU's, each GPU will run a singular VM, each VM will run a singular project, one WU each.

The "hack" is by doing this I could have 4,5,6+ GPU's, each GPU running a different project all on the same box.

I'm pretty sure BOINC doesn't let you do this natively.

@Windows7ge Can we have a screenshot of something similar to what you're attempting?

I can't help but laugh at this point. It cannot be explained in simpler terms. And you TopHatProductions115, you're working on a server with multiple GPUs for GPU pass-though how are you of all the people here confused? *fists shaking in the air vigorously*

Let's try a physical diagram:

VM1 -> RX 570 passed-though -> BOINC: Einstein@home

VM2 -> RX 570 passed-though -> BOINC: LHC@home

VM3 -> RX 570 passed-though -> BOINC: Yo-yo@home

VM4 -> RX 570 passed-though -> BOINC: WCG

Etc.

How's that? Does that make more sense?

@Windows7ge That's better  

In the somewhat near future I'm going to deem it time to upgrade my GPUs. Probably going to be the 5700XT's. For the old 290X's I'll start by getting two VMs w/ a GPU each going on my old 2011-3 platform. If I can do that successfully then making 3,4,5+ work should be easy.

Ah, ok.  So each GPU doing their own different BOINC project.

@Ithanul Yep. Sorry if that's not as cool as you expected. The virtualization part is still pretty nifty and if anything using whole GPUs is better. More resources for each project.

Sometime down the line I'll be making a QEMU guide on GPU pass-though. It's not difficult. Just kind of tedious.

Still very interesting.  Would give more control on what project to run per GPU.  And, depending on GPU if it is strong enough, the GPU can be set to do two to four WUs without fear that if a weaker GPU is in the system wind up doing that many.