How to enable BAR 64 on AM3+ Mainboard for Nvidia K80 support- Bodgy K80 ML Workstation with AM3+

[Langhalsdino]

Hey everyone,

I am currently running an AMD FX8350 on a ASRock 990FX extrem 3 Mainboard and now facing a few issues while getting my K80 to work as a secondary GPU for handling ML workloads.
Sadly i can not get BAR 64 (above 4G encoding) enabled on my mainboard, therefore i am currently looking for a cheap replacement.

Are there any other search phrases besides "above 4G decoding" or "BAR 64bit" that i can search for in the mainboards datasheets? Or does anyone know how a AM3+ based Mainboard with two PCIE 16x slots, K80 support and BAR 64 option?

My current setup is really bodgy 2 PSUs, one for each GPU, and a custom 3D printed fan setup for the K80.
Was really fun to set it up, even though it feels kind of dirty

Best regards
Langhalsdino

[SpaceGhostC2C]

15 minutes ago, Langhalsdino said:

Sadly i can not get BAR 64 (above 4G encoding) enabled on my mainboard, therefore i am currently looking for a cheap replacement.

You don't have a setting for that, or you have a setting that didn't work?

 

15 minutes ago, Langhalsdino said:

 

 Or does anyone know how a AM3+ based Mainboard with two PCIE 16x slots, K80 support and BAR 64 option?

2 PCIe 16x slots limits you to the 990FX chipset boards. I don't know about the other requirement, though. I did use an M2070 on an Asus 990FX Sabertooth R2 without issues, and IIRC it had all 6GB ECC detected correctly. But I don't know if there's a way to test for your particular issue being present in my setup or not?

 

[Langhalsdino]

29 minutes ago, SpaceGhostC2C said:

You don't have a setting for that, or you have a setting that didn't work?

 

I don't have the settings for it and BIOS updates/downgrades did not help.

30 minutes ago, SpaceGhostC2C said:

2 PCIe 16x slots limits you to the 990FX chipset boards. I don't know about the other requirement, though. I did use an M2070 on an Asus 990FX Sabertooth R2 without issues, and IIRC it had all 6GB ECC detected correctly. But I don't know if there's a way to test for your particular issue being present in my setup or not?

So you mainboards bios supports BAR 64bit and lets you enable it in the bios?

If so this would help me a lot, since could just buy you mainboard and hope that everythings will work out. Of couse, this is no guarantee that it will work for me?

What kind of GPU Setup did you use? @SpaceGhostC2C Did you use two GPUs, ...?

[SpaceGhostC2C]

44 minutes ago, Langhalsdino said:

 

So you mainboards bios supports BAR 64bit and lets you enable it in the bios?

I never heard of this setting until seeing your post I just did a Google search about what it was, and found it had to do with 64-bit addressing of memory by PCIe devices. I didn't touch any setting in the motherboard specifically to use the Tesla card as far as I remember.

 

Is there a way to test if the card is working as if this setting is enabled vs. disabled?

 

Quote

What kind of GPU Setup did you use? @SpaceGhostC2C Did you use two GPUs, ...?

I had a GPU for actual video output, and the Tesla for computations. In other words, I had two cards, but I wasn't using them in tandem in any application.

[Langhalsdino]

On 11/26/2019 at 5:26 PM, SpaceGhostC2C said:

Is there a way to test if the card is working as if this setting is enabled vs. disabled?

Is there a bios setting that is called enable/disable "above 4G decoding" or something similar in the bios?

 

On 11/26/2019 at 5:26 PM, SpaceGhostC2C said:

I had a GPU for actual video output, and the Tesla for computations. In other words, I had two cards, but I wasn't using them in tandem in any application.

Do you know their RAM capacity? Mine is 8GB for the video output card and 24GB for the Tesla K80. Is you video output card bellow 4GB? If not i am very confident, that this might work for me, too.

It would be cool to use them in tandem, but honestly i am quite confident, that Tensorflow can handle it. The main issue is, that my current main board is not recognizing the K80 correctly, since it can not adress it without the 64bit pcie adresses (aka. "above 4G decoding"). 

Thanks a lot for you support @SpaceGhostC2C 

[SpaceGhostC2C]

5 hours ago, Langhalsdino said:

Is there a bios setting that is called enable/disable "above 4G decoding" or something similar in the bios?

Not that I remember, certainly nothing I've manually tweaked. I can try to check when I get home (maybe there's information in the manual, you can check it at Asus' webpage).

 

5 hours ago, Langhalsdino said:

Do you know their RAM capacity? Mine is 8GB for the video output card and 24GB for the Tesla K80. Is you video output card bellow 4GB? If not i am very confident, that this might work for me, too.

I used different videocards, in the 1GB - 3GB range, but the Tesla was 6GB. 8GB video cards have been around for a while, though, so I don't think the amount of RAM itself could be an issue (I'm sure there was people running R9 290X or 390X with 8GB in AM3+, or GRX 1060 with 6GB), but maybe something about its management. Do you have IOMMU enabled or disabled?

I've also noticed that it's a dual GPU card (so some form of SLI or NVLINK inside), and ships with some power adapter to increase the power draw over its PSU connector.

My point is, is there "smoking gun" evidence of the 4G decoding thing, or could it still be another form of incompatibility?

I just want to make sure you don't end up with another motherboard that also doesn't work  

 

[Langhalsdino]

4 hours ago, SpaceGhostC2C said:

My point is, is there "smoking gun" evidence of the 4G decoding thing, or could it still be another form of incompatibility?

I feel pretty confident about it, since it shows up in the PCIE device tree but is not able to be recogniezed as a gpu. Furthermore, i tested different GPU configuration. Quadro + Tesla, GTX + Tesla, ... and de PSU supplies enough power corresponding rail. But my confidence might be the result of a lack of other clues.  

 

4 hours ago, SpaceGhostC2C said:

Do you have IOMMU enabled or disabled?

 

I thought that i played around with enableing and disableing it during my last test run, but i will do it again.

4 hours ago, SpaceGhostC2C said:

I've also noticed that it's a dual GPU card (so some form of SLI or NVLINK inside), and ships with some power adapter to increase the power draw over its PSU connector.

Yes, its a dual GPU card and this was one of the reasons why i like the k80 so much. I used it a lot in rented server hardware and wanted to play around with one in my workshop  The connected PSU and the GPU connected rail can supply engouht power to the GPU. Furthermore I will lend me a bigger PSU from a friend next week, so i can completely cross this from my list of possible reasons.

 

4 hours ago, SpaceGhostC2C said:

I just want to make sure you don't end up with another motherboard that also doesn't work  

Thank you, this is my priority, too  

[SpaceGhostC2C]

OK, @Langhalsdino, I'm checking the BIOS now. I don't find any BAR / "above 4G" setting, and the only place where I could think of for PCIe settings is the North Bridge configuration. There I have 3 options

- IOMMU (enable/disable)

- Memory configuration

- Initiate graphic adapter (PEG/PCI or PCI/PEG)

 

Apparently, I had IOMMU disabled (or maybe it got reset at some point?). I looked at memory setting, and the options there are (with my current options):

- Bank Interleaving (Auto)

- Channel Interleaving (Auto)

- Warm Boot RAM (Enabled)

- Memory Clear (Disabled)

- ECC Mode (Enabled)  -> Currently using non-ECC RAM, seems innocuous. Teslas have ECC memory, but they handle that by themselves I think.

- Power Down Enable (Disabled)

- Memory Hole Remapping (Enabled)

- DCT Unganged Mode (Enabled)

All of which seem to deal with the system's RAM, not the devices. So it's just IOMMU in the end.

 

At this point, I could see some motherboard feature needed, although it's hard to think of a 4GB limit when you are using another GPU with 8GB, and higher than 4GB GPUs are widespread (I read something about these cards using some "sideload PCIe" to communicate temps and request chassis fan adjustments, although it did sound like an Nvidia rep saying "this is serverish stuff, go through our business channels, forget eBay" more than anything I'm sure all these fanless Teslas have something like that - yet they work in consumer boards). I used a 6GB Tesla in this board, but there are several differences between an M2070 and a K80 to jump to the conclusion that the K80 would work too. For all I know, the M2070 would work in your board too.

 

Random troubleshooting steps:

Have you tried different PCIe slots? Is there a chance that it landed in a "physical x16, electrical x8" slot? 990FX tend to have multiple physical x16 slots, only two of which are true x16

Which driver are you installing? (If I understood correctly, your PC is booting - some of the results on "4G decoding" lead me to people whose computers would not turn on with the K80 plugged in - maybe some ray of hope?).

[Langhalsdino]

@SpaceGhostC2C Thanks a lot for your help and detailed descriptions.

I tried a lot of different configurations, BIOS versions NVIDIA drivers, ...
Sadly my mainboard does not support some of the described Features like ECC, even though played with a lot of relevant modifications. 

Any how, I just bought a new Mainboard and can probably describe my results by next week. Since this might be interesting for others too.

If it doesn't work, i will probably buy some old server of ebay that officially supports the K80 and start from there