(Working) Tesla K40m in a 32Bit BAR Motherboard.

[Poet129]

I believe I have figured out what is going on finally... so with the K40c BIOS at start, the boot sector of the VBIOS is loaded into VRAM at start. Then flashing the K40m VBIOS correctly Identifies itself. So following this logic all that needs done is to get the sections telling the VBIOS what it is from the OG K40m VBIOS and pairing that with the boot code of the K40c VBIOS. I've yet to get a working configuration due to a corrupt boot sector supposedly. I've still managed to not brick the card though... yet.

[Guest]

5 minutes ago, Poet129 said:

I believe I have figured out what is going on finally... so with the K40c BIOS at start, the boot sector of the VBIOS is loaded into VRAM at start. Then flashing the K40m VBIOS correctly Identifies itself. So following this logic all that needs done is to get the sections telling the VBIOS what it is from the OG K40m VBIOS and pairing that with the boot code of the K40c VBIOS. I've yet to get a working configuration due to a corrupt boot sector supposedly. I've still managed to not brick the card though... yet.

Makes sense actually. But the corrupt sector doesn't sound right. Cause the card still works. Which is good....

 

You're gonna flash this thing to death! XD

[Poet129]

1 minute ago, ShrimpBrime said:

Makes sense actually. But the corrupt sector doesn't sound right. Cause the card still works. Which is good....

 

You're gonna flash this thing to death! XD

How many flashes is the maximum? I actually don't have a clue...

[Guest]

Just now, Poet129 said:

How many flashes is the maximum? I actually don't have a clue...

Never heard such a thing. I've done the same, but to an old motherboard. The bios was actually bricked so hard, it partially took the board with it. Couldn't figure it out. Bought 3 different bios chips from bios depot on ebay. Only the original chip with the partially bricked bios ever worked in that board. M3A32 MVP Deluxe wifi app solo. Let the gods rest its soul. That damn board clocked well.

 

Any ways, no actual number has ever been given. I probably flashed that particular board Dozens of times. It remained bricked until it stopped posting. Along with ln2 runs.... there is some limit. Somewhere.... lol

[Poet129]

4 minutes ago, ShrimpBrime said:

Never heard such a thing. I've done the same, but to an old motherboard. The bios was actually bricked so hard, it partially took the board with it. Couldn't figure it out. Bought 3 different bios chips from bios depot on ebay. Only the original chip with the partially bricked bios ever worked in that board. M3A32 MVP Deluxe wifi app solo. Let the gods rest its soul. That damn board clocked well.

 

Any ways, no actual number has ever been given. I probably flashed that particular board Dozens of times. It remained bricked until it stopped posting. Along with ln2 runs.... there is some limit. Somewhere.... lol

So most likely not a set limit just the hardware wearing out... I've probably flashed this card at least a hundred times or so.

[Guest]

Just now, Poet129 said:

So most likely not a set limit just the hardware wearing out... I've probably flashed this card at least a hundred times or so.

Right. 

The brick ingredient came from NB and memory OC.

I spent several weeks flashing it's brains out.

The LN2 killed the board ultimately, but died on my geothermal loop.

 

I don't see why you couldn't flash it another 100 times actually. Eventually you'll get a bad flash and have to recover I'm thinking. Bricking before that... possible maybe but less likely.

[Poet129]

6 minutes ago, ShrimpBrime said:

Right. 

The brick ingredient came from NB and memory OC.

I spent several weeks flashing it's brains out.

The LN2 killed the board ultimately, but died on my geothermal loop.

 

I don't see why you couldn't flash it another 100 times actually. Eventually you'll get a bad flash and have to recover I'm thinking. Bricking before that... possible maybe but less likely.

Not that we really didn't already know this, but this shows that the limitation has been set in the VBIOS. So NVIDIA choose to do this and choose once again to not release any kind of fix for it... I understand why, but I'd obviously really like them to just fix it. I know it won't happen though. After looking at the views on this page I've started to wonder if anyone from NVIDIA saw this...

[Guest]

1 minute ago, Poet129 said:

Not that we really didn't already know this, but this shows that the limitation has been set in the VBIOS. So NVIDIA choose to do this and choose once again to not release any kind of fix for it... I understand why, but I'd obviously really like them to just fix it. I know it won't happen though. After looking at the views on this page I started to wonder if anyone from NVIDIA saw this...

Have you been over to techpowerup? Look up the exiledairman. He might have some pointers. The guys at Guru3D are pretty smart too. If there's a way, it'll be at one of those 2 places. 

 

Nvidia? Maybe have seen the thread. Maybe one of the engineers might drop you a PM with a little bios file for you. Never know. Hardware works in mysterious ways!

[Poet129]

1 hour ago, ShrimpBrime said:

Have you been over to techpowerup? Look up the exiledairman. He might have some pointers. The guys at Guru3D are pretty smart too. If there's a way, it'll be at one of those 2 places. 

 

Nvidia? Maybe have seen the thread. Maybe one of the engineers might drop you a PM with a little bios file for you. Never know. Hardware works in mysterious ways!

Where can I get one of those programmers? Luck would have it that my pc crashed while I was flashing the VBIOS.

I think I found one...

https://www.overclock.net/threads/brand-new-method-how-to-unbrick-flash-almost-any-card-amd-or-nvidia.1612108/

 

https://www.amazon.com/gp/product/B013Q5P3ES/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

https://www.amazon.com/gp/product/B015W4PKR6/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

[Guest]

21 minutes ago, Poet129 said:

Where can I get one of those programmers? Luck would have it that my pc crashed while I was flashing the VBIOS.

I think I found one...

https://www.overclock.net/threads/brand-new-method-how-to-unbrick-flash-almost-any-card-amd-or-nvidia.1612108/

 

https://www.amazon.com/gp/product/B013Q5P3ES/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

https://www.amazon.com/gp/product/B015W4PKR6/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

He grabbed a clip on with whip. That would be handy.

https://www.amazon.com/gp/product/B015W4PKR6/ref=oh_aui_detailpage_o00_s00?ie=UTF8

[Poet129]

@ShrimpBrime

On 9/11/2021 at 10:04 PM, Poet129 said:

Luck would have it that my pc crashed while I was flashing the VBIOS.

I managed to fix the card without a v-bios flasher, instead with some fancy pcie hot-swapping.

 

I've verified this 'trick' with two motherboards now these being, EVGA Classified SR-2 and Super Micro X8DA3. It has managed to persistent across some shutdowns, no restarts yet. I believe this is due to windows drivers being reinitialized after starting up from a restart. Whereas this is disabled by default on shutdown, however I haven't tested this thoroughly. This is explained much better here.

 

Not really important but I've managed a bios level overclock with the KeplerBiosTweaker application. I've applied this to just the v-bios flashed first at startup. This mod includes upping the wattage and voltage limits to 300W and 1.4V accordingly. Along with a basic +258 and +500 on the core and memory. Along with disabling the boost clock. I've kept the separate non modded v-bios as well just to be safe in case something goes wrong.

 

The code:

cd "%UserProfile%\Documents\nvflash_5.227.0.1-win"
devcon.exe disable *DEV_1023*
nvflash64.exe -A -s -i 0 "Tesla K40m.rom" -6
devcon.exe enable *DEV_1023*
Pause
nvidia-smi -g 0 -dm 0
devcon.exe disable *DEV_1023*
nvflash64.exe -A -s -i 0 "Tesla K40m MOD.rom" -6
devcon.exe enable *DEV_1023*

This is the code that currently needs run at every restart. Devcon.exe is a Microsoft tool that comes with one of their toolkits (Not sure which one you should be able to find it with google). It is used merely to disable and reenable the device (GPU) before and after flashing, however this could be replaced with pauses and the use of device manager. I've found that it works better as well if all other GPUs in the system are disabled to start and reenabled at the currently present pause command. The nvidia-smi call sets the GPU to WDDM mode instead of TCC, which allows for other protocols than CUDA to be used, such as OpenGL, DirectX, etc. The v-bios that is being flashed second in the code MUST be flashed before starting, on either a normally functional board or within Linux, Windows will not let you flash the card without this. Thus why it is flashed back after breaking past Nvidia's driver limitation, if it is not and the computer is restarted then the card will go back to the normal state and will need to be reflashed with either Linux or a normally functional board. The version of nvflash is the same for both Windows and Linux (separate executables), and can be found within the code (5.227.0.1) other versions may work however this is the one I've tested with.

[Poet129]

@ShrimpBrime

Well, I've now figured out why the nvidia driver won't start on the modded v-bios... apparently they are signed however the signatures are stored in the v-bios itself not checked against a known good list. So running the modded K40c v-bios through KeplerBiosTweaker, and changing nothing just using it to resign it makes it all work... one year later wow so simple. So all that needs done is to get a variant of the card's v-bios that will work in your motherboard then mod it to allow it to be flashed to your card via nvflash (I only had to change one byte), then resign it with KeplerBiosTweaker and finally flash it using nvflash on Linux or a normally functional board. No need to reflash every reboot now.

[Guest]

29 minutes ago, Poet129 said:

@ShrimpBrime

Well, I've now figured out why the nvidia driver won't start on the modded v-bios... apparently they are signed however the signatures are stored in the v-bios itself not checked against a known good list. So running the modded K40c v-bios through KeplerBiosTweaker, and changing nothing just using it to resign it makes it all work... one year later wow so simple. So all that needs done is to get a variant of the card's v-bios that will work in your motherboard then mod it to allow it to be flashed to your card via nvflash (I only had to change one byte), then resign it with KeplerBiosTweaker and finally flash it using nvflash on Linux or a normally functional board. No need to reflash every reboot now.

Really? that's all it came down to? 

 

OK, in the future (not sure when) I'll try to get the K20 running this way. Thanks for all the hard work and effort man!

[ABIT-Wolf]

On 2/13/2022 at 11:55 AM, Poet129 said:

@ShrimpBrime

Well, I've now figured out why the nvidia driver won't start on the modded v-bios... apparently they are signed however the signatures are stored in the v-bios itself not checked against a known good list. So running the modded K40c v-bios through KeplerBiosTweaker, and changing nothing just using it to resign it makes it all work... one year later wow so simple. So all that needs done is to get a variant of the card's v-bios that will work in your motherboard then mod it to allow it to be flashed to your card via nvflash (I only had to change one byte), then resign it with KeplerBiosTweaker and finally flash it using nvflash on Linux or a normally functional board. No need to reflash every reboot now.

Hi my first post here, Great work you have done here and some good reading.I been researching the Tesla K40 & K80 for months trying to find some help to make these cards work. I'm getting code 12 in windows 10 Pro..

Will you be sharing the modded vbios? If so please PM me with it? I'm sure your dos flashing as if like my cards it will never load in windows enough to do a flash there. Please package nvflash and readme.txt with commands in the vbios package/zip. Thanks in advance.

 

This is my hold back ATM: 

C:\NVFlash-all_checks_bypassed-v5-206-0-1>nvflash --list

NVIDIA Firmware Update Utility (Version 5.206)
Modified Version by Joe Dirt

NVIDIA display adapters present in system:
NOTE: Unconfigured display adapter found, device not accessible:
      Tesla K40m           (10DE,1023,10DE,097E) H:--:NRM S:00,B:0A,PCI,D:00,F:00

<0> Quadro 600           (10DE,0DF8,10DE,0835) H:--:NRM S:00,B:07,PCI,D:00,F:00
 

[ABIT-Wolf]

After reading and rereading looks like I may have to setup Linux to get the job done.

I have a couple flash drives with live Linux GUI and also with dos picks. v18.xx

Just wondering what version you where using as it may work better for this?

Next up is what all else I may need to get the job done. A walk threw would be very nice.

My Linux is a bit rusty, I've dabbed in it some back in the day with Folding at Home

when they had Big Advanced work units using server boards with 32 & 64 CPU cores.

Three of those where EVGA SR2 all dead now.

I still have a 64 core server motherboard but the PSU went up in smoke.

I don't have any PSU's that have a two eight pin CPU plugs to try and wake the beast.

Any help here getting these 2x Tesla K40m flashed would be very much apprehended. Thanks in advance.

 

Edit: my return window for the 2x Tesla K40m is getting short, hope to hear something soon. 12 days or so left. 

[Poet129]

@ABIT-WolfSorry for the late reply I have had a lot going on with school lately and haven't checked the LTT Forum in a while. I'll get right on getting that V-BIOS for you. I still have my card and it is still working great. Just a note though I couldn't find any crypto mining algorithms that worked on it so if that's your only interest sell the cards. I only managed to get NVFlash to detect the card on 5.227.0.1 on Linux. Windows detected it but couldn't flash it. GUI Ubuntu works but you must uninstall all Nvidia and Nouveau Drivers first. I'll post back in a bit with the V-BIOS I used.

[ABIT-Wolf]

1 hour ago, Poet129 said:

@ABIT-WolfSorry for the late reply I have had a lot going on with school lately and haven't checked the LTT Forum in a while. I'll get right on getting that V-BIOS for you. I still have my card and it is still working great. Just a note though I couldn't find any crypto mining algorithms that worked on it so if that's your only interest sell the cards. I only managed to get NVFlash to detect the card on 5.227.0.1 on Linux. Windows detected it but couldn't flash it. GUI Ubuntu works but you must uninstall all Nvidia and Nouveau Drivers first. I'll post back in a bit with the V-BIOS I used.

Great to hear from you, thank you for taking the time to try and help me.

I'm not looking to mine with these cards, so that isn't a problem.

I know its a lot to ask but could you throw together a simple walk threw & zip with the vbios and the nvflash version?

I've searched the web for 5.227.0.1 but couldn't find it anywhere, all links at TechPowerUp have been updated with newer versions.

Going out for my Birthday, so I'll be out a few hours. This will be the best gift I could get, getting these cards going.

BBL

[Poet129]

@ABIT-WolfI obtained both these NVFlash programs from techpowerup you have to look under show older versions and scroll they are out of order. For Linux use the x86 folder under the Linux zip. I've included my modified and resigned V-BIOS. It is exactly the same as the Tesla K40c V-BIOS but changed to be able to be written to a K40m then resigned totaling 3 bytes of changes. The K40c V-BIOS was obtained from techpowerup as well.

 

After getting all Nvidia and Nouveau Drivers uninstalled under Linux you should run:

nvflash -i x "Tesla K40mc Signed.rom" -6

You should replace x with the index of your gpu. You can find this using the following:

nvflash -a

Feel free to ask any questions you have. I'm here to help.

nvflash_5_227.0.1-linux.zip nvflash_5.227.0.1-win.zip Tesla K40mc Signed.rom

[ABIT-Wolf]

3 hours ago, Poet129 said:

@ABIT-WolfI obtained both these NVFlash programs from techpowerup you have to look under show older versions and scroll they are out of order. For Linux use the x86 folder under the Linux zip. I've included my modified and resigned V-BIOS. It is exactly the same as the Tesla K40c V-BIOS but changed to be able to be written to a K40m then resigned totaling 3 bytes of changes. The K40c V-BIOS was obtained from techpowerup as well.

 

After getting all Nvidia and Nouveau Drivers uninstalled under Linux you should run:

nvflash -i x "Tesla K40mc Signed.rom" -6

You should replace x with the index of your gpu. You can find this using the following:

nvflash -a

Feel free to ask any questions you have. I'm here to help.

nvflash_5_227.0.1-linux.zip 2.03 MB · 1 download nvflash_5.227.0.1-win.zip 1.75 MB · 1 download Tesla K40mc Signed.rom 236.5 kB · 1 download

Thank you for your quick response.The night out was good as was the food. 

I'm back now and ready to give this a shot. What version of Ubuntu or Linux would you suggest?

I will start there and work my way into getting this done tonight on both cards.

Edit BTW I saw here where someone is working with a Tesla K20. I'll try to up load my vbios

that worked out of the box, top of card says K20X when drivers are installed they see it as a K20Xm.

K20XmStock.rom 

nvidia-smi -fdm 0

nvidia-smi --gom=0

nvidia-smi --compute-mode=1 not used and test very little

nvidia-smi -ac 2600,784 sets OC 784 is the max core I've been able to get.

nvidia-smi -q -d SUPPORTED_CLOCKS shows supported clocks

I use a older version of MSI AB to OC the memory. 1550 is as high as I've tried so far.

 

[ABIT-Wolf]

14 hours ago, Poet129 said:

@ABIT-WolfI obtained both these NVFlash programs from techpowerup you have to look under show older versions and scroll they are out of order. For Linux use the x86 folder under the Linux zip. I've included my modified and resigned V-BIOS. It is exactly the same as the Tesla K40c V-BIOS but changed to be able to be written to a K40m then resigned totaling 3 bytes of changes. The K40c V-BIOS was obtained from techpowerup as well.

 

After getting all Nvidia and Nouveau Drivers uninstalled under Linux you should run:

nvflash -i x "Tesla K40mc Signed.rom" -6

You should replace x with the index of your gpu. You can find this using the following:

nvflash -a

Feel free to ask any questions you have. I'm here to help.

nvflash_5_227.0.1-linux.zip 2.03 MB · 1 download nvflash_5.227.0.1-win.zip 1.75 MB · 1 download Tesla K40mc Signed.rom 236.5 kB · 1 download

Took me A while but I got one flashed. Used the latest version of 

ubuntu-22.04.1-desktop-amd64 and some of the commands from here

https://hiveon.com/forum/t/guide-nvidia-gpu-vbios-flashing-with-hiveos/4091

Now to do some testing and get the other flashed. Thank you so much for all the time you put into this to make it work! cheers!

[Poet129]

12 minutes ago, ABIT-Wolf said:

Took me A while but I got one flashed. Used the latest version of 

ubuntu-22.04.1-desktop-amd64 and some of the commands from here

https://hiveon.com/forum/t/guide-nvidia-gpu-vbios-flashing-with-hiveos/4091

Now to do some testing and get the other flashed. Thank you so much for all the time you put into this to make it work! cheers!

Looks like it worked good, you may want to consider using nvidia-smi to switch it from tcc mode to wddm mode depending on your use case. Any Linux version should work forgot to mention... @LinusTech I'd love to help make a video on this if you are interested. @ABIT-Wolf Glad to hear it works on more than just my machine.

[ABIT-Wolf]

Well I don't have hardwire in my shop were the computer is, and most of what I tried wouldn't see my WI-FI.

The new version had the few packages I needed to make it work where as some of the others didn't.

Once those where installed everything fell together. except the _linux in the tot I used had to be removed.

Already got the modes switched.  Thank you!

[ABIT-Wolf]

Mahaha X2, now maybe I play with the SLi mod.    Again many thanks for your hard work!

 

[ABIT-Wolf]

Just a small update in case others may want to know. K20 and K40 both work together in this BTC T37 MB.

I didn't have enough fans to run all three or I didn't think I had ATT K20 was already fixed with a fan.

Got to thinking and remembered I had a old u1 server it has six 40mm fans I can use, so I'm all set.

Have another K20X arriving today.

 

 

[ABIT-Wolf]

This is a bit off topic but was wondering what Nvidia Geforce card is equal to the Tesla K40m, Titan or something like the GTX 1070?

Answers open to anyone that knows.  I did a search here but came up with nothing same with Google.

 I read that the Tesla K20 has been tricked into thinking its a Titan then using geforce drivers.

Thinking of maybe giving this a try with the k40m.