After fumbling around with ESXI 6.5 trying to pass through my old GTX950, I finally got it to work. If you're having a similar issue, hopefully this can help you.
First off, a disclaimer... This isn't a typical "fix" I normally would post and I don't know if all of this is necessary or not so make sure you check on your specific GPU to see if there is anything you need to do to enable passthrough properly instead...the following is more of a quick and dirty bypass to ESXI disabling the checkbox in the hardware menu and NVIDIA's hardware checking disabling the card if it detects it running in a VM... However there may be a good reason your GPU is prohibited from being passed through other than NVIDIA wanting you to purchase their enterprise cards instead...so...yeah...don't mess around with a computer that has all your data and stuff on it until you know this won't adversely affect function or reliability.
First issue with PCI passthrough you may run into. If your NVIDIA GPU is grayed out and you are unable to toggle passthrough...click this spoiler...
First issue you may encounter is the GPU is grayed out under Host, Manage, Hardware. There is a cheat you can do to allow you to enable the checkbox using the page "inspect" feature in chrome or other browsers.
This enables the checkbox. Now you can check both the GPU and the audio device checkboxes and click the "Toggle passthrough" button at the top. Now restart your host. You should now see both devices you checked say "Enabled/Active" with the GPU now grayed out and checkbox disabled again.
Now either you have it enabled or you are now able to have it enabled. You should be able to assign the device to your VM by adding "PCI device" and select the GPU.
Now if you are running Windows with NVIDIA drivers and receive Code 43...click this spoiler...
To fix code 43 in windows you will need to try to add the following parameter to the VM in ESXI...
hypervisor.cpuid.v0 = "FALSE"
Shut down your VM and put it in the parameter windows under "Actions", "Edit Settings", "VM options", "Advanced", "Configuration Parameters", "Edit Configuration..."
Then click "Add Parameter" and type it in so it looks like this....
After you add the parameter and click "OK" then "Save" you should be able to boot your VM and your GPU should work as normal.
