Polargirl

An update is the best thing I can think of especially when 8.1GB is reserved for hardware. The OP should manually seek out update 1903 if it isn't automatically available. 1809 had lots of problems.

On Thursday, May 16, professional overclockers ran a kit of Micron Ballistix DDR4 3600MT/s modules as a blazing frequency of 5,726MHz. 
 

 
Quite the big liquid nitrogen cooling system they got there. Lol. 
 
https://www.techspot.com/news/80127-micron-ddr4-memory-breaks-world-speed-record-overclocked.html
 

This is from may...

On Thursday, May 16, professional overclockers ran a kit of Micron Ballistix DDR4 3600MT/s modules as a blazing frequency of 5,726MHz. 
 

 
Quite the big liquid nitrogen cooling system they got there. Lol. 
 
https://www.techspot.com/news/80127-micron-ddr4-memory-breaks-world-speed-record-overclocked.html
 

thank you thats really helpfull

Yes, a 1060 6GB GPU will balance well with that CPU. The RX 570, fasauceome recommended will work fine but the RX580 will probably give your more bang for the buck.

An update is the best thing I can think of especially when 8.1GB is reserved for hardware. The OP should manually seek out update 1903 if it isn't automatically available. 1809 had lots of problems.

There are options between the two like RX 580 and 1060 6GB you might want to consider. 1070 is above a 1660 but sells used for what a 1660 (non-TI) sells new.

Try and update Windows. A couple updates ago this happened to me, Task manager was telling me a few ridiculous things and not the correct RAM speeds. An update a couple days later solved the issue

When you do list it, please post the eBay link here.

32GB vs 16 not going to make a difference when gaming. There will be a very small difference between 3000 and 3200.

i5-750 at 3.7 Ghz is not even close to being the same as 8100 that has 3600 Mhz all core clock and significantly higher IPC

3.7 is quite the overclock for a processor that has a base speed of 2.66 and a 3.2 turbo. If that speed cannot only be achieved but maintained for many non-stop hours of gaming, then you have a processor that is as good as or even slightly better than the I3-8100 and you can get a 1050 TI with that. Of course that is a really huge if, especially on sustaining that speed for hours rather than just tricked out for seconds to make a video. I'd make sure you know what your CPU can sustain on constant OC before buying a GPU.
 
You will absolutely need 8 GB of RAM just for so so gaming performance regardless of how well your CPU OCs and what GPU you buy as a consequence. Heck, 4GB is good for doing any basic task on Win 10 but not doing many of them at once like watching a 1080P streaming video with many other tabs open and a memory consuming program running in the background. In other words, 8GB is reccomended just to give yourself comfortable breathing room for even non-gaming tasks. 16GB is what most heavy gamers have but that is a bit much even for an actual I3-8100 paired with a 1050 TI.

Yes, your CPU will bottleneck with a 770 by quite a bit. Think 750 or 750 TI the most for that Nvidia generation as a guide.

And that is the new price. I see an open box B360 on eBay for $58.88 and just 2 bids on a used one for $41.00. I've seen multiple H310 motherboards ending in 2 days with zero bids starting at $14.99.

Unless you are planning on holding on to that computer for somewhere near 4 years, I would not buy a warranty. I did see a RX560 (RGB not 2) on Amazon for $106, the rest start at $120 and up and I haven't checked compatibility with your system. EBay has a "Buy It Now" refurb for $75 and some auction ones that might go as low as $50 for used and possibly refurbished.
 
With the system you have, I wouldn't overpay for new components. I'd try to find a RX 560 GB on eBay that I like because...why not? It will be well under your $100 initial budget even if it is a bit over your CPU.
 
I can check compatibility links and offer specific eBay auction or buy it now reccomendations if you'd like or I can link you to the $106 card on Amazon. The future proof people in this thread might have a point on just getting a RX 580 if you choose the Amazon new route since some of them aren't a whole lot more money.
 
Addendum: Since you don't know if your daughter will really be into gaming long term and hard enough to justify more expensive purchases, the eBay route is the perfect audible. Even though that CPU isn't suited for AAA games at 1080P, 60 FPS on ultra, you get to come closer now by doing as much as you can with the CPU she has without spending much. I guess you can alwaysbuy her a newer complete computer system at a later date like Christmas or after if it turns out she really is into this stuff and needs better hardware. If it is a passing phase, then you saved money. Win win.

I would say its about time for a new system since the 960 is starting to show its age and if you wanted to upgrade your cpu then you'd need a new motherboard, ram, and possibly psu. Unless your absolutely set on upgrading instead of starting fresh, id just save up and sell your previous system and gpu.

If you're not wanting to overclock, there is no point to any K series CPU.  The K means unlocked, thus overclockable by multiplier adjustment.  Non K chips can only be overclocked by BCLK adjustment, and very low end boards won't allow for that either.

But to do that, he'd have to spend somewhat over $100 as noted in the OP. Unlikely to be under $130 including tax and shipping to buy a used RX580 or 1060 and probably a bit more.
 
This is obviously an old computer that the OP almost certainly isn't interested in spending money on future proofing. Besides, the RX 580 and 1060 aren't particularly great future proof investments meant more for enjoying 1080P 60 FPS gaming for the next couple of years than as a longer term investment. Getting an old generation card that matches the RX550 in performance but goes for $50-$75 is likely the OP's preference or maybe a 560 closer to $100 but likely still under at most.

But to do that, he'd have to spend somewhat over $100 as noted in the OP. Unlikely to be under $130 including tax and shipping to buy a used RX580 or 1060 and probably a bit more.
 
This is obviously an old computer that the OP almost certainly isn't interested in spending money on future proofing. Besides, the RX 580 and 1060 aren't particularly great future proof investments meant more for enjoying 1080P 60 FPS gaming for the next couple of years than as a longer term investment. Getting an old generation card that matches the RX550 in performance but goes for $50-$75 is likely the OP's preference or maybe a 560 closer to $100 but likely still under at most.

The problem is that anything more powerful than a 1050 is going to be bottlenecked heavily, I think you'd be wasting your money if you upgraded this PC. It may sound strange but I would probably start thinking about building a new PC, even if it's just an R3 2200G system you've got more options for GPU upgrades. 

I really don't see the need for a dedicated wifi card for home use for most motherboards. The Gigabyte X570 board linked below comes with the new WiFi 6 802.11ax standard built in for future proofing. I wouldn't be surprised if most X570 boards do. You can check that out.
https://www.overclockers.co.uk/gigabyte-x570-i-aorus-pro-wifi-amd-am4-ddr4-x570-chipset-itx-motherboard-mb-57y-gi.html

The only downside is that you won't be able to enter BIOS on startup. Once you boot into the OS, you can tell your system to enter BIOS on restart.
 
The benefit is only 1-2 seconds compared to having it off, so it's not a big deal either way.

If you have another drive to boot from and it works, then you can eliminate the new 3900X CPU physically itself as the source of the problem since it is almost certainly compatible with that nvme but by all means check that out. Obviously the motherboard is compatible with the nvme or it wouldn't have worked with the old CPU.
 
You can also try putting the old R5 2600 CPU back in to see if your 960 Pro is still working. If it is, then that completely eliminates the nvme itself as the source of the problem as well as the motherboard physically itself and you can focus on BIOS settings.
 
Although that advice is far from solving your problem, at least it potentially narrows down substantially a long list of possibilities. Sorry I can't do better than that for now.

The CPU could be damaged.  Core or IMC.
I suggest trying a different CPU in that motherboard, even if you have to take the motherboard to some sort of hardware or repair shop and pay a fee to have them test a CPU in that board to see if it boots and allows windows to be installed.
The last thing you want to do is to RMA the board, pay all those charges, only to find that the CPU is kaput somehow.  That would really ruin your day.
 
Surely you have someone--anyone--some friends or some shop somewhere, where you can test another CPU (6700K or 7700K or even 6600K or 7600K) to see if it works?  Better to just pony up the $20 or so to have someone test the board for you and cut your losses.  That place could also test the CPU as well too.  Or a friend.