You are correct that the rx580 would indeed be better, I actually play games and a little bit of editing with an rx 580 and have never had driver issues, the drivers on the gtx are better for overclocking, but if not you should be fine. The rx 580 can render about 1.7x more at a moment then the gtx, though the gtx is faster not that much faster. For a cooler just look for a cooler with a max tdp of at least 175w if not more. The ram that you currently use can not be used with any other ram as you can not mix ram. Look for 2 sticks of 8gb ddr3 ram 1600mz then later if you need to you can get 2 more stick of the same kind and have 32gb ram, but try 16gb first as it may be enough.
To clarify on ram. Your pc supports ddr3 ram no other kind can be put in it.