If this is something you want to do you're going to be devoting a ton of time as I am assuming you want to roll your own tech because you didn't request a specific engine.
Picking up a book on Linear Algebra would be a good start (and very needed for any sort of development in a 2D or 3D space) From there you will want to pick a graphics API whether that be DirectX 11 or OpenGL. I personally would start with DirectX 11 as its fairly easy to use and has some very well written books for learning the API. You will also need some sort of windowing API whether that be Win32, X11 (for linux), Qt, SDL, ect. After you got a decent graphics implementation anything further will make sense on its own.
For DX11 books I recommend Practical Rendering and Computation with Direct3D 11.
DirectX 11 is an API for utilizing the GPU primarily for 3D rendering where as Unity is a complete package that contains things like input, sound, physics, rendering. An application like Unity would use DirectX 11 or similar. Unity would be a much easier approach but it would be less involved and educational IMO.
Unity is one of the major VR developing engines, there is a free version that is somewhat limited but for playing around it should suit your purposes. If you like it the pro version is like $40 or something
