I found online resources that show that it is not impossible to use GPU memory as "System memory"
there are people that have RX480 or RX580 that have 8gb of VRAM, enough to run your entire OS into it
also, since they are faster, would this make it faster or slower?
https://wiki.archlinux.org/index.php/Swap_on_video_RAM
https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/25031-using-graphics-card-memory-as-swap
https://stackoverflow.com/questions/27460141/can-gpu-use-swap-space-when-its-ram-is-full
https://news.ycombinator.com/item?id=17722467
https://www.techpowerup.com/forums/threads/is-it-possible-to-share-graphic-card-memory-with-ram.162038/