AMD generally has better multi-gpu scaling then NVIDIA. As others have stated most games won't need 4gb even in 4k.
How much is vram being used isn't always how much it needs - there can be unused assets sitting in vram that it can cull if more space is required. You only see a performance impact once the scene being rendered requires more data then can be stored in the frame buffer and the gpu has to swap assets in real time to complete frames.