Unmanaged Switches can be pretty picky in how they deal with bandwidth if multiple clients use them at the same time.
Also some ISP really struggle with multiple simultaneous connections to different servers (mostly coax based ones from my experience).
Since your Access Point doesn't seem to be the problem it would be interesting if a wired machine has the same issues and if they are still present if you connect directly to your Router.
Different devices have different ways that they can deal with wireless signals. If your wireless router allows for MU-MIMO then it can communicate with more devices simultaneously rather than having to wait for one before starting the other. In the same vein, some client devices don't have multiple antennae to support more simultaneous data streams from wireless access points, so they will suffer when the AP gets busy with other clients.
I wouldn't completely rule this out as being a cause of your connection issues. If your upload is being saturated, so will your download at some point because data has to be requested/sent before it can be received/downloaded.
