I think you've narrowed it down to either an issue with your router or with your ISP. I am suspecting the router in this case only because you seem to be getting poor ping times between your machine and the router. These should be 1 ms because it's only a single hop between your machine and the router. Try one of these commands:
ping -n 100 <ROUTER_IP> - Sends 100 pings
OR
ping -t <ROUTER_IP> - Continuously pings until you stop it
Look to see if there are spikes in ping times over time. If they stay low then try running them again and then start introducing other congestion from another device, maybe play a YouTube video on your phone or something. If either of these affect ping to your router then it may be time for an upgrade because it's having trouble handling that small amount of traffic that it is getting.
If the router checks out then I would engage your ISP and report that you've done a bunch of testing and found that the latency is in the circuit. They will push back and ask you to reboot, etc. Just do what you need to do to get a ticket opened and work from there. I work in IT and I have to do this all the time, even with business class internet.