Try a bufferbloat test if you haven't already.
Additionally, you may need to connect to your router/modem's settings panel and adjust or disable QoS (Quality of Service).
UDP packets are technically designed to be dropped, they trade reliability for speed, so less smart devices may be seeing those and deciding that it's okay for those to be deprioritized.