A couple of things that I noticed in your config file are that
proxy_cache_bypass $http_upgrade seems weird (but you don't seem to have proxy caching enabled in the first place so that shouldn't matter), and that your upstream seems to be on the same host as your nginx but you're using the public IP rather than localhost. If you change the proxy_pass IP to 127.0.0.1 instead, does that make any difference?
As a sanity check, what's the performance of skipping nginx and hitting the backend directly (either from your local machine or SSHed into that box), and of serving a static file directly from nginx (a 404 page works)? If there's a problem with the proxying specifically, the proxy request would be quite a bit longer than the sum of those two.