DEPENDING on how much traffic there is, you're looking somewhere between 10ms > 50ms average with the distance, but I don't think that is the case here.
Could be bad port forwarding, could be your wireless is conflicting with other local Wireless networks (use a tool to see how many local networks are using your channel. Example for Android phones: Wifi Analyser.) Could be peak hour and everyone is using the net.
Or it could be issues with your ISP and how they have set things up.