That should be on all the time though - you want to encourage air to exit the case, especially since you have warmer (radiated) air entering from the majority of your intake fans.
If your rear fan noise at low RPM is making you turn it off, then it's time to replace it.
Keep in mind that even if your CPU/GPU temps don't change by all that much, your temps across your motherboard may do so.
Check all your temps sensors in each fan orientation to get an idea of how effective your intake air is throughout your case.