In general, the boost behavior of CPUs isn't intelligent enough to know whether you're doing basic tasks or if you're doing something more intensive. The CPU recognizes that there is a load, checks if it has available power and thermal headroom, and then boosts to the max. So if you overclock your CPU, then when it boosts up, it will boost to that higher level.
However, when it comes to power draw and temperatures, while you'll notice a small increase when doing basic things, it is really going to be negligible. You won't notice a difference unless you're constantly monitoring. If your cooler can handle an all-core workload when oveclocked (something like rendering a video) then it will easily handle browsing the Internet. And if you are okay with the power increase seen when doing something intensive, then the minor power increase when working on office tasks shouldn't be a concern.