Pardon me for reviving an old thread; a friend just asked about this issue and I quickly googled good old LTT hoping that I could just pass a quick link to an answer and that someone already elaborate something that should be a very common behaviour.
So I would like to write down something that I learned from my professor many moons ago, lecturing CPU and ram in university. It was regarding L2 and L3 cache but it can be applied to RAM as well.
So, cache acts as a waiter, in a restaurant. So let's say one day you come in a restaurant, and order a soup and a steak. You wait for certain amount of time for it to get prepared. You get your steak, leave. Next day, you again order the same. You wait, get your steak, you leave. Third day, the waiter see you at the door, he already figured you will have a soup, and a steak. You sit at the table. You get your food without waiting. You leave happy.
The next day, you fancy a fish. But the waiter got your steak ready. You sit and order a fish. Now waiter says to a chef to cook it for you. Now you have to wait again.
That is basically the purpose of the RAM (and cache) memory. To intelligently predict what are you going to use next, and have it ready for you ASAP. So if you are just using windows desktop, browsing the web etc. It will allocate as much memory to the tasks it predicted you are going to use next. And it will use the resources it has at its disposal. The moment that cache is required for another task (you order a fish) eg. you load Adobe Premiere, it will flush the amount of memory required by the program.
If you really are paranoid about the numbers, you can use some memory cleaner software but it doesn't really do any real benefit.
TLDR. Everything works as it should. You should not worry about the percentage of memory used by your idling system. It's supposed to work like that, to help your tasks run faster.