CPU Lag Spikes

[DGB]

Before I start off, I haven't been in the PC building scene ever since I built my system 4 years ago, so please pardon my rusty terms if any!

I have never had issues with my rig when gaming besides the usual having to turn down settings on some games due to my hardware obviously not being top notch, but recently (the past week or two) I have been getting immense lag spikes in my games, and when I go to check on task manager, my CPU performance is shown to be tanking for just a few seconds before going back to normal (pic below). I've kept task manager open on my other monitor, and every time my FPS dips, it coincides with my CPU usage/clock speed dipping. MSI Afterburner says my temps are at 40-50 C when gaming, which iirc is pretty normal. Could this possibly be a cooling issues? I make sure to dust my PC every month or so and have never had problems like this before. Thanks for the help!

EDIT: The games I experienced this in were: 

-F1 2019 (never had problems till recently)

-Destiny 2 (same as above)

-Modern Warfare (kind of expected but putting it just in case)

[DGB]

18 minutes ago, BlueScope819 said:

Try and use hardwaremonitor for CPU temps, msi afterburner is generally for GPU temps. It looks like it could be a problem with temps.

I haven't replaced the thermal compound since I put the cooler in. I don't know if it should be regularly replaced after a certain amount of time. Should I look into that?