Very strange bug which lowers fps while playing Minecraft v18.1.2 optifabric

[KokoKaraWatashiNoRrama]

I am playing optifabric 1.18.2 with rtx 3080(1440p) and r5 5600g and get around 120/140 fps on bsl, 80/90 on seus and 180/200 fps on complimentary.

The weird thing is that whenever i go to my base fps drops to 40/35 and cpu usage is 60% but gpu is 25-27%. But when i am not in my base gpu usage is 70-90%.
The most noticeable thing is when i look towards my base standing in front of it gpu usage instantly drops to 25 % and when i look away gpu usage increases to 70-80.

Only for the sole purpose of testing if this was my gtx 1080's fault i bought an rtx 3080 but the performance is almost same.

How can i solve this?