Minecraft server: Low TPS.

[Johannes_Lazor]

Im running a minecraft server, and it was at 20tps at all times with 15-20 people online.
It has been running for about 8 months now, and i started having low TPS problems about a month ago, The TPS is at 8-15 even when nobody is online, and i cant find the issue. Might be the shear number of Plugins used, but i doubt it since we were running even more before a fatal failure forcing me to redo everything exept parts of the worlds.
And sometimes, it is up at 20 again for no apperent reason.

Im using Spigot, which is supposed to be one of the better optimized plugin API's.

Here is the timings from Spigot - http://www.spigotmc.org/go/timings?url=14433603

it looks pretty bad, but it isnt directly the plugins which cause it, it is rather the listeners for them.

SPECS:

Dell Poweredge 2900

CPU: Dual Xeon 5130(bad CPU to be honest, as minecraft servers are limited to one core if im not mistaken)
RAM: DDR2 4x4gb + 4x1gb, all ECC. Totally 20gb(16 of which are dedicated for minecraft, even though it rarely uses more than 2-4gb)
OS:Linux (im only using the command side, im using Putty and Filezilla to connect to it)

Plugin List:

FireworkBuilder,
WorldEdit,
dtlCore,
Buycraft,
MassiveCore,
NBTEditor,
ChatColor,
PermissionsEx,
Reloader,
TerrainControl,
AnvilColors,
Gringotts,
ImmortalNpc,
WorldGuard,
ItemKills,
UnbreakingAnvils,
aAntiSpam,
VoteRewards,
Minetrends,
CapsLock,
OpenInv,
ClearLag,
Votifier,
Essentials,
EssentialsSpawn,
Vault,
MyCommand,
RainbowArmour,
PlotSquared,
EssentialsChat,
AutoAnnouncer,
Factions,
Citizens,
ServerSigns,
Multiverse-Core,
ScoreboardStats,
dtlTraders,
MineResetLite,
PhatLoots,
SurvivalGames,
Sentry,
HolographicDisplays,
Signs,
AncientGates,
Multiverse-Inventories,
Multiverse-NetherPortals

 

The IP is Mc.Kubeen.Net if you want to join, the main attraction is Mazerunner and RPG.

[TacticalCuda]

network speeds?

[TheKDub]

network speeds?

 

Network speeds won't cause a low TPS.

 

 

SPECS:

Dell Poweredge 2900

CPU: Dual Xeon 5130(bad CPU to be honest, as minecraft servers are limited to one core if im not mistaken)

RAM: DDR2 4x4gb + 4x1gb, all ECC. Totally 20gb(16 of which are dedicated for minecraft, even though it rarely uses more than 2-4gb)

OS:Linux (im only using the command side, im using Putty and Filezilla to connect to it)

 

The IP is Mc.Kubeen.Net if you want to join, the main attraction is Mazerunner and RPG.

 

Lets see...

1. Hosting server specs are pretty poor.

2. How long has it been since the last map reset? I find that after about 6 months or so maps tend to get buggy/have corrupt chunks, which can cause issues with the server TPS.

 

The plugins shouldn't be an issue, if the lower TPS started right when you added a new one, then it is probably that plugin, otherwise I'd say it's something else.

 

I've had my server running for around 1.75 years now, and had ran some before then, so I'm decently familiar with issues they can encounter.

(My server runs Tekkit Classic 3.1.2, Minecraft 1.2.5)

[MatthewEnderle]

I've ran a server with average 30 people on at a time on a simular setup. I'm going to say it is your drives. Get a SSD. Even a used 120GB will work. Under the Perc 5 raid controller create a raid 0 on the ssd. you will be super stable then.

[Johannes_Lazor]

Network speeds won't cause a low TPS.

 

 

 

Lets see...

1. Hosting server specs are pretty poor.

2. How long has it been since the last map reset? I find that after about 6 months or so maps tend to get buggy/have corrupt chunks, which can cause issues with the server TPS.

 

The plugins shouldn't be an issue, if the lower TPS started right when you added a new one, then it is probably that plugin, otherwise I'd say it's something else.

 

I've had my server running for around 1.75 years now, and had ran some before then, so I'm decently familiar with issues they can encounter.

(My server runs Tekkit Classic 3.1.2, Minecraft 1.2.5)

last map reset was about 4 months ago or so, Most players who join dont walk more than about 100 blocks away from spawn areas as the maze is fairly hard to beat.

shouldnt be that people are walking too far away and causing lag

[Johannes_Lazor]

I've ran a server with average 30 people on at a time on a simular setup. I'm going to say it is your drives. Get a SSD. Even a used 120GB will work. Under the Perc 5 raid controller create a raid 0 on the ssd. you will be super stable then.

why would the drives cause low TPS? it should only affect the load-times, which isnt the issue. The server runs perfectly good when it comes to loading chunks and players moving around, But everything else is slow, mobs walk at half the speed, NPC's become derpy and moves very slowly, that kind of thing

[Johannes_Lazor]

I'm surprised you didn't have TPS problems with that CPU in the beginning... it's a large cache tho

As @TheKDub pointed I'd look into the maps

yea, Im suprised it even held up.

thinking of building a new PC in the future, one or 2 months from now, so i might move over the server to this one(760k and 8gb of RAM)

[conspiravision]

What flavor of Linux are you running?  I don't know anything about minecraft hosting but if I had to guess it would be network connectivity or CPU.  I don't think minecraft is very hardware intensive but I'm not sure.  When did you last reboot the server?

[Johannes_Lazor]

What flavor of Linux are you running?  I don't know anything about minecraft hosting but if I had to guess it would be network connectivity or CPU.  I don't think minecraft is very hardware intensive but I'm not sure.  When did you last reboot the server?

im using Ubuntu 14.04.2.

i reboot the server every other day or so, it fixes the issue for a minute, then it is back again.

[TheKDub]

yea, Im suprised it even held up.

thinking of building a new PC in the future, one or 2 months from now, so i might move over the server to this one(760k and 8gb of RAM)

 

High single-threaded performance and a fair amount of RAM is the main things it will need, a fast internet connection and SSD are great to have too.

[conspiravision]

what is "free -m" showing?  I doubt TPS would be affected by your memory but its weird that it works fine when you first reboot then gets slow?  Also run "cat /proc/sys/vm/swappiness" after doing the "free -m" command.  I 

 

And I'm guessing you also know how to use the top command to show useful information while the server is running?

[stconquest]

1000 chickens in a single block is my guess 

[Johannes_Lazor]

what is "free -m" showing?  I doubt TPS would be affected by your memory but its weird that it works fine when you first reboot then gets slow?  Also run "cat /proc/sys/vm/swappiness" after doing the "free -m" command.  I 

 

And I'm guessing you also know how to use the top command to show useful information while the server is running?

Not really, the main command that i use for checking the RAM usage and such is /gc, it shows entities, tiles and chunks loaded per-world

[Johannes_Lazor]

1000 chickens in a single block is my guess 

doubt it, limited the number of mobs able to spawn in a 30block radius to 20.

[conspiravision]

On the linux console run the commands I typed earlier.

 

top

free -m

cat /proc/sys/vm/swappiness

uptime

 

Then copy the results

[stconquest]

doubt it, limited the number of mobs able to spawn in a 30block radius to 20.

 

...but someone might have wanted lots of eggs.

 

I am half serious about the chickens though.  How well do you know the people playing?  Last server I played had tools to see what chunks are hogging resources.  The mods/admins would tp to the location and investigate.

[Johannes_Lazor]

On the linux console run the commands I typed earlier.

 

top

free -m

cat /proc/sys/vm/swappiness

uptime

 

Then copy the results

top - 01:38:43 up  5:19,  1 user,  load average: 0,22, 0,21, 0,29

Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie

%Cpu(s):  4,8 us,  0,0 sy,  0,0 ni, 95,2 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st

KiB Mem:  20562008 total, 11270016 used,  9291992 free,    46876 buffers

KiB Swap: 20969468 total,        0 used, 20969468 free.   335320 cached Mem

 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 1730 isac      20   0 19,048g 0,010t  19260 S  20,6 51,4 192:59.08 java

 6073 isac      20   0   27364   3100   2632 R   0,3  0,0   0:00.03 top

    1 root      20   0   33524   4028   2712 S   0,0  0,0   0:01.79 init

    2 root      20   0       0      0      0 S   0,0  0,0   0:00.00 kthreadd

    3 root      20   0       0      0      0 S   0,0  0,0   0:00.09 ksoftirqd/0

    5 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/0:0H

    7 root      20   0       0      0      0 S   0,0  0,0   0:02.85 rcu_sched

    8 root      20   0       0      0      0 S   0,0  0,0   0:00.74 rcuos/0

    9 root      20   0       0      0      0 S   0,0  0,0   0:00.76 rcuos/1

   10 root      20   0       0      0      0 S   0,0  0,0   0:00.55 rcuos/2

   11 root      20   0       0      0      0 S   0,0  0,0   0:00.48 rcuos/3

   12 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuos/4

   13 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuos/5

   14 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuos/6

   15 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuos/7

   16 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcu_bh

   17 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/0

   18 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/1

   19 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/2

   20 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/3

   21 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/4

________________________________________________________________________

free -m

             total       used       free     shared    buffers     cached

Mem:         20080      11263       8816          5         45        327

-/+ buffers/cache:      10890       9189

Swap:        20477          0      20477

________________________________________________________________________

/proc/sys/vm/swappiness

60

________________________________________________________________________

uptime

 01:40:57 up  5:21,  1 user,  load average: 0,32, 0,28, 0,31

 

[Johannes_Lazor]

...but someone might have wanted lots of eggs.

 

I am half serious about the chickens though.  How well do you know the people playing?  Last server I played had tools to see what chunks are hogging resources.  The mods/admins would tp to the location and investigate.

they wont spawn that way either, not even with spawneggs, and same with things such as boats and minecarts.

Most of the players are just random people, about 60 people/day

[stconquest]

they wont spawn that way either, not even with spawneggs, and same with things such as boats and minecarts

 

Ok... step away from the chickens for a sec...

 

If the server ran fine, now does not, something changed.  Hardware failure aside, find what is cause the server tick rate to tank.  The answer is out there somewhere, get you Holmes pipe and find the issue.

[conspiravision]

So what I am seeing is that you have plenty of memory free so you can rule out that, nothing is swapping so you can rule out that.  CPU doesn't seem to be having a large load on it, you're only using 31% of it at most.  You could check to see if its the networking but I'm kind of iffy on that if you say it was working fine before.  I bet its a bad plugin or some of the minecraft files got corrupt and performance  has degraded because of it?  I'm not sure if there is a way to repair that or not.

[Johannes_Lazor]

So what I am seeing is that you have plenty of memory free so you can rule out that, nothing is swapping so you can rule out that.  CPU doesn't seem to be having a large load on it, you're only using 31% of it at most.  You could check to see if its the networking but I'm kind of iffy on that if you say it was working fine before.  I bet its a bad plugin or some of the minecraft files got corrupt and performance  has degraded because of it?  I'm not sure if there is a way to repair that or not.

The reason to why it is at 31% load is becouse minecraft only uses a single core, 25% of the total performance

[conspiravision]

Right, so CPU isn't a problem.  Between when the problem didn't exist to when it started to happen something changed.  Possibly an update that you did to the server that changed a previous config file (linux or the minecraft server itself), or a plugin module becoming unsupported or degrading the application.

[MatthewEnderle]

Before you just throw out what I say, Try using this start file. I had the same issues you had, and this helped a bit. I had a Java programer of 12 years who is in his late 30s find some targets that assisted the server.

java -server -Xms1024M -Xmx49152M -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:MaxGCPauseMillis=250 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -Xnoclassgc -XX:UseSSE=3 -XX:+UseLargePages -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+AggressiveOpts -jar craftbukkit.jar noguiPAUSE

[Johannes_Lazor]

 

Before you just throw out what I say, Try using this start file. I had the same issues you had, and this helped a bit. I had a Java programer of 12 years who is in his late 30s find some targets that assisted the server.

java -server -Xms1024M -Xmx49152M -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:MaxGCPauseMillis=250 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -Xnoclassgc -XX:UseSSE=3 -XX:+UseLargePages -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+AggressiveOpts -jar craftbukkit.jar noguiPAUSE

changed the startup arguments, about 5 minutes later i get the exact same problems, right now im the only one online and it is sitting at 9 TPS

[Johannes_Lazor]

Dude, do you realize that sets the maximum RAM to 48GiB, right? He doesn’t have that much RAM and shouldn’t need that much either.

Anyway, @Johannes_Lazor, it’s obviously a plugin issue. Try disabling your plugins one-by-one until you see a TPS change. I’ve run a Minecraft server on an octo-core Avoton Atom C2750 just fine, and it does have a lower single-threaded PassMark score than your ancient Xeon.

dont worry, im smart enough to edit the arguments myself.

i have narrowed down something which probably is one of the causes, Sentry, for citizens. it is fairly heavy as it has to create an AI for each npc. i will disable sentry for NPC's which dont actively engage in combat.

i cant disable it entirely as i need it to make the RPG work.

not much of an improvement, disabled about a third of the sentrys, and disabled paths for them as they wont need a spot to walk back to anymore, i got some improvements, but i can rule out citizens/sentry as the only cause, had it running with a similar amount of NPC's in the past.