Your phone buzzes at 3:04 a.m. and the Discord webhook says the server is down with 40 players online. What you do in the next fifteen minutes decides whether players lose a night or lose a world. This runbook exists to help you spend those minutes on the right commands instead of on panic and a blind restart.
It is written for owners and admins who can read a log but were never taught an order of operations. The timings come from our own crash lab: Paper 26.2, 16 vCPU, a 512 MB heap, an empty test world, and a systemd unit using Restart=on-failure with RestartSec=2. We ended the JVM three different ways and timed what systemd did about it. An empty world boots faster than your live one, so treat every number here as a floor, not a guarantee.
Why Most Crashes Are Not Crashes
A crash is a process that exits. systemd only restarts a process that exits. That single sentence explains almost every “it did not come back” page you will ever get, and it is the reason a triage runbook beats a monitoring dashboard at 3 a.m.
In the lab, a hard kill (kill -9) behaved exactly as advertised. systemd restarted the process in 2.07 seconds, and the server logged “Done” again 7.12 seconds after the kill. That is the happy path, and it is why people trust restart policies.
Out-of-memory was not the happy path. A purpose-built leak plugin drove the JVM to OutOfMemoryError 8.74 seconds after startup, and the server never auto-restarted within 120 seconds. The JVM stayed alive, thrashing in garbage collection, so systemd saw a running process rather than an exit. A frozen JVM was worse still: after a SIGSTOP, nothing restarted it within 95 seconds, and no watchdog thread-dump lines appeared, because a frozen JVM cannot run its own watchdog. External health checks are the only thing that catches that state.
The lesson: uptime is not health. Before you touch anything, work out which of those states you are actually in.
The First 15 Minutes, Minute by Minute
0-2 minutes: declare, stop the loop, preserve evidence
Declare the incident out loud in staff chat so two people are not debugging the same box. Then stop the restart loop with systemctl stop minecraft if the unit is cycling. A looping service rewrites latest.log on every boot and will erase the evidence you are about to collect.
systemctl status minecraft --no-pager
journalctl -u minecraft -n 200 --no-pager
tail -n 100 logs/latest.log
ps -o pid,rss,etime,cmd -C javaCopy, do not move, the whole logs folder and crash-reports to a dated directory outside the server root. If a Java process is still alive, take a heap dump before you kill it: jcmd <pid> GC.heap_info for a fast summary, then jcmd <pid> GC.heap_dump /var/backups/heap.hprof. The dump is large, so confirm free space first with df -h.
Classify the Failure
Exit codes tell you who ended the process. 137 means 128 + 9, SIGKILL: either you or the kernel OOM killer ended it without warning. 143 means 128 + 15, SIGTERM: a polite stop, usually systemctl stop or a shutdown command. Both count as failures to systemd unless you configured otherwise, which is why a clean planned restart can look identical to a crash in the journal.
| Failure class | Symptom | Auto-recovered? | First action |
|---|---|---|---|
| Clean shutdown | Exit 143, “Stopping server” in log | No, by design | Confirm it was planned, then start |
| Hard exit | Exit 137 or 1, no shutdown lines | Yes, ~2 s in our test | Read journal, check for OOM kill |
| OOM inside JVM | OutOfMemoryError, process still alive | No | Heap dump, then restart by hand |
| Frozen JVM | Port open, no ticks, no log lines | No | Thread dump, then hard restart |
| World corruption | Fails during chunk load, repeats | Loops forever | Stop unit, restore last good world |
| Plugin crash loop | Same stack trace every boot | Loops forever | Stop unit, disable the plugin jar |
| Disk full | Odd IO errors, missing Done line | No | df -h and df -i |
That is why a crash loop needs systemctl stop before you edit anything. While the unit keeps restarting, your edits race the boot sequence, half-written configs get read, and the log you need is truncated. Stop the service, fix the file, then start once and read the result. If you are unsure how much history you are losing, our guide to reading Minecraft server logs covers where Paper writes each message.
Stabilise and Recover
5-10 minutes: remove the cause, not the symptom
With the unit stopped, disable the suspect plugin by renaming the jar (mv plugins/Suspect.jar plugins/Suspect.jar.disabled) so a later restart cannot load it. Then confirm the JVM was not killed by the kernel rather than Java itself:
dmesg -T | grep -i oom
df -h / /var
df -i /
ss -lntp | grep 25565Raising heap is the most misapplied fix on the internet. Set -Xms equal to -Xmx, leave room for the OS, the page cache, and any other JVM on the box, and remember that a heap larger than physical RAM converts a Java error into a kernel kill. A 512 MB heap was enough for an empty world and is nowhere near enough for 40 players with plugins, which is exactly why the lab numbers are floors.
If the world itself is the problem, restore the last known-good world and config set from your automated backups and accept the rollback. A partially loaded region file will otherwise crash you again on the next boot, ten seconds after players rejoin.
10-15 minutes: verify, then communicate
Join with a test account. Check TPS and MSPT with /tps and /mspt, watch chunk loads as you fly, confirm permissions and economy commands still resolve, and only then reopen the whitelist or the proxy. Tell players what happened in one sentence, when it was fixed, and whether anything rolled back. Silence is what turns a 12-minute outage into a trust problem.
Prevent the Next 3 A.M. Page
Automation covers exits, not health. Configure systemd.service with Restart=on-failure, a short RestartSec, and StartLimitBurst so a broken jar cannot hammer the CPU forever. Then add an external check that queries the port and TPS from outside the JVM, because in our SIGSTOP test the frozen process produced no watchdog output at all. For tuning advice, read the PaperMC documentation rather than copying flags from a forum post.
Alert on MSPT and player count, not just on process state. Pair that with scheduled restarts at a quiet hour, and write a postmortem every time.
- Timeline: first alert, actions, recovery, in UTC.
- Impact: minutes down, players affected, rollback depth.
- Root cause: the specific plugin, file, or allocation.
- Detection gap: why monitoring missed it or paged late.
- Corrective actions: each with an owner and a date.
Automated vs. manual recovery
- Automated pros: fast, consistent, works while you sleep; 2.07 s beats any human response.
- Automated cons: restarts on exit only, hides root causes, and can loop a corrupt world.
- Manual pros: preserves evidence, catches OOM and frozen states, fixes the cause.
- Manual cons: slow, depends on who is awake, easy to skip steps under pressure.
Use both. Let systemd handle exits, and let a human handle anything a restart fails to fix twice.
Common mistakes and how to avoid them
- Restarting first. You overwrite the log that names the cause. Copy evidence at minute zero.
- Editing configs during a crash loop. Stop the unit; racing boots produce half-read files.
- Raising heap blindly. Bigger than RAM means the kernel kills you, not Java.
- Trusting uptime. A frozen JVM reports “active (running)” while doing nothing.
- Never testing restores. A backup you have not restored is a guess.
- Ignoring inodes. df -i catches a full disk that df -h hides.
Expert tip: keep a one-page card taped to your desk with the five commands above. Under pressure, nobody remembers flags, and a card beats a search engine.
Frequently Asked Questions
The process was ended by SIGKILL, which is 128 + 9. Either someone ran kill -9 or the Linux OOM killer stepped in. Check dmesg -T | grep -i oom and the unit’s journal before restarting.
Because the JVM never exited. It stayed alive and thrashed while systemd kept seeing a running process. In our lab test it did not recover within 120 seconds. You must restart it by hand, and take a heap dump first if you want to find the leak.
Run systemctl stop minecraft so the unit cannot restart, then read journalctl -u minecraft -n 200 and logs/latest.log. Rename the suspect plugin jar or roll back the world, then start the service once and watch the full boot.
On our test box systemd relaunched the JVM 2.07 seconds after a hard kill, and the server logged Done again 7.12 seconds later. A production world with plugins and pregenerated chunks takes far longer, so plan for 30 to 90 seconds and alert on anything beyond that.
Conclusion
Fifteen minutes of ordered work beats three hours of guessing. Stop the loop, keep the evidence, classify the failure, remove the cause, then verify before you reopen. If a crash ever looks like a freeze, remember that no JVM can diagnose itself once it has stopped scheduling, which is why your monitoring has to live outside it. For official flag reference, keep the jcmd documentation and the kernel OOM tunables bookmarked. Then rehearse the runbook on a test box this week, while nobody is watching.
Leave a Reply