Headless Server Crash [0.15.28] After High Uptime

[Myth494]

We've experienced our headless server crash about 4 times now, each time we had to go back to our latest autosave. the server never gets shutdown (unless it crashes) we've got about 160 hours into it.

It feels like after an extended period of time the server will crash.
This is the last part of the logs:
12.940 Info ServerMultiplayerManager.cpp:671: mapTick(43740001) changing state from(InGameSavingMap) to(InGame)
13.024 Info ServerMultiplayerManager.cpp:755: mapTick(43740001) received stateChanged peerID(1) oldState(ConnectedWaitingForMap) newState(ConnectedDownloadingMap)
19.380 Info ServerMultiplayerManager.cpp:755: mapTick(43740001) received stateChanged peerID(1) oldState(ConnectedDownloadingMap) newState(ConnectedLoadingMap)
20.965 Info ServerMultiplayerManager.cpp:755: mapTick(43740095) received stateChanged peerID(1) oldState(ConnectedLoadingMap) newState(TryingToCatchUp)
21.598 Info ServerMultiplayerManager.cpp:755: mapTick(43740133) received stateChanged peerID(1) oldState(TryingToCatchUp) newState(WaitingForCommandToStartSendingTickClosures)
21.599 Info GameActionHandler.cpp:3323: MapTick (43740133) processed PlayerJoinGame peerID(1) playerIndex(0) mode(connect )
21.710 Info ServerMultiplayerManager.cpp:755: mapTick(43740140) received stateChanged peerID(1) oldState(WaitingForCommandToStartSendingTickClosures) newState(InGame)
68.170 Received SIGINT, shutting down
68.172 Info MainLoop.cpp:315: Saving map as /home/minecraft/factorio/saves/_autosave1.zip
68.377 Info MainLoop.cpp:326: Saving progress: 0.005851%
68.909 Info MainLoop.cpp:326: Saving progress: 100.000000%
69.206 Info ServerMultiplayerManager.cpp:116: Disconnecting multiplayer connection.
69.206 Info ServerMultiplayerManager.cpp:671: mapTick(43742928) changing state from(InGame) to(DisconnectingScheduled)
69.240 Info ServerMultiplayerManager.cpp:671: mapTick(43742928) changing state from(DisconnectingScheduled) to(Disconnecting)
71.181 Info ServerMultiplayerManager.cpp:671: mapTick(43742928) changing state from(Disconnecting) to(Disconnected)
71.181 Info ServerMultiplayerManager.cpp:140: Quitting multiplayer connection.
71.181 Info ServerMultiplayerManager.cpp:671: mapTick(43742928) changing state from(Disconnected) to(Closed)
71.537 Info HttpSharedState.cpp:44: Downloading https://multiplayer.factorio.com/remove-game/2635846
71.768 Info HttpSharedState.cpp:111: Status code: 200
71.768 Info UDPSocket.cpp:234: Socket closed
71.790 Info UDPSocket.cpp:234: Socket closed
71.801 Goodbye

[Loewchen]

This is not a crash, this is a shutdown by user.

[Myth494]

This is not a crash, this is a shutdown by user.

It's only me and one other that plays the server, we did not shutdown the server. We host it on a vps that we have full control over. If it was shutdown by the user shouldn't the server save? instead of the latest save being an autosave one.

[Rseding91]

Thanks for the report however the log file doesn't show any crash. It shows something sent a SIGINT to the game and it shut down gracefully.

[Oxyd]

This is not a crash, this is a shutdown by user.

It's only me and one other that plays the server, we did not shutdown the server. We host it on a vps that we have full control over. If it was shutdown by the user shouldn't the server save? instead of the latest save being an autosave one.

68.172 Info MainLoop.cpp:315: Saving map as /home/minecraft/factorio/saves/_autosave1.zip
68.377 Info MainLoop.cpp:326: Saving progress: 0.005851%
68.909 Info MainLoop.cpp:326: Saving progress: 100.000000%

It did save. Presumably, _autosave1.zip is the save you used to start the server, so that's what it saves to.

[Myth494]

Thats what i thought was weird, having no error and it showing the USER doing something and the fact that there was no input from us to the server, but it shutdown.. it has only ever happened after it has been running for quite a few hours.
When the server shuts down, the main save is 30+ hours behind and the latest autosave has sometimes been 15 minutes or so behind. So not every time it saves the server. i will try to grab the logs for it next time it happens, just thought i would bring this to someones attention.

I would suggest having a shutdown command that perhaps an admin can do ingame to gracefully shutdown, or a command in console even

[tclack]

I use /quit on console for the game to exit gracefully and save back to the original file.

[Oxyd]

Indeed, /quit on standard input (or over RCON) works. SIGINT, SIGTERM and SIGHUP also result in clean shutdown, which means you can just Ctrl-C for a clean shutdown.

Nevertheless, signals like that don't have to be generated by you – it's entirely possible that some other process is sending those signals. “Received SIGINT” doesn't mean that a user sent it. Perhaps some watchdog or something goes like “Huh, this process has been running for 30 hours, it must be stuck” and helpfully kills it. That is, of course, a question of your server configuration and outside our control.

[TruePikachu]

The best thing to do when unexpected signals are being sent is to check the system log files (e.g. `/var/log/messages`).