[1.1.80] Headless Server - Seemingly unavoidable crash at same point

[xeode]

Hi there,

Hoping someone can help, I went to connect to my long running Vanilla server the other day and found it not responding and after logging into the console of the server I see that the game has hung without any error. If I reload the last save within a consistent amount of time the factorio process stops responding and connected clients are also frozen (can't even load the menu). Factorio 1.1.80

If I load an earlier autosave all is well until the same time after the same numbered auto save. I have tried connecting just after the earliest autosave available (before going into the backups of earlier games and losing a few days of progress anyway) and connecting to turn off all factories and trying to stop any activities which might be causing a repeatable crash too. But seemingly whatever i do, a few minutes after autosave4 is saved and the game resumes, the same hard crash happens. I have found lots of cases of it crashing during save but it seems to be well clear of completing autosave4 by the time the crash happens.

Server itself seems to be fairly quiet when it happens and no shortage of disk space or other thing I could think to check and if I save the game to my PC and load it locally it seems to crash after the exact same amount of time. Have never had this happen before either local or mp and can't find anything about it through search, just those who are having issues during saving.

Save file if anyone needs it: drive dhowt google dhowt com/file/d/1peDjRlsquxww1-EALETNT7sm8YeHEBsA/view?usp=sharing

Are there any things I can do in game to try and alter the collision course before the deadly juncture rolls around or any extended logging that can be turned on to understand what it tries and fails to chew on, or indeed any thoughts at all as I'm at an impasse and have several years on this server now I'm keen to keep hold of.

Thanks for any thoughts that come to mind anyway.

Kind regards,
xeode

[Rseding91]

How long do I need to wait before the freeze happens?

Does it only happen if hosted as a headless server or does it happen if loaded as a single player game?

[xeode]

Hi there,

So this is the earliest save from the rotation so i could have as much time as possible to go run around and turn things off before the crash came but not sure how many minutes that is. It does seem to load and crash at the same point on the server and on the single player, will have another try and have a timer going. Thanks for having a look.

Kind regards,
xeode

[xeode]

Hi there,

Looks like it's about 37 minutes from that save, here's one that's a bit sooner if you don't need the time:

drive dhowt google dhowt com/file/d/1gPLPxqZUnBgZJJZYhujQ0tm8TAqrNBve/view?usp=sharing

Kind regards,
xeode

EDIT: dies just after 4 minutes after the above '_autosave3.zip', thanks for any insights.

[Rseding91]

The save file is 600 megabytes. Loading saves that large typically takes > 32 GB of RAM. Your computer shows having 16 GB which means it will likely be hitting the page file long before loading finishes.

If it ran out of RAM and started paging to the disk that will significantly slow things down. Did you run some "chart huge area" command? Why was the first save you provided 30 megabytes and this one 30 minutes of playtime later 600?

[Rseding91]

I think you provided the wrong save file the first time. The 2nd save file you provided is very close to the max play time possible on a given save (2.269~ years).

It looks like you disabled auto-pause and left the save running on the server while you weren't playing which let it accumulate time. There are some work-arounds people have done to continue playing but there is no official way to play past 2.279 years in a single save file.

[xeode]

Hi there,

Aaaaahhhhh! Never heard of a maximum world age or imagined there would need to be a register for every tick since 0 or whatever that equates to. Yes have been running the save for a long time, the vast majority of time running on my computer in Steam though. My home computer does indeed struggle to run it (45UPS with everything running) but on a 32GB server is only reserving 20 or so GB.

Ah yes very sorry have seemingly uploaded a completely different save at first beg your pardon! That looks like a space exploration one too doh.

Will have a look into the workarounds then as was hoping to run this just to see how far it could go etc and glad to find it wasn't some terminal corruption or something.

Thanks for your help!

Kind regards,
xeode

[xeode]

Hi there,

Great thanks for your tip off, have done the tick rollover with all belts removed and done the re chart option to fix the radar areas being shown as fogged and things look to still be soldiering on even after showing low tick numbers again.

Only issue now is that even with only one factory turned on the UPS a week or so later is still reallly chugging. Most of my artillery decided to notice all the enemies that were in range but were somehow being ignored until the radar shenanigans so still managing all of that though but not clear where all the computational load is now coming from and when it will calm itself down and return to 60 UPS.

There was a fair bit of time while every factory, even those that were not switched on, refilled their belts that were deleted/replaced, and I could see where a lot of work might be going, but that seems to have almost completely taken place now so might just be targetting calculations by the artillery that are yet to find their happy equilibrium again?

Most of the update time seems to be coming from entity (5-6) and electric network (0.6) when doing debugging (screenshots attached) and it seems to be in permanent slow motion/half speed still. This was fairly smooth before even with everything turned on but it seems to be really struggling with performance now with 1/8 factories going and have never seen this server wait strobing, doing like half frame rate. Went and ran a kill all visible enemies command and entity update went down by 1ms or so but still no change to performance.

Have connected from a machine with 32GB ram and seemingly makes no difference and not really sure what to try next. Had previously seen some slowdowns when opening the map etc but no matter what I do now it's in permanent half speed mode.

Thanks for any thoughts you might have.


Kind regards,
xeode

[xeode]

Hi there,

When running this save locally on a machine with 32GB of ram only 14 are actually being allocated and CPU is very quiet, and yet the world updates at 20-30 UPS even with most things not running?

Am I being irrational to presume that the performance after the game tick rollover should be the same as before it for reasons I'm missing or is there something wrong/correctable going on?

Save file below in case anyone can see anything not shown in the debug stat images.

drive dhowt google dhowt com/file/d/1vsewIb4iiCp4w9zz0qrWOkc2J5Cj6h5s/view?usp=sharing

Thanks for any thoughts that people may have or things to look at next.


Kind regards,
xeode

[xeode]

Hi there,

Hmmmm have since gone and deleted every artillery, powered radar and all the main factories (leaving a few mining bases that are spread out and annoying to find) and it still chugs so presume the save is just terminally borked at this point. Entity update taking 26ms but entity time debug only showing less than 1ms and it's been a few days without change.

Is there a good way to find your map seed and generation settings, uncover specific mask map area of new save based on the previous one, paste in a few million objects and manually set research levels back to where they were? I like the challenge of managing biters rather than just pure building but if that complicates things can sack that off. Any other ideas for getting something useful out of it?

Will be sad if I've got to delete something I've spent thousands of hours building, got done with space exploration and have a server I built for megabase ant farm going spare lol.

Kind regards,
xeode