Reducing savegame size/disabling replays

[max]

A friend and I have been playing multiplayer for around 40~ hours total on the same map and the map sync time has become increasingly problematic. I looked at the saves folder and the latest savegame is a ~48MB zip file, which on a 5mbps connection upstream takes around 90 seconds to upload, this becomes worse as we try to add players.

I noticed that inside the zip file there is a file named 'replay.dat' which clocks in at around 128MB uncompressed. If I delete this file and rezip the save, it decreases in size more than 80% to ~8MB. The game now loads significantly faster, transfers to other players faster, and autosaves faster. History graphs of production and electricity seem to still work fine as well.

Short of wanting to watch a 40 hour replay of everything that's happened to the map, is there any reason to actually keep this file around? Why are replays saved by default? Is there any way to turn them off short of deleting them?

I noticed there are multiple locale files in the saves as well. Are those needed? Are there any other tricks to reducing savegame file-size?

[roy7]

Oh man, if that replay data is only needed for replays and a game could be configured to disable replay, that'd be sweet for multiplayer with lots of players.

[max]

I even tried to load some of the replays in our old map, and every single one of them fails claiming "desynch at tick 93", so the replays aren't even useful for their intended purpose. It's not entirely clear why this feature is on by default.

[OBAMA MCLAMA]

I'm not finding any replay.dat in any saves that were saved via 11.10, But i wish i knew about this earlier.

[Choumiko]

I'm not finding any replay.dat in any saves that were saved via 11.10, But i wish i knew about this earlier.

Haven't updated to 11.10 but are you using mods? If you change/toggle(update?) mods it seems that the replay.dat gets deleted. At least that's what my savegames i just looked at suggest.
But a setting would definitely be usefull

[max]

I'm not seeing replay.dat in new savegames now that I'm on 11.10 so it may be off by default now.

[ssilk]

A replay for Multiplayer is quite useful!
Why?
Timestamp 1: player 3 connects to a game with two other players. The game-state is copied in the background. The game is sved. None of the two players sees any glitch. The game runs smooth. Player 3 begins to download the saved game.

Timestamp 2: player 3 is ready with downloading. The game is initialized, but he cannot connect yet, cause his save is from time stamp 1 not 2. Instead of that, he will download the replay save. Player 3 can now "fly" over the landscape, while the replay is replayed.

Timestamp 3: it tooks only half a minute, but now player 3 is comming near to the real game time. A complex synchronization process begins, so that all game instances on all three computers have the same state. In the end of that, the game might glitch a bit, but then it runs fluid.

So this is - in my eyes - the plan: instead that all players need to wait, only one needs to, which is the one, which wants to connect. He downloads the map from timestamp 1 an runs the replay from that point, until the simulation catches up. This works very well, if the simulation is much faster, than the game-speed. Which works if they continue the optimizations. In the other case, the worst thing that happens is, that the connecting computer is too slow, to catch up. In that case, the others need to wait. But in that case the whole game might slow down. Time enough to kick him. :)

[immibis]

A replay for Multiplayer is quite useful!
Why?
Timestamp 1: player 3 connects to a game with two other players. The game-state is copied in the background. The game is sved. None of the two players sees any glitch. The game runs smooth. Player 3 begins to download the saved game.

Timestamp 2: player 3 is ready with downloading. The game is initialized, but he cannot connect yet, cause his save is from time stamp 1 not 2. Instead of that, he will download the replay save. Player 3 can now "fly" over the landscape, while the replay is replayed.

Timestamp 3: it tooks only half a minute, but now player 3 is comming near to the real game time. A complex synchronization process begins, so that all game instances on all three computers have the same state. In the end of that, the game might glitch a bit, but then it runs fluid.

So this is - in my eyes - the plan: instead that all players need to wait, only one needs to, which is the one, which wants to connect. He downloads the map from timestamp 1 an runs the replay from that point, until the simulation catches up. This works very well, if the simulation is much faster, than the game-speed. Which works if they continue the optimizations. In the other case, the worst thing that happens is, that the connecting computer is too slow, to catch up. In that case, the others need to wait. But in that case the whole game might slow down. Time enough to kick him. :)

The complaint is that the replay is part of the savefile, so player 3 downloads the 80MB replay of everything that's happened so far that he doesn't need.

[FishSandwich]

I quite like ssilk's scenario. I hope something like that gets implemented in the future.

[max]

I think it's reasonable, and background loading would be nice, but not at a potential 500% increase in save file size. If the current save is only 8MB, the new connecting player only needs replay data from the moment the save began sending in the background, not from 40 hours previous when the game was first started. Right?

On a side note, more multiplayer options would be very nice. I've been hosting a game where people connect from various parts of the world; I happen to be geographically in the center of them. It would be helpful if we could set preferred hosts (so when I desynch momentarily, it doesn't start hosting the game across the world from everyone else). A writeup about how all the multiplayer systems work would be really interesting!

[immibis]

It would be helpful if we could set preferred hosts (so when I desynch momentarily, it doesn't start hosting the game across the world from everyone else). A writeup about how all the multiplayer systems work would be really interesting!

The game runs symmetrically on all computers; none of them have special powers, except for minor things (like in the event of a desync, which computer is decided as having the "correct" map).

[ssilk]

The complaint is that the replay is part of the savefile, so player 3 downloads the 80MB replay of everything that's happened so far that he doesn't need.

I agree, that - from MP view the transfer of this is currently not useful.
I think there should be a "world" (one (zip?) archive?) and in that archive we have the current saves to different timestamps. The quicksaves are then just one of many.
And also the replays.

When a new player wants to connect, it will look up for the latest save, transfer that and then the replay-save from that point.

But this offers some more cool scenarios:

Player A and B are playing a game together. Player B leaves, but A stays in game. Player B wants to re-connect (some hours or a day later).
If Player B has the save he just needs the replay save from the time, he has left the game.

More is thinkable: If Player B's Rechner is so fast, then it is possible to load the former save (which should be smaller, cause the map etc. is eventually not so big yet) and do a longer replay instead. Less data, more CPU.

This goes so far back, that it might be possible to just have the map seed and the replay. How crazy would that be? I think it might be possible, if you played in the early game just for an hour to replay everything in a minute or so...

[daniel34]

Was thinking of posting this in the bug reports forum, but I'm adding it here because this thread already contains relevant posts.

I'm currently running a public dedicated server in this thread, which works fine most of the time, except for desync-loops every few days or so.
I'm usually setting up a new map locally on my pc, so I can preview the map and look around for a few minutes before I load it onto the server.
Before I upload it, I remove the replay.dat so that the save file stays small.

The problem is that sometime later the replay.dat is back in the savegame. So I stop the server, remove the replay.dat from the save again, launch the server with the edited _autosave file and it's running fine again. When I check sometime later (few hours or maybe a day) the savefile contains the replay.dat again. I've deleted them several times over the last few days and a few hours later (maybe just minutes) they reappear.

It doesn't make sense for now that the save only stores a partial replay.dat because there currently is no use for it. Even taking ssilks idea into consideration the joining user only needs the replay data from the moment he starts to join to replay the map to it's current state.

[keyboardhack]

Apparently the replay.dat is used by the devs to find out why your game crashed when you file a bug report with an uploaded save. if you don't care about allowing the devs to fix bugs then you should be able to remove the file with any problems.

[daniel34]

Apparently the replay.dat is used by the devs to find out why your game crashed when you file a bug report with an uploaded save. if you don't care about allowing the devs to fix bugs then you should be able to remove the file with any problems.

Like I said i'm removing the replay.dat, but it is somehow added to the save sometime later. Maybe a checkbox for that when generating a new multiplayer save might be a good idea, because most people won't even be aware that the full replay is saved in the savegame, and on a dedicated server with Marathon+RSO mod it can get huge. I'm not sure how the replay data is saved but I suspect the more players there are the more data gets generated.

[FishSandwich]

Enabling/disabling mods will disable replay saving.

[daniel34]

Can I do that on a dedicated server? Removing the mods, then launching the game, save and quit it, add the mods back in and then start the server again?

Or do I need to copy the save to my local pc and try it there?

EDIT:
Removed the mods from the mods folder, started the server, then exited the server (CTRL+C) which saves the game
The replay.dat was automatically removed from the savegame, and then I restarted the server with the mods.

EDIT2: Looked promising, but didn't work, game desyncs as soon as a player joins. I've used an earlier autosave and removed the replay.dat.

[daniel34]

Tried a second time, removed mods then restarted the server, canceled it with CTRL+C, added mods again and started the server.
Now it's running, hopefully the replay.dat is gone for good, the save without it is already 20MB.