Debugging a multiplayer desync?

[lettherebelight]

I've got a bug report from someone using my lighted electric poles mod that when the poles are used in a blueprint during a multiplayer game it results in a desync.

They sent me a desync report, but I'm not familiar with how to interpret the report. I took a peek at log.txt and all it says is errors like this:

5025.883 Error GameActionHandler.cpp:2101: Multiplayer desynchronisation: crc test (heuristic) failed for crcTick(13657932) serverCRC(-2126664724) localCRC(-314132532)

There's no information in the log file about what line of my mod caused the desync.

How do I go about debugging this?

Thanks!

[matjojo]

All I know about that is that for multiplayer compatibility your mod should do the EXACT same thing for all computers it runs on. And make sure to not use math.random() as that works differently on different OS's

[Articulating]

And make sure to not use math.random() as that works differently on different OS's

Can you verify that? I've never had an issue with it and I'm pretty sure it doesn't desync. If I'm wrong, well... that kinda sucks. Also @OP I think you should check which other mods the user in question has installed. Also should you be making the hidden-small-lamp non-blueprintable? Since it can't be made by the user and you place the lamp anyway (unless it is already not able to be blueprinted, from what I saw on your GitHub it doesn't seem this way but I could be wrong.)

[daniel34]

And make sure to not use math.random() as that works differently on different OS's

Can you verify that? I've never had an issue with it and I'm pretty sure it doesn't desync. If I'm wrong, well... that kinda sucks.

Using math.random() in multiplayer doesn't cause desyncs. It is based on the map seed and deterministically written by the devs to always give the same result on all clients, independent of the OS.

viewtopic.php?f=25&t=12451#p83708

Factorio doesn't allow you to read system time and the math.random uses the determistic Factorio random based off the save game. That means you can use math.random all you want and it won't break the game or cause desyncs.

[matjojo]

viewtopic.php?f=25&t=12451#p83708

Factorio doesn't allow you to read system time and the math.random uses the determistic Factorio random based off the save game. That means you can use math.random all you want and it won't break the game or cause desyncs.

Well thanks a lot, I never knew that, I don't really like randomness all too much, but when I need it this will be useful. I'm sorry for spreading misinformation.

[lettherebelight]

Thanks for the reply. I'm not using math.random(), and the mod is pretty simple. Anyone have suggestions for how to debug? I don't want to just stare at the code and randomly guess at what might be wrong.

[Choumiko]

A link to the mod would be helpfull, else we stare at nothing and guess what might be wrong

I assume it's this one: https://mods.factorio.com/mods/letthere ... ctricPoles ?

I'll take a quick look later today if i can spot anything obvious.

Edit: since the control.lua is so small: Nothing obvious sorry.
100% sure the desync has to do with your mod/can reproduced with only your mod active?

[lettherebelight]

Yes, that's the mod, my bad

I only have one report from one mutli-player user about the desync, but apparently it happens when a blueprint is placed during multiplayer with one of the poles. I'll follow up with them and ask what other mods they're using, although I suspect the list will be... lengthy.

I might have to figure out how to multiplayer just to try and reproduce the problem with only my add-in. Ugh. There's nothing in the desync dump they sent me that would help identify the culprit is there?

[Nexela]

I might have to figure out how to multiplayer

Download the DRM free version from the website and install it into a separate folder.

Install the mod into both versions

Start multiplayer on your first one using lan and if there is a verification option disable that, Connect to lan with the second factorio.

[Mooncat]

Install the mod into both versions

Or add --mod-directory to the shortcut target and point to the original mods folder.
Reference: https://www.reddit.com/r/factorio/comme ... o_use_the/

[lettherebelight]

So back to my original question: how do you debug something like this? Even if I do set up a multiplayer game and reproduce the problem, how on earth do I track down the line of code that's doing something bad and causing the desync?

[Klonan]

So back to my original question: how do you debug something like this? Even if I do set up a multiplayer game and reproduce the problem, how on earth do I track down the line of code that's doing something bad and causing the desync?

First step is getting concrete steps to reliably reproduce the issue

[Optera]

I'm trying to debug a "crc test (heuristic) failed for crcTick" myself.
Everything in on_tick is either global or a metatable derived from global tables during on_load (that's mp save right?)

Is there a way to get more informative debug info than a crc failure?

[Articulating]

You can diff (I use Beyond Compare but I only have a free trial) the files in the desync report, specifically the level.dat, level_with_tag_ticks.dat and script.dat files would probably be the most useful. You aren't guaranteed to be able to figure it out from that and interpreting the results can be tricky at first but it's the best we've got.

[Nexela]

You can diff (I use Beyond Compare but I only have a free trial) the files in the desync report, specifically the level.dat, level_with_tag_ticks.dat and script.dat files would probably be the most useful. You aren't guaranteed to be able to figure it out from that and interpreting the results can be tricky at first but it's the best we've got.

Also If this is easily reproducible doing it on a new small map with only your mod present will have a lot less unnecessary data in it.

[Optera]

You can diff (I use Beyond Compare but I only have a free trial) the files in the desync report, specifically the level.dat, level_with_tag_ticks.dat and script.dat files would probably be the most useful. You aren't guaranteed to be able to figure it out from that and interpreting the results can be tricky at first but it's the best we've got.

Thanks for the information. I really wish we could get a desync in function xyz at line 123 error message.

Also If this is easily reproducible doing it on a new small map with only your mod present will have a lot less unnecessary data in it.

Sadly I'm not so lucky. So far I got the desync only once and have failed to reproduce it.

[Adil]

The desync logs are only useful for devs. As a modder you basically have only the way of staring at the code persistently till the bug presents itself in the open.
For the lighted poles I think the cause lies elsewhere, maybe there's some mod that does things to blueprint or something.
By the way, the log file already contains information on which mods were used by player.

Everything in on_tick is either global or a metatable derived from global tables during on_load (that's mp save right?)

on_load is only called for a person who is connecting, with some ingenuity it is still possible to write the code that makes things different.

[orzelek]

The desync logs are only useful for devs. As a modder you basically have only the way of staring at the code persistently till the bug presents itself in the open.
For the lighted poles I think the cause lies elsewhere, maybe there's some mod that does things to blueprint or something.
By the way, the log file already contains information on which mods were used by player.

Everything in on_tick is either global or a metatable derived from global tables during on_load (that's mp save right?)

on_load is only called for a person who is connecting, with some ingenuity it is still possible to write the code that makes things different.

Might also try using on_player_connected to double check the init.