Friday Facts #76 - MP inside out

[immibis]

Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?

Well yeah... why do you think it re-downloads the map if any one of those hundred thousand entities differs slightly?

[MF-]

Have the "auto-save" become "auto-push", so all the clients still run on there own but one player pushes there game to all other clients. So this way everyone wins.

Also auto-save hapens in a reasonable time because it doesn't happen over the internet.
"auto-push" would therefore be the initial map download happening every now and then.

They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.

That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?

Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?

What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?
Everyone downloads the master player's version, hoping that not too much went wrong in their camps?

[ssilk]

They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.

That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?

When multiplayer is stable of course. Which is in my eyes with the next version. If I would be a developer I would say, to replace this checking-code with the next version. Not now, cause that has the risk of bringing in new bugs.

Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?

As explained, it's a weighting of risks. I won't do that, cause the current version is not thought for that. Simple logic. The devs need to find bugs. If they add an option "turn down checksum traffic" now, they add the risk, that it is not only turned of by those, who need it, but also by those, which often play and so reduce the chance of finding new bugs easily.

What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?

The cause and the error will have a larger distance in time. It will be much harder to find the cause then, cause it can happen at any time after the last check.

Everyone downloads the master player's version, hoping that not too much went wrong in their camps?

That would be the worst case. they need the bugs first! Then you can continue playing.

[kovarex]

Have the "auto-save" become "auto-push", so all the clients still run on there own but one player pushes there game to all other clients. So this way everyone wins.

Also auto-save hapens in a reasonable time because it doesn't happen over the internet.
"auto-push" would therefore be the initial map download happening every now and then.

They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.

That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?

Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?

What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?
Everyone downloads the master player's version, hoping that not too much went wrong in their camps?

The checksum is crc of some easy to calculate data, crc is 4 bytes, so you don't have to worry

[ofca]

Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?

Well yeah... why do you think it re-downloads the map if any one of those hundred thousand entities differs slightly?

Please note that this answer doesn't even fly close to answering my post and I'm still interested.

[chainedlupine]

Good job on the NAT punchthru, BTW.

Seriously. Even with the LAN issues.

I've played AAA+ big-budget titles with peer-to-peer lockstep networking which couldn't even get that part right.

[Tribble]

Definately loving the technical posts! Keep up the good work!

[starholme]

Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?

Yeah, you are correct in saying that anything that breaks determinism(other than the player) is a bug, cheating, or hardware issue. The whole point of lockstep computing is that things are identical over a period of time. You can't solve the last two generally, so you still require things like the crc checks to determine when a client is out of sync. Once a client is out of sync, you must place them back in sync, rolling back any changes since the last time the crc passed. Currently factorio appears to use similar functionality as joining a multiplayer game: Download the entire save file from the host/other clients. That's probably the simplest approach, you really don't require much 'new' code.

I'm not a dev, so I can only guess, but it appears that they want to get the game to the point where you only ever get a desync when a player cheats, or there is a hardware glitch. At this point desync's would hopefully be so rare that it will NEVER make sense to spend developer hours making desync recovery faster. If it takes 100 dev hours to halve the desync recovery time, and your average player only see's a desync every couple hundred hours of gametime, those 100 hours are better spent elsewhere.

[darkrising]

I like the idea of p2p but what about the poor guy that is sitting on a 3G connection behind some crappy gateway which doesn't allow open ports? I'm assuming they would have negative effects on other parties playing the game. If a server could be setup on a box open to the internet with the same p2p principle surely that would make everything better for the dude on the 3G connection

[ofca]

Main problems in multiplayer that I see are:
- Desyncs
- Conflicts
- Responsiveness / Latency / Drops, general connectivity
- Governance / Administrative issues

Desyncs:
Solved in Factorio by redownloading the game state from [somewhere]. How the game determines who is desynced and who's not? How is this [somewhere] determined when there are, lets say, 20 players? Like starholme said, preventing random rare non-recurring non-deterministic desyncs will be a waste of developer time, since they are easily fixed after being detected. One direction that might be worth exploring, especially in light of game's ambition to grow infinitely which may cause problems with many megabytes (or more) data to be re-downloaded, may be some notion of snapshots/milestones, so not whole state has to be downloaded, but instead changes from last known good milestone? Obviously there's still the last resort of downloading the game state from [somewhere]. This milestone may be an autosave - there's the question of how much memory/computational time such an approach will take. Anyway, just an idea, in case somehow devs didn't think about it

Conflicts:
Two or more players are manipulating/interacting with the same object, be it a tile with conveyor belt and resources, or an enemy being hit by bullets. How are the conflicts resolved? Or are they even resolved, or the game is just allowed to go into desync state, which is then taken care of by desync resolving mechanism?

Latency:
Well, to be honest I don't see how this will be resolved while keeping the current architecture. Currently there's just no way for player with inferior hardware or terrible connection to play with others while not ruining fun for everyone. Everyone needs to receive this lagging player's action(s)/crc of game state before game may proceed to next step, which depends on 1) player's machine fully calculating current state of simulation, 2) player transmitting previously calculated information to every other player.

Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc.. Steam ID integration would work nicely here, to keep track of players' resources/inventory. There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can, and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.

What are your thoughts? Are there more aspects that I didn't think of?

[Nova]

Can't we just let this topic rest until we get the "real" multiplayer?

[ofca]

Will getting "real multiplayer" satisfy my curiosity? Or is that illegal in your state?

[Piranha]

Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc..

One could probably create a mod to do this, if not.. post it to the ideas section and I strongly feel the devs would implement it for one of the next updates. It really shouldn't be to hard to set up the code for this.

Steam ID integration would work nicely here, to keep track of players' resources/inventory.

This is obviously the easier answer, and would make the kick/ban thing simple. Soon steam soon you will be blessed with having Factorio.

There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can

Check out the Videos section, there are currently 3 PvP series that have been played. None that were anything but friendly.
-Fishsandwich vs Xterminator
-UK vs USA
-Piranha vs Klonan
..and no DDoS attacks.. Although Fish and Nick may have been attempting that at one point

..and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.

I disagree, once 24/7 servers are up having a server that purposely ruins game play for specific people would be pointless as they would just leave and find another server.


My guess is 24/7 servers would be under 3 main category, Vanilla, Modded, PvP. Vanilla and Modded could just be ran 24/7 with admin privileges given to players so that no one person can destroy the world for others, with the possibility of pausing the game when no one is in the server, and maybe even private servers with a password or something similar. PvP would be need to be handled differently where two teams build bases and attack each other with one team the eventual winner and map resets. This is where my idea of having a capital structure would come in handy.

Anyways, whatever is in the future will be great. This game is awesome and I can see the potential for it to grow and grow and grow!

[ofca]

Uh, you totally misunderstood my post. I seem to be getting this a lot here, wonder if it's on my end

Lets go from bottom to top, because why not

Servers deciding whether a player is in sync or not wasn't meant as a way to ruin fun for players joining, but to prevent the clients from messing with others' game. Again, I'm not sure how the sync state and who's in sync or not is decided today.
DDoS attacks, hacking and stuff is the future once the game hits the wide world. Lets not compare current friendly PVP matches played by dedicated factorio fans to bored kids that will get their fun out of ruining game for others, especially streamers.
On the administrative part, I actually would like to agree with Nova - this is one part that may easily wait for final multiplayer shape.

[kovarex]

Desyncs:
Solved in Factorio by redownloading the game state from [somewhere]. How the game determines who is desynced and who's not? How is this [somewhere] determined when there are, lets say, 20 players? Like starholme said, preventing random rare non-recurring non-deterministic desyncs will be a waste of developer time, since they are easily fixed after being detected. One direction that might be worth exploring, especially in light of game's ambition to grow infinitely which may cause problems with many megabytes (or more) data to be re-downloaded, may be some notion of snapshots/milestones, so not whole state has to be downloaded, but instead changes from last known good milestone? Obviously there's still the last resort of downloading the game state from [somewhere]. This milestone may be an autosave - there's the question of how much memory/computational time such an approach will take. Anyway, just an idea, in case somehow devs didn't think about it

Waste of time is creating snapshots. The solution is, to just avoid desyncs completely, which we kind of achieved already, at least for our game.

Conflicts:
Two or more players are manipulating/interacting with the same object, be it a tile with conveyor belt and resources, or an enemy being hit by bullets. How are the conflicts resolved? Or are they even resolved, or the game is just allowed to go into desync state, which is then taken care of by desync resolving mechanism?

This problem is non-existent. One of the players is always the first one to interact with the object.

Latency:
Well, to be honest I don't see how this will be resolved while keeping the current architecture. Currently there's just no way for player with inferior hardware or terrible connection to play with others while not ruining fun for everyone. Everyone needs to receive this lagging player's action(s)/crc of game state before game may proceed to next step, which depends on 1) player's machine fully calculating current state of simulation, 2) player transmitting previously calculated information to every other player.

This has nothing to do with latency, but rather the sustainable game speed. Well the solution is to optimise the game. People with bad hardware will just have to avoid playing huge maps anyway, we are not building a MMO here.

Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc.. Steam ID integration would work nicely here, to keep track of players' resources/inventory. There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can, and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.

Why would you DDos anyone? Why don't just disconnect from the game and pretent the other one disconnected? This problem is also non-existent. If you play competetively, you have some kind of account. People that are reported to be cheating a lot will just get banned, really easy.

[ofca]

Yes, it's obviously best to get rid of desyncs all-together, and I didn't see much of them even before 0.11.17

I wouldn't DDoS anyone. And most of the multiplayer problems don't exist if you plan with one other person - problems, actually all kinds of them, start to show when you're playing with multiple people over the Internet -- but it was stated numerous times that 0.11.x in no way represents the final quality of multiplayer, and it wasn't intended for Internet-based play; that's why I'm so interested what major changes are in the works for 0.12 that will make Factorio Internet-worthy.

From what I gather the effective latency (time it takes for tick to be processed -- or plainly, time between pressing w and your character moving) is slowest player's time to render next simulation frame+time needed to reach furthest player, right?

And one last thing that interests me the most: why did you choose the peer2peer approach instead of some twist on client-server? I don't advocate that it's a bad choice or anything, merely would like to know the reasons (and especially gains of one over the other) - some of the reasons may be extrapolated from latest FFF, but it feels sadly incomplete and leaving me hungry for more details

Thanks for your time

[kovarex]

The main reason why we don't consider it to be fully internet ready is the latency. The problem is the delay between sending message and receiving it (ping time).

Solving this problem is the main part (apart making some better support for opposing teams) of the changes we plan for the 0.12.

The solution is to act like player started moving immediately, although the "real player" will start to move later. The mechanism needs to be simple, and there is never ever some kind of reversing, you really can't reverse in Factorio. The "ghost player" will just be reversed to the real player when he can't get to the position anymore, but this is mainly visual trick to make the game feel more responsive.

[starholme]

And one last thing that interests me the most: why did you choose the peer2peer approach instead of some twist on client-server? I don't advocate that it's a bad choice or anything, merely would like to know the reasons (and especially gains of one over the other) - some of the reasons may be extrapolated from latest FFF, but it feels sadly incomplete and leaving me hungry for more details

I thought they went into pretty good detail. The biggest factor is that Factorio has the interesting property of ridiculous amounts of active things. The quantity of things that are happening essentially requires that all the work be done client side to reduce the network traffic. The slowest client computer controls the tick speed of the game.
In a client server model, you could run the tick speed at whatever the server could handle. But you have to send every action of every entity every tick(or at least every entity within a certain range of the client avatar). Requiring way more bandwidth.

So the client only sends player actions and crc's, and only receives player actions and crc's. Nothing else. Pretty much impossible to reduce the amount of traffic below this.

Currently, each client sends all their actions and crc's to every other client. So the latency is the round trip time between the two farthest clients. If you went to a 'lite' client-server model where the clients send all the actions and crc's to the server, then the server sends to all the other clients, your latency would be the round trip time between the farthest client to the server, PLUS the round trip time to the second farthest to the server. The server would also require enough bandwidth to forward all these messages.

Another overlooked benefit of p2p is that you should be able to keep playing if original hoster disconnects. When he boots back up(cause windows update restarted his computer without asking), he can just join again, using the address of any player still running.

Looking forward:
I wouldn't be surprised if eventually one of the clients in the p2p group becomes 'special'. There may be a special one already for all I know. This special client might handle the tasks that a server would in most games.
Things like being a proxy between clients if a client isn't reachable from the internet directly. Or deciding who can join the game, or who should be kicked from the game.
A 'dedicated' server wouldn't need to be special, really just another peer that doesn't have an avatar. Ideally, it could run without a GUI, and handle pausing/unpausing when there are no other clients.

[DaveMcW]

If you went to a 'lite' client-server model where the clients send all the actions and crc's to the server, then the server sends to all the other clients, your latency would be the round trip time between the farthest client to the server, PLUS the round trip time to the second farthest to the server. The server would also require enough bandwidth to forward all these messages.

No. The latency for 'lite client-server' would be:

farthest client->server + server->farthest client
OR
farthest client->server + server->second farthest client

In other words, every client has latency equal to the average of their own connection to the server and the slowest client's connection to the server. The slowest computer would still set a limit on the speed of the simulation, due to the lockstep architecture.

This method scales up extremely well (assuming your server can handle everything). It is the only reasonable choice for massively multiplayer games.

[ofca]

Yes, I would be looking at some hybrid mode too, but when game hits steam and our thousands of positive reviews convince others to buy the game, we will see how well Factorio will scale into tens of players on the wild internet