Multiplayer "Catching Up" - 5000 bots out of storage

[Elok]

Hello everyone.

So I'm playing with friend on my server (I'm connected directly via my router, my friend connect over the internet).

We were playing fine but, all of a sudden, the game started lagging like crazy so we all disconnected and I restarted the server. But when we tried to connect, we were all having problem to connect on the server as our game failed to catch up.

After moving my game locally on my computer I did a few test. It turn out it happenned after we send our construction bot deconstruct our old base. We done it a few time without any problem but, this time, we reached the limit of our storage and all of our ~5k construction bots all went into "no storage avalaible" at the same time. So I just added a few storage chest and the problem disappeared.

So I went back into my server and tried to connect (I stopped all irrelevant program on my server and computer) and able to connect after a good 30sec of catching up. The game was running at less then 1FPS. But as soon as I placed a storage chest all 5000 bots moved and the FPS raised to 30 FP and the ticks dropped to 7-10 ticks.

It seem there's a sort of memory leak there.

[Rseding91]

It seem there's a sort of memory leak there.

Thanks for the report but that's not what a memory leak is. A memory leak is memory allocated by the program that it nolonger has any way to release.

What you described is just your CPU not being able to simulate the game fast enough to keep up with the server.

[Elok]

What you described is just your CPU not being able to simulate the game fast enough to keep up with the server.

Are you saying that the cause is the CPU of our computer or server not good enough?

EDIT : I just saw that you moved my post in the "Not a bug" category and I have to disagree.

The game on our server run perfectly smootly and our base isn't "that" big. It's not normal that the game goes from +60 FPS to less than 1 FPS because the bots ran out of logistic storage.

Unless you consider this "working as intended"?

[Jap2.0]

What you described is just your CPU not being able to simulate the game fast enough to keep up with the server.

Are you saying that the cause is the CPU of our computer or server not good enough?

EDIT : I just saw that you moved my post in the "Not a bug" category and I have to disagree.

The game on our server run perfectly smootly and our base isn't "that" big. It's not normal that the game goes from +60 FPS to less than 1 FPS because the bots ran out of logistic storage.

Unless you consider this "working as intended"?

In multiplayer, the game sacrifices FPS to gain UPS to attempt to keep up with the server.

[Elok]

In multiplayer, the game sacrifices FPS to gain UPS to attempt to keep up with the server.

I'm not following.

If you're right, you're simply adding a paramater in the issue.

So then, why 5000 bots filling all avalaible logistic storage space and falling of "no storage left" is suddently using that much "UPS" causing the server to sacrifice so much FPS that nobody can connect on the server anymore?

[Zavian]

UPS is updates per second. It is measure of how many game update your processor has processed in the last second. The game isn't using ups. It is using cpu cycles. UPS measures whether the game is slowing down because it is running out of cpu cycles. 5000 bots all looking for through all your storage chests every game update will use more cpu cycles than if those 5000 bots could each find a chest with space and fly to it.

[posila]

Hi, if you think there is an performance issue, the best way to report it is to provide save file with the issue happening.
From the description it looks like just as a straw that broke the camel's back.

[mrvn]

Seems to be 2 problems here, one of them I consider a bug:

1) Bots without storage chest consume a lot of CPU. This makes the UPS drop like crazy with 5k bots lacking storage.

It's not good but probably not a bug. Could maybe be optimized in some way like keeping all bots without storage in a lists (one per item) and only checking once a tick if there is any storage. That would cut the cost down to O(num item types), a constant, no matter how many bots you have.

2) The server does not slow down UPS to allow clients to catch up and match speed.

Now this I consider a bug. It basically means that the server must be the slowest computer in the game. Otherwise at some point you no longer can join the game. Also as the game reaches the CPU limit you get massive lag making the game not fun if not outright unplayable.

If you experience increasing lag in the game that isn't just your internet being slow (like on a local LAN) try setting the game speed lower. Suddenly the lag is gone. And joining clients have a much easier time catching up when the server runs the game slower.

Imho this should happen automatically. The server should automatically slow down the further behind clients are so that they can catch up easier on join and keep the game responsive later.

[Jap2.0]

Seems to be 2 problems here, one of them I consider a bug:

1) Bots without storage chest consume a lot of CPU. This makes the UPS drop like crazy with 5k bots lacking storage.

It's not good but probably not a bug. Could maybe be optimized in some way like keeping all bots without storage in a lists (one per item) and only checking once a tick if there is any storage. That would cut the cost down to O(num item types), a constant, no matter how many bots you have.

2) The server does not slow down UPS to allow clients to catch up and match speed.

Now this I consider a bug. It basically means that the server must be the slowest computer in the game. Otherwise at some point you no longer can join the game. Also as the game reaches the CPU limit you get massive lag making the game not fun if not outright unplayable.

If you experience increasing lag in the game that isn't just your internet being slow (like on a local LAN) try setting the game speed lower. Suddenly the lag is gone. And joining clients have a much easier time catching up when the server runs the game slower.

Imho this should happen automatically. The server should automatically slow down the further behind clients are so that they can catch up easier on join and keep the game responsive later.

1. Could just be the last little thing (the straw that broke the camel's back, as Posila said) and not have that much of a preformance impact, but I'm guessing that in this case there's probably room for optimization. Of course, I have no idea how any of this works internally, so that's not my call to make.

2. It'd be nice to have a way to do that automatically - for those who don't know, this can be done with /c game.speed=X where X=1 is normal speed, so X=0.8, for example, would change the server speed to 48 UPS.

[Elok]

Please put yourself in our shoes.

We play normally, everything is fine and then, suddently, the server crash and it's impossible for us to join the game, leaving alone diagnosing the cause. And the only error message the game is giving us is "failed to catch up with the server". And after 1-hour of google search without any help, we almost threw the tower and deleted our game.

I had to be stuborn enough to import the game on my computer, play the game at 1 FPS and try to find the root of the problem. It wasn't easy and I had a lot of luck to find the reason in the end.

If anyone find that "this" is working as intended, I think that, at least, the game should give us an error message explaining what's the main cause for the game failling to catch up.

Elok

[mrvn]

@elok As said, I consider the catching up failure a bug. I had the same problem in a multiplayer game and we had to try running the server on different computers till we found one that was slow enough so all of us could join again.

@jap2.0 Note that using the lua command disables achievments. There are (more than one iirc) mods to change the game speed though. But then you only get the modded achievments, not vanilla achievments. Personally I don't care about that but it remains sub optimal.

[Elok]

@elok As said, I consider the catching up failure a bug. I had the same problem in a multiplayer game and we had to try running the server on different computers till we found one that was slow enough so all of us could join again.

I know mrvn don't worry, my post wasn't about you.

[Jap2.0]

@jap2.0 Note that using the lua command disables achievments. There are (more than one iirc) mods to change the game speed though. But then you only get the modded achievments, not vanilla achievments. Personally I don't care about that but it remains sub optimal.

I agree that it's a suboptimal solution and there should be a way to do it without losing achievements/automatically, but for the moment it's a fairly good workaround - it's neccessary to make most multiplayer megabases work, and I'm past the point where I can easily get any achievements for multiplayer (though I get that this may be different in some scenerios).