[14.3] Unable to "Catch Up" with capable hardware

[posila]

Thanks for the save.
Not related to underlying problem, but small quality of life improvement: The way you use Warehouse mod for coal murders performance. Chest inventories are not optimized for so many slots and having inserters trying to insert to almost full Warehouse all the time just kills it. On my home Core-i5 3470S it ran ~42 UPS and after removing inserters inbetween coal Warehouses it jumped up to ~78 UPS (I set game.speed = 2 in order to get more then 60 UPS)

[Lee_newsum]

I'm going to assume that you mean to say that Factorio has a large scale. So... just about any real time strategy game does what Factorio does. Starcraft, Command & Conquer, Railroad Tycoon. Simulation games like SimCity does as well. First person shooters do this. I remember an old game I used to play, The Settlers II. It did exactly what Factorio does... and on a 486... on about a 1/100th scale.

I'd bet you'd love to see a game called EVE Online where literally _thousands_ of players end up playing together; the server needs to simulate hundreds of thousands of weapons firing, damage being processed, hit chances, AI movement, and even ship movement... and not a conveyor belt where movement is consistent in speed and direction but rather in space with dynamic speeds. And CCP has done very well to combat the actual problems by implementing solutions instead of denying that they're problems.

so I will pick out the games I no all Command & conquer work on 50 units and 30 buildings per team, 8 teams 50 + 30 X 8 =640 c&c has 640 items to keep up to date but no more than 1000 items.
eve online this looks like a big game and it is but 90% of the dater has no connection or no links between them. this thinks can be done in parallel. the biggest think is 1000 vs 1000 ship fights (this has to be done in real time) so one ship has 50 thinks to track 50 X 2000= 100,000 that is a lot and will done to ccp for doing that. and this is done on 1 core.
Factorio there about 220 items, use 1-6 Assembling machines for this 4 , that have 3+ Inserter on them that is 2640. add to that 200 Electric Mining Drill, add 20,000 iron ore, copper ore, iron ,copper, steel and 7= items that is 12X20,000=240,000 and a not more. this game has a lot of items to track in real time and on 1 core. the devs has said in the past there is no game/ software out there that dues what Factorio can do or has to do in a game tick. the bigger the map get the more items to keep up to date. this game works with items counts of 500,000 and more all in real time

[Loewchen]

2) acknowledging that our situation is already an edge/corner case, because we've got like 60 hours of factory building and a 30+mb map

Just to clear up if you misunderstood me when I filed this as not a bug:
The corner case or small window as I described it is not in regards of your map being especially big, old or performance taxing. The special case is only, that the usually inferior Intel i5-6300U processor handles the task better than the FX-8350 (this is most likely because of the shared FPU, as all Bulldozer chips perform poorly in FP operations in my experience) giving the incorrect impression that worse hardware performs better.

To sum it up:

1. Yes mulitcore support would be good, but not having it is not a bug.
2. Yes having the option to allow clients to slow down the server if they can not keep up could be useful, but not having it is not a bug either.

Greetings Loewchen

PS.: It would be best if you could set the minimum ticks/sec a client must be able to perform until he gets dropped, in the server settings, I assume this will came sooner or later anyway.

[Rseding91]

I'm going to assume that you mean to say that Factorio has a large scale. So... just about any real time strategy game does what Factorio does. Starcraft, Command & Conquer, Railroad Tycoon. Simulation games like SimCity does as well. First person shooters do this. I remember an old game I used to play, The Settlers II. It did exactly what Factorio does... and on a 486... on about a 1/100th scale.

I'd bet you'd love to see a game called EVE Online where literally _thousands_ of players end up playing together; the server needs to simulate hundreds of thousands of weapons firing, damage being processed, hit chances, AI movement, and even ship movement... and not a conveyor belt where movement is consistent in speed and direction but rather in space with dynamic speeds. And CCP has done very well to combat the actual problems by implementing solutions instead of denying that they're problems.

so I will pick out the games I no all Command & conquer work on 50 units and 30 buildings per team, 8 teams 50 + 30 X 8 =640 c&c has 640 items to keep up to date but no more than 1000 items.
eve online this looks like a big game and it is but 90% of the dater has no connection or no links between them. this thinks can be done in parallel. the biggest think is 1000 vs 1000 ship fights (this has to be done in real time) so one ship has 50 thinks to track 50 X 2000= 100,000 that is a lot and will done to ccp for doing that. and this is done on 1 core.
Factorio there about 220 items, use 1-6 Assembling machines for this 4 , that have 3+ Inserter on them that is 2640. add to that 200 Electric Mining Drill, add 20,000 iron ore, copper ore, iron ,copper, steel and 7= items that is 12X20,000=240,000 and a not more. this game has a lot of items to track in real time and on 1 core. the devs has said in the past there is no game/ software out there that dues what Factorio can do or has to do in a game tick. the bigger the map get the more items to keep up to date. this game works with items counts of 500,000 and more all in real time

Your counts are a little off. The save file linked a few posts ago has 5,180,800~ entities of which 90,400~ are active (running logic each tick).

[inetknght]

Other problems: what if somebody with significantly slower computer connects to the game? What if you want to play with such person? What if you don't?

Yup. Personally, I'm okay with having an option on the server which reduces game speed to either a configurable minimum or else to the lowest common denominator.

The game we want to make is one where Factory you build does its job even when you don't see it. We also want every item to be accounted for, no faking like "this chest gets average of 10 red circuits per minute so when you look at it after 20 minutes, it will have 200 more than the last time". That is why Factorio exists as a standlone game and not as mod in Minecraft (see IndustrialCraft and BuildCraft mods, which inspired Factorio). If that is not a game you want to play, that is perfectly understandable.

I don't believe clients need to have the entire game state for that to happen; just the server needs it. If a client wants to be able to play in single-player, then that specific client should have the option to save the game then, which would then request the entire map data from the server.

So basically we require Factorio to update whole factory all the time. That is not a bug nor oversight, that is intentional behavior. Can we make it so that only server needs to update whole game state? I don't know. Probably yes, but it seems like huge rewrite of ... everything, unless we just send draw commands to clients, which would require massive amount of bandwidth. And frankly, it is quite opposite to what majority of our players want. They would like to run the server on Raspberry Pi or on some crappy machine with Intel Atom.

First, can you point me to the ARM-compatible (eg, Raspberry Pi) Factorio game binaries? I happen to have a few Raspberry Pis laying around and would be interested in trying it out. As for Intel Atom... I don't see how changing how the client processes a multiplayer game would detrimentally affect the hardware requirements for the server. As a matter of fact, I would argue that taking better advantage of threading would benefit the Atom-based server; many Atoms (not all, sure, but many) have more than one core.

Second, being a huge rewrite of... everything? Yes, indeed. I have no doubt. But I strongly believe that Factorio would come out with far superior performance both on server-side and client-side as a result. The better architecture to handle that, in my experience, would also lend itself to facilitate the ability to find and resolve bugs, too, as well as unit testing. I mean, the game does have unit tests, right?

As for multithreaded update. It is very likely it will come in not so distant future.

@OP: Your anecdotal evidence has been completely countered.

viewtopic.php?f=53&t=32649

Over 150 people connected, caught up, and played just fine. Obviously it's not a problem with the software.

As @kogimus already stated, this really doesn't prove anything at all.

• 14.5 vs 14.3 (I want to work with @kogimus tonight and try 14.5)
• vanilla server vs mods (honestly vanilla has a lot of blandness to it, particularly with inserter capabilities and storage sizes)
• fresh server vs one with a medium-ish factory already built
• fresh server vs one with (probably) tens of thousands of aliens spawned
• unknown CPU vs overclocked i7-6950X
• unknown RAM vs overclocked DDR4 3200 14/14/14
• unknown network bandwidth vs 10Mbit upload

More to the point, the mere fact that we did have problems connecting rather proves that it's a problem and not something fake or made up ("anecdotal"). Don't get me wrong, it's pretty cool that vanilla Factorio starting fresh can handle 159 connections. But get all of those people to play off-and-on for a week and let me know how that goes.

Mind you, I don't imagine it won't go well. @kogimus and I have a server running on old hardware and we're able to play on that; we can connect just fine because as we indicated at the beginning of the thread, our client computers have faster hardware than the server. So there's no reason we can't keep up. It's when we move that server instance to my computer with the most recent hardware that suddenly nobody except myself (and apparently a laptop) can keep up.

Thanks for the save.
Not related to underlying problem, but small quality of life improvement: The way you use Warehouse mod for coal murders performance. Chest inventories are not optimized for so many slots and having inserters trying to insert to almost full Warehouse all the time just kills it. On my home Core-i5 3470S it ran ~42 UPS and after removing inserters inbetween coal Warehouses it jumped up to ~78 UPS (I set game.speed = 2 in order to get more then 60 UPS)

Good to know! The warehouse mod wasn't just for coal but it solved a problem with regards to running out of it before we realize the belts are empty

It sounds to me like that info should hopefully make it into a performance optimization task. We're talking "merely", what, 15 or so fast inserters? And, what, 2400 or so inventory slots?

so I will pick out the games I no all Command & conquer work on 50 units and 30 buildings per team, 8 teams 50 + 30 X 8 =640 c&c has 640 items to keep up to date but no more than 1000 items.

Okay, sure, so it doesn't meet up to the scale.

eve online this looks like a big game and it is but 90% of the dater has no connection or no links between them. this thinks can be done in parallel. the biggest think is 1000 vs 1000 ship fights (this has to be done in real time) so one ship has 50 thinks to track 50 X 2000= 100,000 that is a lot and will done to ccp for doing that. and this is done on 1 core.

I'm not sure I follow you. No links between them is probably true to an extent. It absolutely can be done in parallel and I believe their devblogs have pointed at that indeed.

Factorio there about 220 items, use 1-6 Assembling machines for this 4 , that have 3+ Inserter on them that is 2640. add to that 200 Electric Mining Drill, add 20,000 iron ore, copper ore, iron ,copper, steel and 7= items that is 12X20,000=240,000 and a not more. this game has a lot of items to track in real time and on 1 core. the devs has said in the past there is no game/ software out there that dues what Factorio can do or has to do in a game tick. the bigger the map get the more items to keep up to date. this game works with items counts of 500,000 and more all in real time

If you want to start tracking individual items, yeah sure. Maybe. But, like I said, I'm a software engineer. I write software to do things in parallel all the time. Moving stuff from one belt to another? Relatively trivial. Inserters moving? Trivial. Assembler/smelter producing something? Relatively trivial. Most of these you can throw into a SIMD structure (SSE/AVX or GPU) and make multiple calculations simultaneously. I hope Factorio already does that.

But my point about being able to break that down into separate chunks, I think, still stands. Why does one core have to process the entire map? I don't believe it does.

Just to clear up if you misunderstood me when I filed this as not a bug:
The corner case or small window as I described it is not in regards of your map being especially big, old or performance taxing. The special case is only, that the usually inferior Intel i5-6300U processor handles the task better than the FX-8350 (this is most likely because of the shared FPU, as all Bulldozer chips perform poorly in FP operations in my experience) giving the incorrect impression that worse hardware performs better.

The FX-8350 is a Piledriver chip.

As far as Bulldozer is concerned, your statement does echo performance characteristics I've encountered when writing AVX instructions for use on quad-socket AMD Opteron 6272 (which is Bulldozer). We had to disable the "faster" AVX instructions in favor of SSE, when the running CPU was AMD. I believe the reasoning was that AMD actually ended up taking a pair of 64-bit registers and hacking them together to make a 128-bit register, instead of actually creating dedicated 128-bit registers.

[inetknght]

Your counts are a little off. The save file linked a few posts ago has 5,180,800~ entities of which 90,400~ are active (running logic each tick)

I would be interested to see a breakdown of how many of each type of entities there are. Is there a debug options which shows how much time each entity is taking to calculate logic? Eg, in Space Engineers you can press Shift-F11 to see a bunch of summary timing statistics. I'd found that it had been helpful to identify, eg, that ore detectors in Space Engineers eat logic cycles like crazy.

[inetknght]

Lastly, would it be helpful to provide access to a virtual machine on the i7-6950X? It's set up using a barebones Fedora Server 24 with four assigned cores, 8GiB of RAM, and effectively nothing else installed or running. It was intended to be a replacement of the old server's hardware, roughly apples-to-apples, but using current generation hardware instead of something that's a fair bit dated.

If that would be beneficial for, eg, performance testing, it would be trivial to clone it or snapshot it for a developer to play with.

[Sunder1977]

Apparently OP and friends have no idea what anecdotal evidence is. Here, I'll help.
https://en.wikipedia.org/wiki/Anecdotal_evidence

Yes, your "edge case" is the very definition of anecdotal evidence.

[Rseding91]

Your counts are a little off. The save file linked a few posts ago has 5,180,800~ entities of which 90,400~ are active (running logic each tick)

I would be interested to see a breakdown of how many of each type of entities there are. Is there a debug options which shows how much time each entity is taking to calculate logic? Eg, in Space Engineers you can press Shift-F11 to see a bunch of summary timing statistics. I'd found that it had been helpful to identify, eg, that ore detectors in Space Engineers eat logic cycles like crazy.

Per-entity no, but you can use the timings shown in the F4 debug menu to see generally what the time is being spent on (entities, map gen, scripts, rendering and so on).

[inetknght]

Per-entity no, but you can use the timings shown in the F4 debug menu to see generally what the time is being spent on (entities, map gen, scripts, rendering and so on).

Would that be available to a client which is catching up?

[Lee_newsum]

I do not think so.
my F4, F5, F6, F7 can help looking to get more fps

[posila]

First, can you point me to the ARM-compatible (eg, Raspberry Pi) Factorio game binaries?
...
I don't see how changing how the client processes a multiplayer game would detrimentally affect the hardware requirements for the server.

I can't because it doesn't exist. I saw several threads asking if it is possible to setup server on Raspberry Pi, or asking if they can run server on a old computer that have had stashed somewhere for 10 years. It seems to me majority of people who want to run servers, want to be able to use the cheepest HW possible, or to rent some cheap VM and run it there. So it would be best if only powerful gaming rigs that run clients also ran the simulation and server did just some synchronization without running the simulation. This is the first thread (I know of) debating the opposite.

I don't believe clients need to have the entire game state for that to happen; just the server needs it.
...
Second, being a huge rewrite of... everything? Yes, indeed. I have no doubt. But I strongly believe that Factorio would come out with far superior performance both on server-side and client-side as a result. The better architecture to handle that, in my experience, would also lend itself to facilitate the ability to find and resolve bugs, too, as well as unit testing.

Yes, it would be nice feature to have. But it is not going to happen, at least not any time soon. I works well as is, except for the case when server can simulate the game faster than client

I mean, the game does have unit tests, right?

It doesn't have unit tests, it does have tests.

Even though optimizing the game and save size is ongoing process we realize that whatever we do to make the game run faster, players will just build bigger Factories. If not in vanilla, they will use mods to do so.

We had to disable the "faster" AVX instructions in favor of SSE, when the running CPU was AMD. I believe the reasoning was that AMD actually ended up taking a pair of 64-bit registers and hacking them together to make a 128-bit register, instead of actually creating dedicated 128-bit registers.

What do work on? If you don't mind me asking.

[starholme]

Off on a little tangent here, but I want to mention that one problem regarding performance in factorio (and almost all other software) is that performance impacts are not intuitive. Underground belts have better performance? I guess I can see that, less things to render. Bots have better performance than belts(depending on use case)? Wait what? Inserters into large modded inventories have outsized impact? Strange but the explanation makes sense.

It's very difficult to understand what things impact performance as an end user. And most of the time that is ok. But sometimes you just want to build these monsters, and it would be nice to have some 'best practices' to keep performance high.

In a non-factorio example, it's boggling how easy it is to convince Excel to save a 63kb document as a 100,000kb document by fiddling with formatting in otherwise empty cells. And almost impossible to explain this to the end users who call your helpdesk...

[ssilk]

It's very difficult to understand what things impact performance as an end user. And most of the time that is ok. But sometimes you just want to build these monsters, and it would be nice to have some 'best practices' to keep performance high.

That is indeed an interesting aspect. Cause "normal" games never need to look for such problems. But in my last games I rebuilt whole parts of the factory to gain better performance. And lately I made a suggestion around that (viewtopic.php?f=6&t=32513 Timers...)

Factorio is the only game I know, where you have to care about, how you build things, cause in the large scales you can build stuff you come always to some limits. Not saying that is a bad thing! The truth is: In reality we need also always care about such things. Lack of power. Lack of endurance. Lack of reliablity. Lack of cpu-performance (which could happen, in reality, too); that is just ONE aspect of that.

And now that aspect is into a game! Something, the devs always wanted to avoid. But for me as player I must say: That aspect being part of a game makes the game even deeper. I think performance issues belong to Factorio as a part of the game. It would be stupid to ignore that. Developers should of course try to increase game-performance as far as possible (and under normal circumstances you shouldn't think about it), but the problem will keep: Players will find always ways to built bigger bases. Which is good.

So, YES, as an advanced player we need information about performance. Which means: We need to know how much better a design is vs. another. We need something to "profile" game-constructions.

[inetknght]

I can't because it doesn't exist. I saw several threads asking if it is possible to setup server on Raspberry Pi, or asking if they can run server on a old computer that have had stashed somewhere for 10 years. It seems to me majority of people who want to run servers, want to be able to use the cheepest HW possible, or to rent some cheap VM and run it there. So it would be best if only powerful gaming rigs that run clients also ran the simulation and server did just some synchronization without running the simulation. This is the first thread (I know of) debating the opposite.

Interesting. Well, I'm sure you know the Pi 2 and 3 would both benefit from better use of threads. I'm unfortunately unfamiliar with any SIMD that ARM might provide, but I think it would be fun to take advantage of the Pi's GPU. Factorio is written in C++, right? That's my favorite language You might get some benefit from using OpenMP. I've never specifically used it, but I do believe it's relatively plug-and-play. And doing that should also give you advantages on the x86 platform too.

Honestly I don't know any game wherein the server is just a relay. I think an important consideration is... if you're going to create the game logic, then storing the game logic in the server is relatively trivial. And if the client needs to be able to host its own game (eg, not a dedicated server) then putting game hosting logic in the client is relatively trivial. So it would seem to me that it should also be trivial to have an option, "I am just a relay" verses "I am an authority in game logic and may tell you that you are desynced" verses "I am effectively just a thin-client and must either direct- or relay-connect to an authority". Getting that logic in would then make both paradigms relatively easy to configure.

It doesn't have unit tests, it does have tests.

I am curious, what is your distinction?

Even though optimizing the game and save size is ongoing process we realize that whatever we do to make the game run faster, players will just build bigger Factories. If not in vanilla, they will use mods to do so.

There's always a bigger fish...I mean factory. Definitely a bigger fish factory.

Indeed, people building bigger factories will always happen. But I see little reason to artificially limit people to a single core when I think it's easy (albeit time consuming to implement) to make use of the entire computer available to the game.

What do work on? If you don't mind me asking.

I'm assuming you mean me and not all of my friends and I; we work at different companies of course.

I've been writing software for about 18 years, so that's a bit of a loaded question with a lot of depth. But, for the past few years, I have been writing software which analyzes DNA; a focus on high throughput has been important here... there's just so much data, and of course the business wants it all now.

Immediately prior to that, I have worked on file sharing / synchronization software; a focus on solving multi-machine and multi-process synchronization across operating systems and networks was important. It's interesting learning the massive differences between Windows and Linux aren't really all that massive when you boil it all down. Sure, there's a few quirks here and there, a few incompatible concepts, but for the most part those are very few and far between.

Before that, utilities and anti-cheats for games; I think you can imagine what kinds of hacks I have seen. And, uhhhh, created...

It's very difficult to understand what things impact performance as an end user. And most of the time that is ok. But sometimes you just want to build these monsters, and it would be nice to have some 'best practices' to keep performance high.

Yes, exactly! I don't think I'm anyone's typical end-user as I do understand that performance problems are often difficult to diagnose, frequently even more difficult to resolve, and usually very difficult to ensure regression tests don't fail due to jitter from background tasks simultaneously running. It could be a data structure problem (it might take one operation to get the next object, but if it's not cached you're going to end up killing performance), an algorithmic problem (ten thousand operations to get the next object...), both, or even a hardware problem (eg AMD bastardizing their registers).

In a non-factorio example, it's boggling how easy it is to convince Excel to save a 63kb document as a 100,000kb document by fiddling with formatting in otherwise empty cells. And almost impossible to explain this to the end users who call your helpdesk...

That scenario makes me cringe.

[inetknght]

So, YES, as an advanced player we need information about performance. Which means: We need to know how much better a design is vs. another. We need something to "profile" game-constructions.

^ QFT

[quadrox]

It doesn't have unit tests, it does have tests.

I am curious, what is your distinction?

Usually you call something a unit test only if it tests a single function/method - or as wikipedia calls it, tests the application at the level of the smallest possible unit.

If you have tests that verify the interation of several functions and/or classes these tests are often called integration tests or similar.

You may also want to check out the V-Model

[mknejp]

It's funny to see people throw the number of entities at each other as if it was a contest. '"My game has 10k entities!" - "Oh yeah? Mine has 10M entities!"
In the end it rarely even matters. You can have a simulation with two entities whose AI is so complex it's barely interactive. And you can have one with billions of entities making 60 Hz no problem. It really depends on the work that needs to be done, not necessarily how many entities you have to do it for. A typical 3D game has to do a lot more work per entity (more computationally expensive collision detection, pathfinding, physics, animation, object sorting, frame setup, frame commit, ...) than the typical 2D game and hence simply cannot afford to have as many objects fizzing around. And in terms of entity count, go look at a serious scientific simulation to see software that *really* pushes a machine to its limits because every additional ms the simulation step takes translates directly into hours or days you have to wait for the final results.

Today's CPUs can crunch through ridiculous amounts of data but in order to make that possible the data structures must be arranged in an appropriate manner. It seems counter intuitive to many people (especially the OOP crowd) that data-logic separation and scatter-gather reorganization, which look like more work for the CPU, but can in reality significantly improve performance by making the result of every memory round trip contain 100% information density. Constant factors matter. You have to keep the prefetcher busy by having predictable memory access patterns. There are good reasons why entity-component-systems are so wildly used these days. They have very high data and instruction locality, and are trivial to parallelize, vectorize, offload to GPUs, and test. Unfortunately Factorio went with the classic OOP-based approach with virtual update functions and considering how close the potential 1.0 release might be it's probably too late to change any of that. Virtual functions are an absolute performance killer, the most common kind of icache thrashers and the random allocation patterns of entities are a guarantee for constant L2 cache misses, the biggest performance offender for modern CPUs.

Please try not to interpret this as implying what the devs achieved here wasn't impressive. It totally is, but it does sometimes show gaps in experience/knowledge (that all of us have in some form) like the use of std::map (fortunately averted), or the use of uint8_t as induction variable, etc. Sometimes it's the small things that accumulate.

And I can only imagine Rseding bashing his head against the desk seeing one of my performance rants again

[inetknght]

@mknejp, has it exactly right. I've had to work pretty damn hard to squeeze as much performance out of everything as I can in my own software. Removing inheritance trees goes a long way to helping with instruction cache. There's other things to consider, too, such as the branch predictor (which can have massive performance offsets), as well as data cache.

This stack overflow answer was a pretty massive eye opener to me, as far as how much effect the branch predictor can have.

And the kicker: both solutions provide correct results, but one is simply faster. That's really the takeaway from it.

[MrGrim]

@mknejp, has it exactly right. I've had to work pretty damn hard to squeeze as much performance out of everything as I can in my own software. Removing inheritance trees goes a long way to helping with instruction cache. There's other things to consider, too, such as the branch predictor (which can have massive performance offsets), as well as data cache.

This stack overflow answer was a pretty massive eye opener to me, as far as how much effect the branch predictor can have.

And the kicker: both solutions provide correct results, but one is simply faster. That's really the takeaway from it.

How the different compilers handled the test code in that stackoverflow.com answer was interesting.

I deal with so many "developers" on a daily basis that remain willfully ignorant of how the hardware they're developing on actually works (they think modern abstraction absolves them of the need) that it brings a smile to my face every time I see discussions by developers that have a clue.