Generic entity

[3maven]

I'd really appreciate an entity type with no predefined behaviors.

Right now, to create a new entity I must first look around to try to find an official entity that's somewhat close, then work around the differences. This sometimes leads to GUI elements from the behavior that don't make sense. (E.g., "use signal colors" toggle on an entity based on lamp.)

Would you be willing to supply a dummy entity type like that, please? I think it would let modders produce higher-quality mods with less work.

In addition to what's already in LuaEntity and LuaControlBehavior, I'd love to have the ability to choose whether entities of that type support logistic network and/or circuit network, number of module/item/signal/equipment slots, and some way to specify a custom GUI for the circuit network pane.

[aubergine18]

Have you tried 'simple-entity' ? It's one of the most basic going.

If you need circuit connections, something like a gate entity might suffice. Still very basic but has a good number of useful features.

[mknejp]

simple-entity is fine if you don't need the thing to have a force or be compatible with blueprints/ghost placing. I think a generic-entity that is essentially equivalent to simple-entity in terms of game logic except that it has an associated force and can be ghosted/blueprinted would be very useful. I am currently using walls to simulate that but it just feels wrong to specify the same sprite eight times. And who knows if it being a wall has any unforeseen consequences.

edit: considering circuit connections you have to know how the circuit connection is supposed to behave which is going to determine what type of entity it has to be.

[Rseding91]

I'd love to have the ability to choose whether entities of that type support logistic network and/or circuit network, number of module/item/signal/equipment slots, and some way to specify a custom GUI for the circuit network pane.

You're not a programmer are you? There's a plethora of backing logic and itegration to add any one of those things to a given entity. Loading the definition from the prototype, creating it on load if it didn't exist before, GUI additions, transfer logic, saving/loading, forces logic and so on.

To put it simply: that's never going to happen.

[mknejp]

I'd love to have the ability to choose whether entities of that type support logistic network and/or circuit network, number of module/item/signal/equipment slots, and some way to specify a custom GUI for the circuit network pane.

You're not a programmer are you? There's a plethora of backing logic and itegration to add any one of those things to a given entity. Loading the definition from the prototype, creating it on load if it didn't exist before, GUI additions, transfer logic, saving/loading, forces logic and so on.

To put it simply: that's never going to happen.

In object-oriented systems where every entity has a fixed type and hardcoded properties/logic this is certainly true. Entity component systems circumvent this by having entities only be defined by the components they are composed of. So it is trivial to specify "this thing here has an inventory, a logistic interface and a power generator" and then let the appropriate systems deal with it. This intrinsic ability to easily customize entities leads to very short iteration times at basically zero performance overhead.

[Rseding91]

In object-oriented systems where every entity has a fixed type and hardcoded properties/logic this is certainly true. Entity component systems circumvent this by having entities only be defined by the components they are composed of. So it is trivial to specify "this thing here has an inventory, a logistic interface and a power generator" and then let the appropriate systems deal with it. This intrinsic ability to easily customize entities leads to very short iteration times at basically zero performance overhead.

Yet no large scale performant game exists that uses such a system. That says something.

[mknejp]

In object-oriented systems where every entity has a fixed type and hardcoded properties/logic this is certainly true. Entity component systems circumvent this by having entities only be defined by the components they are composed of. So it is trivial to specify "this thing here has an inventory, a logistic interface and a power generator" and then let the appropriate systems deal with it. This intrinsic ability to easily customize entities leads to very short iteration times at basically zero performance overhead.

Yet no large scale performant game exists that uses such a system. That says something.

Wow. I honestly don't know what to say. I certainly think the folks at Frostbite, Crytek or Ubisoft would strongly disagree. Most information obviously (unfortunately) isn't available publicly but just go to GDC (EU) and talk to the engine and game engineers of small and large studios alike. The notion that a very large number of modern games would be impossible to run on contemporary hardware (including consoles) without a data-oriented ECS driving them is pretty much accepted. And there's also the HPC crowd who put the scale of any game into a healthy perspective. Dismissing architectures specifically designed to exploit hardware to its fullest potential as silly ideas or toy examples is simply delusional. It is a tested and repeatedly proven framework for performance critical applications on architectures from mobile phones all the way up to server farms.

[Rseding91]

No one has made a game to the scale Factorio has with any of those systems you've listed. Frostbite (battlefield primarily) has 64 players and limited size maps. Crytek has some fun physics but chugs if you get any medium amount of things going on (blow up 1 building and it slows). Ubisoft - same thing. They're are impressive game engines for what they're meant to do.

[mknejp]

And Factorio of course has 3D collision detection, physics, animation, deformation, AI, and many more things that are computationally intensive, like every good 2D game does. Have you counted how many peasants walk around in the latest Assassin's Creed games? Like the game or not but one has to acknowledge that there are quite a lot of them and simulating/rendering as many as they do requires more work than running an assembling machine. You cannot compare an entity (however that is defined) in one game and an entity in another and conclude that because one game has less of them its engineering must be inferior. There are numerous amounts of components/entities in these games that you cannot see but that doesn't mean they don't count and aren't doing work. But tell me then, if data-oriented ECS are not the right tool to do data-heavy and computationally intensive simulations (games are just simulations after all), then why don't those applications go the "inheritance hierarchy with virtual methods"-approach? Ignoring how the CPU works and not organizing your data structures to feed it as efficiently as theoretically possible is not engineering. You keep repeating the word scale but it really has no meaning. That is unless you consider "number of entities on the screen" the only relevant metric.

[Mooncat]

While I agree that using game scale to be a reason to reject such (fairly common) suggestion isn't helpful. At the end, it is just about performance and optimization. I don't think component system will be a big trouble to the Factorio devs. It is just a big paper and you should be able to eat it.

On the other hand, I understand that we won't have such system in the near future. Let's not say it will never happen. Maybe someday an insanely talented guy will join your team and redesign the whole system in just a few days.
The whole "entity type system vs component system" problem is just like the problem of vertical train stop before 0.13. The engine has already been designed like this. It will take much time to change it fundamentally. Not many people would be patient enough to wait for months or even years for such a change that only benefits modders.

Even just an additional "generic entity" wouldn't be an easy thing either. How generic it should be? What components should the devs provide? We modders are hungry. We will demand many things that the game wasn't designed for. It is hard to satisfy us especially if such entity isn't tested in the actual game. We will expect it to be the "god entity", but then disappointed by its limitations and eventually make more requests or even assume the limitations are bugs... not to mention the components will be a new concept to the engine. It will take time to implement and test and debug.

So, do I think it is a good suggestion? Yes. But what do you feel when the dev rejected it? I understand that it will not be implemented in near future. Maybe in Factorio 2?
(Do you think the reason to reject is good? We can find a better one. )

[mknejp]

It isn't really about the modding interface. Whether that would change after such a refactoring is an entirely different issue since it would likely break every mod in existence.

No I'm wondering about the game itself. I find the notion of dismissing a design that has repeatedly proven itself to be the only viable contender for performance critical applications as some kind of joke or toy example highly irritating. It is specifically designed around the characteristics of modern CPU/GPU architectures to achieve the highest possible throughput.

We hear the devs mentioning optimization and multithreading a lot lately and if they are serious about this topic they have to stop pretending that virtual functions are a solution, that randomly allocated entities are efficient or that their threading model is remotely capable of fully saturating all CPU cores. We know virtual functions run on the average of 400-600 cycle latency for cold unsorted icache accesses. We know dcache misses from randomly placed objects incurr up to 300 cycle latencies. We know dependent reads suppress the prefetcher until the next cache line is available. We know branch misprediction is a big perf killer. We know job systems are better at utilizing all available CPU cores compared to threads running dedicated subsystems. These are all facts. Programmers have accepted that we can't hand-waive these inconvenient truths away and have to work around these constraints.

While their attempt at splitting up chunks into "parallelizable" packages for inserters may sound noble, it is ultimately a naive approach that doesn't scale well. So if the devs are taking the topic of optimization really seriously they have to look into these architectural issues. However if they are only aiming for "good enough", well then that's another topic. But then I find it rude when servers kick people out of multiplayer games because they cant "catch up" when their CPUs or GPUs aren't burning at 100% utilization. As long as there is underutilized potential on the machine there is no justification for not being able to "catch up" or dipping below 60fps and the application is clearly not making use of all the available computing power.

I don't understand why one wouldn't strive for making their application run as fast as possible, especially when performance is basically part of the functional requirement, but maybe I'm just weird like that
I'm evidently also not very skilled at conveying these paradigms so let me just leave this quote here from this article that does a better job at explaining things

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

So, back to the original topic: to me the generic-entity is a tiny little evolution from simple-entity. simple-entity is almost perfect, but the fact it doesn't have a force means it cannot be ghost-placed or blueprinted and enemies don't attack it. I feel like an entity with zero logic of its own but these additional properties on top of simple-entity would make many mod efforts a lot easier because often they only need "something" to represent their entity in the world but it has to "behave" like a normal entity that is attacked by enemies and plays nice with blueprints. Right now this means abusing some other prototype that does too much or has undesired interactions with other entities.

[Klonan]

While their attempt at splitting up chunks into "parallelizable" packages for inserters may sound noble, it is ultimately a naive approach that doesn't scale well. So if the devs are taking the topic of optimization really seriously they have to look into these architectural issues. However if they are only aiming for "good enough", well then that's another topic. But then I find it rude when servers kick people out of multiplayer games because they cant "catch up" when their CPUs or GPUs aren't burning at 100% utilization. As long as there is underutilized potential on the machine there is no justification for not being able to "catch up" or dipping below 60fps and the application is clearly not making use of all the available computing power.

I don't understand why one wouldn't strive for making their application run as fast as possible, especially when performance is basically part of the functional requirement, but maybe I'm just weird like that
I'm evidently also not very skilled at conveying these paradigms so let me just leave this quote here from this article that does a better job at explaining things

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

So you're saying we should utilize all the power of a modern multi-core machine while also saying it is not a good idea to try to implement multithreading to the game?

The game utilizes as much of the CPU as it can, but the main game update is a single threaded process. Once it is using 100% of a single core, there is nothing more to do.

We know threading is a tough topic, we are working on many many single threaded improvements all the time, such as transport belt performance, but saying threading is not a scalable solution it somehow wrong.

We can optimise a lot, we can squeeze as much out of a single core and streamline the single threaded update as much as possible, but people will still hit the limit. When that point is reached, the only viable solution is using multiple cores for the main update.

Modding a component based system would be a muderous task that would make optimization incredible hard. One game i know which uses such a system is Kerbal space program, and the performance between the two is hardly comparable.

Asking for specific entities via modding requests works well, if there is a need fro something, we are happy to look into it. But rewriting all of the prototype code to support modular entities isn't the right path.

[mknejp]

So you're saying we should utilize all the power of a modern multi-core machine while also saying it is not a good idea to try to implement multithreading to the game?

The game utilizes as much of the CPU as it can, but the main game update is a single threaded process. Once it is using 100% of a single core, there is nothing more to do.

We know threading is a tough topic, we are working on many many single threaded improvements all the time, such as transport belt performance, but saying threading is not a scalable solution it somehow wrong.

We can optimise a lot, we can squeeze as much out of a single core and streamline the single threaded update as much as possible, but people will still hit the limit. When that point is reached, the only viable solution is using multiple cores for the main update.

Modding a component based system would be a muderous task that would make optimization incredible hard. One game i know which uses such a system is Kerbal space program, and the performance between the two is hardly comparable.

Asking for specific entities via modding requests works well, if there is a need fro something, we are happy to look into it. But rewriting all of the prototype code to support modular entities isn't the right path.

Hm I think this came across the wrong way. Adding multithreading in and of itself is better than not doing it, but as far as I understood the FFF it is not an attempt to parallelize the game update as a whole. I am not referring to parallelization itself not being scalable but the specific way of parallelization. Multithreading only a specific part of the update means the rest doesn't benefit from it and even if the inserters now finish faster, the freed up cores still can't do anything else than to idle around until the frame finishes. If any of the subsystems that run on dedicated threads finish earlier their cores can't participate in the rest of the frame processing. Running the game on the next generation of CPUs with who-knows-how-many cores won't speed the game up unless in those very specific corner cases. That's what I mean with scalability: in an ideal world if you have twice the cores it should finish twice as fast (which of course is impossible due to Amdahl's law, reaching the memory transfer limit, etc but I hope you get the idea. Sorry if I didn't make that clear before. As you say yourself, once you hit 100% on a single core that's it. The point is to spread the game update across all cores and run the frame in phases. This is obviously a lot more difficult in an inheritance based system where there are arbitrary side effects happening at unknown points in time. A big selling point of data-oriented(!) component systems is that they are trivial to parallelize, vectorize, and even put on the GPU if needed. Parts of the same component, data-independent components, can be run in parallel without data conflicts. It basically combines the best of both worlds: the highest possible single-core throughput coupled with parallelizability (is that even a word?) for almost no effort compared to the alternatives.

KSP's primary performance barrier is Unity's physics simulation and that is close to impossible to parallelize due to the constraints all being part of the same equation. The best they can do on that front (afaik) is to simulate vehicles that are sufficiently far apart in parallel hoping they never move fast enough that they might intersect between frames. (KSP is also not written in C++ )

Modding is doable as well. The recipe for that is to first run all hard-coded systems and internal updates and defer all events for the scripting system until the game is done with its simulation step. Only then does the game descend into madness and call external code, obviously single threaded. But at least you know at that point that all simulation is done and that mods have only a very limited set of mutating actions and side effects they can perform, and they can only affect the next frame. In terms of data-access and cache locality that approach at least is not worse than what is happening now.

I could talk about these things all day long but as already mentioned I am not discussing this because of the modding interface but because I'm very curious about the game itself so I don't know if this is the right place/thread for that. If you're interested in this and don't feel like I'm the most annoying person ever (I know that happens) let me know as I am more than happy to discuss this in all its dirty details.