Optimization idea: abstraction

[Chromatix]

TL;DR

Solve performance issues of large factories by creating a new feature that will allow user to "compile" part of their factory into single entity.

Why ?

Let's start with a justification and then continue to actual proposal and technical details.

Issue of UPS
If I were ask to point out a single issue that the most of discussions is about (implicitly or explicitly), it would be the performance (UPS) of large factories. It may not be that obvious, since the community spans whole spectrum of players, but generally speaking large amount of discussions is about "mega bases". And with a megabase, sooner or later the player will hit the "issue" of UPS. I pesonally think that Factorio has amazing performance for vanilla experience, and you can only hit the UPS drop zone when you go for it, playing late endgame. Nevertheless, it is an engine limit and the player needs to either play around it (using optimized design patterns), or feel the consequences. Many of the discussions are therefore about megabase design, which evolved over the years, but is inherently tied to engine UPS limitations.

I am not big fan of an opinion that player needs to play the game paying attention to engine limitations. I think that using solar power instead of other types, or using trains instead of insanely long and resource-inefficient belts, or using inventory-to-inventory item movement, just because it is nicer to the engine, isn't something that player should be forced to do.

Propositions I've seen so far aimed towards player's behavior optimization (use engine-optimized patterns, don't play beyond first rocket launch), general single-thread optimization, hardware improvement suggestions, or unrealistic and simplified parallelism suggestions. Although all these can still be applied, I believe there is another option, that is moreover significantly more scalable.

Before we dig into the details, let's elaborate why players, Wube owners and employees should care.

• Players:
  • The feature could be natural extension of blueprints and allows players to move from micromanagement to macromanagement without necessity to care about UPS or even some tiny factory bugs.
  • It would allow to work with parts of the factory as indivisible sectors serving particular role, which is innevitable gameplay pattern of the endgame anyway. Players would get a dedicated feature for it beyond blueprints.

• Wube owners:
  • I have no idea how hard would be to implement such thing in Factorio, but I (a player) would kindly pay 20 Euro on top of the base game for such content - because it would change the endgame. If there was a real endgame goal (launching rocket is too easy), it would be valuable even more. This post of mine is not an evaluation of business opportunity, I have no interest in it beyond an interest of an avarage fan. That is also the reason why I spent only limitted time on development of my suggestion.

• Wube developers:
  • This forums section is meant to be for suggestions that cannot be created by a modder. I believe this is the case - that such feature has to be part of the base game, similar to blueprints. In this context I suggest an interesting new feature to work on.
  • Technical part of the feature would require to solve really hard, but interesting problems. I would personally find very entertaining working on those. I don't know what all of your competitors in genre of factory simulations are doing, but I find Factorio on top of them in terms features related to factory construction and operation. Solving the proposed feature would move it up another level.
  • If there is performance ticket somewhere in your backlog, think of this as a smart workaround. It will not solve a game core performance issue, but it will allow players to handle it. Of course it is not a solution solving all performance problems forever, but just another way to approach the problem. I think of it as IPv4 to IPv6 transition.

What ?

Let's summarize the problem:

• Game engine requires update the game state in real time.

• The game state complexity grows with amount of objects in the map interacting with each other.

• Large factories can hit the state where amount of updates per second is smaller that frames per second - factory time flow speed becomes slower than real time flow speed, which reduces the amount of items the factory can produce per single real-life second. Simply speaking, factory simulation speed is getting slower.

• Factory "size" is measured by amount of items it can produce - its output.

• General way how to reduce number of object interactions while keeping factory output constant is to increase speed of item crafting and increase bulk size of items moving. This also means that increasing factory output can be achieved by an object that is "printing" items very fast. But this goes against core gameplay idea - crafting items take some time and there has to be complex chain of components. Cannot craft rockets from iron ore.

• Having complex factory section where hundreds of objects interact is another core gameplay feature. It cannot be simplified by having a single huge "box" that prints items at rate of the original factory section. It would "not look good and realistic".

• Factory view and size must be "realistic and appealing". It should preserve the original complexity and visuals.

• Joy of building of a factory is another core feature, but simplification by using the blueprints is ok. It is not considered "cheating".

From the above, the problem statement can be summarized as:
Reduce amount of object interactions while keeping constant factory output, but avoid "black boxes" that are both visually unpleasant and unrealistic. Abstraction of individual items into blueprints that allow printing whole factory sections is ok.

Finally, elaborate on the statement from TL;DR.

The idea is to create a feature in Factorio, that will allow user to turn selected part of their factory to single entity. An "item", which will have the same visual appeal, same input-output properties (transformation capability, speed, resource and electricity consumption), but will abstract from internal interactions. It will simply transform input items to output items, but doing that will visually, sprite-wise, look the same. The difference would be that it will be a single entity - an object with an input and output slots, instead of set of belts, inserters, furnaces, assembling machines etc.

I know it may look too ambitious, and maybe it is. But I believe that technical solution consists of few well-defined tasks that need to be solved.

Me, a complete ignorant of Factorio engine, can identify at least these:

• Solve item transformations and consumption.
  • Calculate a directed acyclic graph (DAG) from object interfaces between input and output - what is being consumed and produced, what items participate on it?
  • Define how the user will select where is the start and the end of the section. Can there be an input inventory that will accept only predefined item?
  • Calculate overall crafting speed - resource consumption, production; power consumption.
  • Detect that items must not loop around; has it to be a DAG? Detect when a user creation cannot be transformed to a DAG.

• Solve graphics
  • Graphical appearance should remain the same; how to merge sprites?
  • How to simulate items flowing "in" the object. How unrealistic would be to not show them?

• Other tasks
  • How to specify borders of a section that is meant to be transformed to single entity?
  • How to work with it once it is a single object? Can it be destroyed or repaired? Partially, totally? Once "compiled", cannot "uncompile"?
  • How will it work in multiplayer?
  • How to release?
  • Free or payed feature?

There are clearly lots of tasks to solve. For me, the most appealing (and probably the hardest) part is solving the first one - how to calculate an end-to-end item flow from arbitrary player's creation? It seems hard, but it reminds me problems from data engineering area. Actually, real-life ETL data pipeline in a corporation is very similar to a Factorio factory in sense that inputs are transformed to output through series of interfaces and transformers.Types of items in a factory are equally important as data types, and individual blocks are connected together to create a single well-oiled machine with a purpose.

Transforming the problem to an oriented graph is standard excercise, the only problem is how to do it in that particular example and what to do with it once there. Anyway, I find it very interesting problem and gave it a try on a trivial case of small section of steel smeltery. See below.

Suppose a user wants to transform their little steel smelting block into a single object:

factorio_furnace.png (3.32 MiB) Viewed 4140 times

What is the basic item flow? Clearly there are 6 parallel smelteries that grab items from common belt getting items from a input chest an putting finished items to an output chest. Also, excuse my right-left direction please, wanted to keep visual connection to original smeltery screenshot.

first_diagram.png (18.07 KiB) Viewed 4140 times

But it is not that simple, there are individual inserters and the each belt consists of two individual bets (two sides).

second_diagram.png (40.01 KiB) Viewed 4140 times

Ok, but that diagram shows only schematic connections. It does not show the particular transformation of items, nor the speed of the transformation. Also single, unbranched item trajectory may not be visible.

Let's extract only what happens in one furnace. Some interface speeds are simplified, namely inserter has common speed 60 items per minute.

third_diagram.png (49.88 KiB) Viewed 4140 times

From the picture it is clear what is the input and what is the output of each operation. Some in-game objects has to be decomposed to multiple nodes, because they actually consists of multiple interfaces. Interface speeds are denoted, and there are speeds that limit the whole transformation marked in red.

Let's show the overall composed view when all 6 furnaces work in parallel.

fourth_diagram.png (110 KiB) Viewed 4140 times

Ok, the last diagram may look little crazy, but it shows the problem. When the goal is to calculate input and output speeds, it needs to take into consideration all interfaces and their throughputs. In case of paralellism, the throughput is summed over the parallel segments. But there may be a bottleneck anywhere in the system (anyone who plays Factorio knows very well). So the goal is to find that bottleneck, because it defines the IO speed of whole system. Which in case of single-object system is its "crafting speed".

I know my explanation may be in few places little hasty, but this is not a research paper and I don't want to spend excesive amount of energy on it, when the most probable reaction on this wall of text will be something like: "it's nonsense", or "nice, but won't do".

So, have a nice day.

[Koub]

[Koub] Merged into older thread with same suggestion.

[ptx0]

you can already do this, just place an infinity chest in the editor.

[Brambor]

I was reading a rant about UPS here and I thought about an idea I have had for some time.

Idea

The machines in factorio do repetitive work. An intelligent entity could optimize performance by finding cycles that repeat. I think this is a very hard problem.

However we, as players, could select a part of the factory and tell the game "this repeats" <==> "take this as one entity". As simple as that.

Take a smelting column for example. Select it and suddenly, the inputs are simulated, the outputs are simulated, but everything in between is "in a cycle", not simulated per entity, but as a sum of ins & outs. More of the technical details of how to achieve this later. From the outside perspective, it behaves EXACTLY the same. Or if my vision is too greedy, we will allow for some leeway. But no! Exactly the same!

The big idea is the cycle repeats. It can be simulated once. Then only all the input and outputs as a sum will be simulated. that's two belts with generated and consumed items instead of a smelting stack.

As long as the belts in are entirely full, the electricity is at 100%, no biters, or any other entity enter, there are no roboports, no way of interaction with the outside except "belt in belt out", or rather all of these in and out:

• items on belts (in & out)
• fluid in pipes (in & out) - is it tricky? No! I hope...
• electricity (behaves as a single entity)
• pollution (consumed by trees, ground, water, produced by machines)

Ok, pollution might be simulated separately still, I don't see a clear cycle there.

Simple example

Simple case: belt -> inserter -> smelter -> inserter -> belt. single stack
Just one smelter.
The input belt is full, the output is how much one smelter can make.
It is all packaged into a single entity. Looks the same, it just is simulated as a single entity.

Synchronization / finding a cycle by hand

The coolest bit, in my mind. The core of the idea.
You know how the smelting column should synchronise. Give the player tools to do the following:
The Player has built a smelting column already.

1. Player can tell every entity to finish its job (animation) and then stop and wait.
2. Player can tell every entity at once to start playing its animation.
3. Player can tell when all entities should stop and wait for all entities to finish a cycle. (TODO This is a bit lose).

Finding a cycle automagicaly

This is probably not going to work because of tiny differences, therefore the tools to stop animation and wait.

Anyway, the idea is simple:
Select entities, you can see should be in a cycle, like a smelting column.
The game will run until it finds exact match to any frame it already simulated (hashing and stuff).
Then it presents the cycle, the length, when it began, ended, all the stuff

Let's define a cycle

A cycle has a length measured in seconds (ticks, actually).
Also a time it started.
Example: Cycle has a length of 2.4 seconds and it started 10h 25m 24sec, tick 48, into the game.

Cycle has inputs and outputs throughout the cycle.
Notably: single stack (see "simple example") has electricity consumption graph. It is not constant. The inserters swing to life for a short time to consume 14kW and the rest of the time they are waiting, consuming 400W.

Break the cycle

Every cycle has cycle length C. if we break a cycle at time T, we get to the internal state at T mod C.
Example: cycle length is 4.8 seconds, We break the cycle (biter comes in?) at 49 seconds. Therefore, we completed 10 cycles, each long 4.8 seconds and we are 1 second into the cycle.
At this point, the game behaves as if all the entities are separate again and it knows the exact states of all the entities since it is 1 second of simulation from the initial state. Which it simulates I guess.

Observe the cycle

The idea is you have hundreds of these on your map. Whichever you do observe has to be simulated, so no gain there. All these that are not observed can be abstracted into cycles. Not sure yet about the minimap.
Also: if this does not work, we could draw a black rectangle over cycled entities. This breaks the immersion immensely, but it is the first step in implementation. To see a performance gain in megabase where you do not move and look around.

UI and stuff

I have some cool things in my head, simple rectangles on the map, all the cycles listed somewhere, and automatic restarting of cycles after breaking. Cycles within cycles.

Cycle merging, where both have different lengths, forms a single bigger entity. Still worth it, as internal implementation will not make 2.4 sec cycle and a 1 sec cycle into 24 sec cycle, (it would be on the outside), but it would break in stages: 24 cycle breaks at 5.5secs, so 1 sec cycle breaks at 0.5 secs and 2.4s sec cycle break at 0.7 sec. Something like that. No matter the details now.

However, it would be a waste of time to discuss it now.

Feedback?

Devs? I can imagine all of you discussed it already. Is it a good idea? Where does it break?
(If you read it eventually.)

Anyone? Can you link me to similar attempts?

I don't think it is a vanilla idea, it is specific for some sort of megabase optimization. An amazing and ambitious idea for a mod. Maybe a bachelor's thesis? That would be so cool! I study Theoretical Informatics at CTU, FIT.

Some Notes

All entities will have to be thought out.

For example miners: I am not sure how to cycle them beyond the simple example of "one patch of ore, 1 to 4 miners", where you define how many times the cycle can run by how much ore is in the patch. If there is 400 ore, every miner mines one in the cycle then the cycle will run 100 times.
Of course, it would be a total waste to even define such a cycle.
Anything beyond that, I don't want to break the ore consumption randomness by assuming/calculating by the wrong statistical model.

[Nidan]

• fluid in pipes (in & out) - is it tricky? No! I hope...
• electricity (behaves as a single entity)

These are floating point math, which has limited precision as opposed to "real" math. Any rounding error introduced due to that might change something that looks like a repeating cycle into something diverging. (Technically it must still be a repeating cycle due to the fixed amount of memory used for representing the state, but it probably wont be a short cycle)
Even the simplest rules like (a + b) + c = a + (b + c) do not hold for floating point math. Thus, summing the electricity consumption of all machines and subtracting that from the larger network will likely have a different result than accounting for each machine individually.


Recommended reading / merge candidate: /viewtopic.php?f=6&t=82060

[mmmPI]

Anyone? Can you link me to similar attempts?

I think this is doing things similar to what you describe, the way the cycle is defined differs but i saw similarity in the grouping of machine in a smaller number of entity based on cyclic behavior.

https://mods.factorio.com/mod/factory_modules

Those i feel have less similarities but are based on the same idea of grouping entities to reduce the need for computation :

https://mods.factorio.com/mod/modular-buildings-redux

https://mods.factorio.com/mod/Manifolds

[mrvn]

This won't work well if belts that aren't fully compressed or with fluids (because they have waves).

There was a mod though that would allow you to define a sub-factory and then create copies of it. The original sub-factory is simulated normal and all the copies just copy then input and output.

So you have to compute everything for one (one per ore) smelter column and all the other columns just mirror the input and output belts.

[Koub]

[Koub] Merged with older series of threads with similar or same suggestion.

[ssilk]

Even the simplest rules like (a + b) + c = a + (b + c) do not hold for floating point math. Thus, summing the electricity consumption of all machines and subtracting that from the larger network will likely have a different result than accounting for each machine individually.

True for unlimited precision, but not so true, if you use for example fixed-point floating numbers, which are in fact just integers shifted right. For those the above rule is still correct.

See for example https://www.factorio.com/blog/post/fff-64

where they used it for other reasons.

Attention: Not saying the solar panels are calculated in fixed-point arithmetic!



The Dupebox mod is in my eyes stiil the most promising try resulting from this thread.

Btw: links to mentioned mods in this thread: search.php?keywords=Mods.factorio&t=82060&sf=msgonly

[Nidan]

Even the simplest rules like (a + b) + c = a + (b + c) do not hold for floating point math. Thus, summing the electricity consumption of all machines and subtracting that from the larger network will likely have a different result than accounting for each machine individually.

True for unlimited precision, but not so true, if you use for example fixed-point floating numbers, which are in fact just integers shifted right. For those the above rule is still correct.

They exist, but they aren't used in the situations I quoted, so I didn't mention them. It's "fixed point numbers" btw, the point can't be fixed and be floating at the same time.
But since we're already talking about them: I wish factorio would use them in some (more?) places instead of floating points, e g. a machine's progress bars. Seeing them with empty progress and full productivity bar and requiring an extra tick of work to create the extra result sometimes just feels odd.

[Brambor]

I caught up with this thread, I see no one suggesting precisely what I did suggest.

Implementation clarifications

Note: I am talking about a furnace column, but the cycle wouldn't be 4.8 seconds, since they consume coal or other fuel, so it could be the length of burning the fuel (if multiple of 4.8 seconds). Easy fix: simple stack is from electric furnaces instead of smelters.

• fluid in pipes (in & out) - is it tricky? No! I hope...
• electricity (behaves as a single entity)

These are floating point math, which has limited precision as opposed to "real" math. Any rounding error introduced due to that might change something that looks like a repeating cycle into something diverging. (Technically it must still be a repeating cycle due to the fixed amount of memory used for representing the state, but it probably wont be a short cycle)
Even the simplest rules like (a + b) + c = a + (b + c) do not hold for floating point math. Thus, summing the electricity consumption of all machines and subtracting that from the larger network will likely have a different result than accounting for each machine individually.


Recommended reading / merge candidate: /viewtopic.php?f=6&t=82060

True, I am willing to sacrifice that. It shall be a mod that changes things and the cases where floats break (a + b) + c == a + (b + c) don't really matter in the vanilla anyway, the player will never notice them.

Questions

I wonder what the performance would be like. (By the way, I never moded any game.)
Q1: Is belt -> entity (my cool entity made from entities) -> belt fast? I read there something about "spawning resources with Lua" being slow. I expect that this doesn't fall under that? I think loaders/unloaders will be related to the answer...
Q2: Can I make an entity at runtime?

CPU spike when electricity < 100 %

I also read about CPU spike, so UPS drop once the electricity drops. I see that as a shortfall of this mod, but not as a roadblock. If you happen to use this mod and suddenly your UPS is 6, I see those workarounds/solutions:
1. You can reload from 15 mins ago and prevent the electricity issue before it arises.
2. The mod can provide additional tools to turn off things so they are off completely and don't consume CPU time. This solution might not work, I read in this thread about the "active state off" still consuming CPU etc. I wouldn't want to implement it anyway, just if the mod is a big hit and everyone needs to solve "the UPS drop issue". Which, I declare: "Will not happen".

Graphic?

I don't think anymore that there should be graphics involved. I think the entity should fall apart on observation. This is a pure performance opinion. If I can record the entity (cycle length, ... we have all the information to compose the animation), and it will turn out faster, then sure! But I don't think that enormous sprites of entire columns could be efficient. It is true, that if you take N columns that are the same, the entity produced will be the same and the sprite will be the same (of course, only one entity and sprite for all N columns). Somebody would have to tell me what is faster or I should make a test, I wonder how hard it would be...

Endnote: I don't plan to make this mod now, finishing school, finishing work projects and having fun/relaxing take precedence right now.

[mrvn]

You can run one instance of the abstraction for real and place views of that instance wherever there is a copy. For an example how to mod this look at how factorissimo can show you the inside of the factory building.

Using factorissimo as base might also be an option. You build your factory inside the building and then create copies of the building that will have the same input/output without actually running the inside. You just run one instance and mirror the input/output to all copies.

As for the UPS concerns: Removing the inputs and creating the outputs in LUA is costly. So you better make the inside of your factory big. A simple yellow belt column of furnaces might be faster than faking one with LUA. But that just means you need to go bigger and put a blue belt comlumn inside or something.

[FuryoftheStars]

You build your factory inside the building and then create copies of the building that will have the same input/output without actually running the inside. You just run one instance and mirror the input/output to all copies.

Isn't there a mod that does something like this already?

[Brambor]

You can run one instance of the abstraction for real and place views of that instance wherever there is a copy. For an example how to mod this look at how factorissimo can show you the inside of the factory building.

Using factorissimo as base might also be an option. You build your factory inside the building and then create copies of the building that will have the same input/output without actually running the inside. You just run one instance and mirror the input/output to all copies.

As for the UPS concerns: Removing the inputs and creating the outputs in LUA is costly. So you better make the inside of your factory big. A simple yellow belt column of furnaces might be faster than faking one with LUA. But that just means you need to go bigger and put a blue belt comlumn inside or something.

That has been already discussed here already thoroughly and yes, there has been even a mod, see:

Btw: links to mentioned mods in this thread: search.php?keywords=Mods.factorio&t=82060&sf=msgonly

My implementation should be exact, except for floating-point shenanigans. But no difference in item count, possibly small epsilon in electricity. I wouldn't implement fluids thought, it would change flow behaviour significantly.

[ptx0]

just use Schall Machine Scaling, or Clusterio.

[blazespinnaker]

So, I've talked to some different folks about this. At the end of the day, it's a mod. A great mod, but will always be a mod. Factorio is a game, not a science, and while abstraction for some of us sounds like a huge amount of fun, for the vast majority it's more chore than game playing.

Abstraction done right though would be fantastic. Coupled with mods which visually represent factories as singular warehouses, it really adds a lot. This latter concept I think is pretty important, as abstraction isn't just about UPS, but also about removing clutter.

There is dupebox, which in theory is cool, but I could never get it to work properly. Perhaps it's been fixed and I should try again. I'd be curious how it fares under a load test as well.

[joonazan]

I think it is possible to optimize faraway parts of a factory because this has been done for a similarly interconnected simulation already. The Hashlife algorithm can compute Conway's Game of Life far into the future very quickly. It has no benefit whatsoever for simulating one step, but it can perform extremely well for computing a million steps.

Both GoL and Factorio are unpredictable, as they are Turing-complete. A factory might produce 100 iron plates and then do that again, but it might also suddenly decide to write messages in morse code onto the output belt. Still, both don't do this most of the time.

Both are deterministic. This means that the same local state will always result in the same outcome. This is exploited by Hashlife by recording past states of local areas, so that if the same state is reached again it doesn't need to be computed again. Note that Hashlife needs to look at a bigger area than the area to fast-forward because there may be influence from outside.

The challenges for accelerating Factorio in this way is defining locality in a good way and dealing with its larger state space.

Locality

In Factorio, different methods of transportation move at different speeds. For example, a train can travel almost anywhere in a minute but a yellow belt won't get very far. What is "near" must be based on the different modes of transportation rather than a singular speed of light. Otherwise simulating a single second would require knowledge of all the places that a train could reach in that time. Limiting the optimization to belts and inserters only might be a good idea.

Red and green wires are very dangerous for locality, as they effectively make all connected entities next to each other. So is fluid, due to the speed at which waves travel. Fluid is also very bad because it can have so many different levels.

Complex state

Since every tile in GoL is only on or off, there are often situations where an area returns to exactly the same state it was in earlier. In Factorio, this would be rather unlikely. For example, the likelihood that a chest supplying a production line has the same contents after an insertion cycle is highly unlikely.

But this can perhaps be fixed by viewing the chest's contents from the perspective of locality, too. If we are looking 10 inserter swings into the future, it only matters if the chest has ten or more of the required item!

Conclusion

Memory use can be worked around. Maybe. Also, when a player approaches an area he hasn't visited in a long time, there might be a big stutter as the area catches us. Even worse, using the map view could be bad and massive multiplayer would destroy all the benefit.

But if you are making a new automation game that is all about scale, do this and rework systems like fluids that produce large noise for little gain. You should be able to far surpass Factorio with the caveat that not everything can be seen at once, be it due to limited view distance or putting factories inside buildings.

The locality viewpoint does seem like a good way to multithread Factorio but while it might make the game overall faster, it would increase it power consumption, as the overhead wouldn't be trivial. Partitioning the world into different local areas certainly isn't free (or easy to implement).