Optimization idea: abstraction

[Rseding91]

When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:

Does that mean that active=false only affects the entity loop and every other loop is unaffected?

Yes

[Nidan]

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.

Event-based is inventory: on-item-added: send event, on-item-removed: send event. Circuit logic is polled: each tick go over each thing that has signals to send and add them together. Then that's the state of the circuit network for that tick. Next go over each thing that wants to read it and have them do their logic.

There's no event for fluids flowing into a storage tank that's connected to a circuit network. There's no event for any of the circuit network; it's all sum-all, iterate users and calculate.

The fluid system not having events makes sense.
But if you were to store the signal the tank provided in the current tick and, in the next tick, compare the signal the tank would provide with the stored one, you could decouple both systems. The same could be done for other entities with a similar situation.
Given that, you'd have a method for detecting change and the decision whether to poll each tick or to use events becomes a CPU vs memory trade. You'd have to store the signals provided by each entity and (for display and faster updating) the sum of each circuit network, but save time when no input changes. In addition mods could make use of the event and wouldn't have to rely on on_tick.

[Impatient]

...
But if this is possible, think about how much UPS it would save. How larger factories could be or how better lower end machines could run it. I dare to say it might be the biggest optimization for a modular factories this game have ever seen.

The basic idea you are bringing on the table there, is the idea of black boxes. This idea has been juggled around time and time again.

Now, IMO and as stated by many posts already, in regards of micro simulation, this can not be done in factorio with the same degree of accuracy. Currently the accuracy in factorio is 100%.

The next thing that comes to mind is, just to accept lower accuracy as trade off for more UPS. Why not? That is something that would be possible.

But, ... the question is IMO, how should factoio determine what the production rate or throughput of such a black box is/should be ( without it being possible to be exploited? ) That is a big thing and has yet to be answered.

Btw, some modders took a very simple approach and created new entities which have the same throughput and production stats as known blueprints ( eg a whole smelting line ). That is a great way to improve UPS, if you want something like that.

[eradicator]

Now, IMO and as stated by many posts already, in regards of micro simulation, this can not be done in factorio with the same degree of accuracy. Currently the accuracy in factorio is 100%.

The next thing that comes to mind is, just to accept lower accuracy as trade off for more UPS. Why not? That is something that would be possible.

Actually that is a point i've thought about with @Qon's idea and i have to admit that for me personally the accuracy might actually be quite important. There's so many games out there that give you only a vague idea of what is actually going on, and factorios precision clockwork is a very welcome change from that. So a mod that manipulates statistics and lies straight into my face about *everything* that's going on might not be as appaealing to me to play as it is technically intriguing to talk about.

[ssilk]

Actually that is a point i've thought about with @Qon's idea and i have to admit that for me personally the accuracy might actually be quite important. There's so many games out there that give you only a vague idea of what is actually going on, and factorios precision clockwork is a very welcome change from that. So a mod that manipulates statistics and lies straight into my face about *everything* that's going on might not be as appaealing to me to play as it is technically intriguing to talk about.

I think that’s an important point.

And my automatic next question is:
How much can a simulation of Factorio “lie”, to be accepted?

To be more exact: how many percent exactness do we need, to say “this is an exact simulation of Factorio.”?

And to measure it I suggest a simple way: we measure how many items are made in reality (over very long time) and measure the items produced by the simulation. And the error of the simulation can be expressed in items per time.

E.g. an error of 1 item per hour means, that there was produced 1 item too much r too less in an hour for that part of the factory. And because “that part of factory” can be expressed as an area, that produces some items per hour we came to the unit of
errors per items

... and the question that follows is, how much errors per items is acceptable and what not? Some say zero, others don’t bother with one error per 10 produced items.

How much would be ok for you??

[yagaodirac]

I literally accept all the changes, all the levels of changes.
One thing I have to point out, within all the similar games I've played or at least watched the videos or streaming, Factorio is very special. The design of belts with 2 sides, inserters, loaders, and even the variation of shapes of train stations, are only be seen in this game. They are too complicated. If you accept new entity, replacing is a great implementation which looks like merging. If you receive some new mechanism of communication between nodes(furnaces, assemblers) and segments(belts), all the inserters, loaders, are gone. If you accept a pseudo power system or removing the whole power system from this game, the game could be simplified a lot. People can literally concentrate on crafting better.
But, you are not gonna accept any of them.
In fact I knew some studies on how to save cpu at least 1 year ago. But those dudes didn't do any actual experience. But even without clues, we still think the belts and drones harm ups the most. We even tried to carry every single item inside wagons, but nobody eventually made any experience on that. I even invented a railway net pattern to simplify and speed up the trains, but yet any experience is done about it. And very few people know how to put items from a wagon into another wagon with out red inserter or extra vihicle.
So, which would you prefer? Modding some half cheating entities? Modding some algo to lie? Reinventing the way you build up the super factory? Or even make scenario to separate things into multiple surfaces. I don't know if this make use of multi core of your cpu.

[yagaodirac]

OK, besides. Some of you are interesting in algorithm to directly simplify the computation in vanilla. But you need to know, if you parse a specific structure, like smelting zone, only the corresponding zones benefit. The more a zone is logically near to mining, the practical to do this.
If an assembler or furnace is built, no matter how, by player or by robot, you can parse the belt lane provides to it and the lane receive products from it. And if some other assembler carries out the same producing task, you could merge them logically. But if you don't involve the inserters and belts, it doesn't help very much imo. But if it's possible to parse a whole zone, 48furnaces,nearly 200 belts, 96inserters,3 splitter 2 underground belts, 18 poles, it's very signifisantly. I think this is what you truly need.
The disadvantage is that, this method only works well with some traditional and widely used structure and logically near to mining. Imo, it works only with mining, smelting and some very basic assembler zones like circuits, and the very final lab.
But as I know, people invented at least 5 smelting zone layout. And if people link 2 assemblers with inserter (the case when producing green and red circuit), it makes the parsing complex. And 2 zones are not possible to share any part. It means, you have to put corper cables on a belt, and send them to somewhere else to craft circuit. The issue harms the designing of green, red circuit, belt, inserter, red bullet, and some more.
Besides, ore zones are irregular, no easy way to keep track of all the tiles with some specific kind of resource. If you track them in lua, it's slow. I asked some container in cpp to help us handle data in lua. But the staff is not interesting in this. Obviously, very few mod would make use of this container.
So, I really can't help. Good luck.

[ssilk]

@Yagaodirac parsing is not useful (to take up your words), the calculations you make there are just theoretical output. The true output can only be measured against fully working assemblies.

Hm, how to I explain this?

Let’s assume you can mark some part of your factory to be measured some time. And what we are searching for are the assemblies (furnace, refineries etc.).

Now we can have two situations:
A) not all assemblies work 100%. Some need to wait for input items, some cannot output items.
B) all assemblies work 100% all the time.

B is what we are looking for!! Because only then the real output is equal to the theoretical output and only then we can really simulate it, because we can calculate the exact throughput so, that the error per item is minimized (I think lower than one error than 100,000 items).

And to achieve that we need to build such areas a little bit differently than now.

Every assembly needs to work 100% of time, which means we need to make sure, that there is always enough items to be inputted and the output never get stuck.

This can get quite complicated, because we need also to take in account the buffers (in chests, on belts, in the assemblies, in the hands on the inserters). We can only ignore them, if we have another buffer that feeds the whole construction, which is much bigger, than the internal buffers. The same for the output.

So again the steps:
A) built an “area”, that can be simulated (separated). Which means: no circuits, no bots, small internal buffers, assemblies work always 100% and the only connection is power and some input- and output-buffers (chests?). There shouldn’t be also no big internal buffers, because that enables cheating.
B) before we can start to simulate that, we need to measure for a while, if this is really the case. We can calculate the turnaround time for all assemblies (length and types of belts + production time + inserter time) and double that. But at least input and output needs to have been stabilized, before we can stop measuring. Otherwise the possibilities to trick this out somehow is too high.
C) when that is stabilized, we can simulate it. Calculate the theoretical throughput. Pick items from input buffer and put items into output buffer. The area closes its roof, so we cannot see, what’s happening inside (and not need to simulate the inserters etc.)
D) if one condition changes (no power, not enough input item or output is full), stop simulation.

I surely have forgotten some things, but I say it is possible.

[yagaodirac]

The @ symbol didn't show any notice to me. Anyway.
We share the same way. Except for some details, which don't count at all.
I think we should review the design of this game, rather than base on it and then try something epic but not very practical.
1, What is the purpose of scaling base. Does giant base give us fun? Personally I don't believe.
2, What is the purpose of this game? I don't really take this game as a game. I spend time studying its api, the same way I did when I learnt UE4 and Unity. For now, even the Mtn Fortress or Crash Site is not good enough. This game is not playable at all. It's more like 1/2 game engine + 1/2 pseudo physical experiment simulator. I believe we should spend some time figuring out what is funny enough and possible to base on the api.
3, Some questionable design. Why inserters? Why loaders? Why put poles on the same layer with other entities. Why 2 sides for the belts? Any alternative to the assembler mechanism? Better way to manage the belts?
4, I saw only one one-mod-scenario, which is named singistics. It shows a great idea, but not good enough. I tried to improve it, but didn't push it any further. Is vanilla mode holy enough and worth people paying that much attention?
I prefer to abstract every question before I try answering it.
The problem shouldn't be how to parse the "block", how to simulate it. The point is, what should a game do, how to design a mechanism, how to design the rule, and eventually, what is fun?

[Qon]

I think we should review the design of this game, rather than base on it and then try something epic but not very practical.

I already know it to be great. Why should we review it, in this context? If we are modding then if we think something sounds cool and worth implementing and do it then that's justification enough. If others like it then that also tells us a bit about if the idea is good or not.

1, What is the purpose of scaling base. Does giant base give us fun? Personally I don't believe.

Yes, giant bases provide more fun. Every time you scale up production by an order of magnitude you find a new type of bottleneck that gives you a new type of play experience and new design problems to solve. Basically it's another "level" of play.

You might go from belts to trains because belts don't scale and become too cumbersome when going from 10 to 100 SPM. But when going from 100 to 1000 SPM you might need to redesign your rail network because you get weird congestion.

When going from 1k SPM to 10k SPM you might do it by building 10x 1k SPM factories. But now you need a modular 1k SPM design that can be replicated easily and can easily accept resources by rail. You will want to make the factory compact enough that the space becomes manageble and you might want to reduce trains for each 1k SPM factory to make it easier to keep track of and set up trains for each new factory.

At 100k SPM with 100x 1k SPM factories you will get tired of managing everything about clearing and claiming space, connecting mining outposts, traveling etc. If you didn't make the whole 1k SPM factory and all the other infrastructure needed to build that buildable from map view then you will have to redesign your blueprints again. You will also want to think about how to automate the logistics of shipping building materiel and how to manage outposts that only build more infrastructure like modules, rail and assemblers, because the robot flying speed for building things this far away is going to be really limiting your expansions and you need separate construction networks but need automated logistics of entities for building. At this stage you are not really optimising for SPM, but for SPM added per hour (or Science/(minute^2)), that is, how fast can you scale production? Because if you don't speedrun building and do it all from map view then you are easily get stuck at creating a megabase every 20 hours or so and you won't get there within thousands of hours.

For 1M SPM you have probably already hit the limit a while ago of keeping UPS at 60 from just building with hundreds of thousands of construction robots, even if the factories you have built require 0 processing time. But you need ConMan/Recursive Blueprints and completely programmatically expand your base to practically be able to reach something like this. Which is another fun challenge to do even if reality of physical hardware stops anything like this from actually reaching this stage.

Disclaimer: numbers are wildly approximated within a few orders of magnitude. CPU cost of running factories is assumed to be about 0.

[posila]

I think we should review the design of this game, rather than base on it and then try something epic but not very practical.

I already know it to be great. Why should we review it, in this context? If we are modding then if we think something sounds cool and worth implementing and do it then that's justification enough. If others like it then that also tells us a bit about if the idea is good or not.

yagaodirac is trying to design a new game around the abstraction optimization tech. Which is a proper way to design a game if you indend to push some optimization to extreme limits - don't do things that make the optimization too complex or impossible to implement.

But I think it's a bad idea attempting to do it as a mod for Factorio. You'd be attempting to circumvent the game simulation in a game that's built around everything (that is built by player) being simulated and accounted for a all times. Making a new game in UE4/Unity seems easier than trying to bend the game this way through its modding API.

[ptx0]

But I think it's a bad idea attempting to do it as a mod for Factorio. You'd be attempting to circumvent the game simulation in a game that's built around everything (that is built by player) being simulated and accounted for a all times. Making a new game in UE4/Unity seems easier than trying to bend the game this way through its modding API.

ew, ew, ew. Multiplayer in those two engines is awful.

[yagaodirac]

So, would you remove inserters in 1.1?

[ssilk]

@qon: the ultimate goal of Factorio is to built a self-replicating game. So that the player is not needed anymore.

But that makes only sense , if the factory is so big, so reliable, that you don’t find any good idea how to play it now.

[yagaodirac]

According to my researchment, people believe a game is sacred when the people don't know how to develop games, neither do they have any idea how the current game was made. I revealed this pheonominon multiple times in different games through some genre. I don't remember who it was proposed to speed up the game with super base only by new algo, and came up with a brunch of limitation. The whole proposal is not practical at all imo. But what's more, it doesn't make the game any funnier even if it's done already.
I study on game rules and mechanism in my spare time. Giant base means more addictive than more fun. It's different imo. People who love super giant base would probably love idle games as well. If that is the truth, many free idle games are available on websites. Some of them are even in good enough quality, and for totally free. A series is named idle defender and made by barbasu. Another is synergism. If you like them or not, let me know.
But Factorio is not very compatible with long term incremental style gameplay. Some of the modders make some extra lvl of assemblies which may help some how.
Anyway. I recommend removing inserters and loaders. They are meaningless and helpless, ruin the gameplay and waste both real life time and cpu.

[posila]

So, would you remove inserters in 1.1?

No. 1.1 is not a new game.

[Qon]

yagaodirac is trying to design a new game around the abstraction optimization tech.

Ah ok. I missed that somehow.

But I think it's a bad idea attempting to do it as a mod for Factorio. You'd be attempting to circumvent the game simulation in a game that's built around everything (that is built by player) being simulated and accounted for a all times. Making a new game in UE4/Unity seems easier than trying to bend the game this way through its modding API.

Making a new game would require remaking everything. The mod might take a lot of work but a hacky proof of concept with a more limited feature set and other tradeoffs that can be improved over time is probably feasible at least. Depending on how much all the other update loops (that aren't affected by active false) use processing power when all entities are active=false.
It's not a bad idea for a mod if people want it to be made.

The core idea is rather simple actually, for a "stable" factory. The hard part is getting unstable behaviour right and to make partial factory parts run and splitting the abstraction etc.

[ptx0]

no one mentioned Schall Machine Scaling mod yet?

[NotRexButCaesar]

Anyway. I recommend removing inserters and loaders. They are meaningless and helpless, ruin the gameplay and waste both real life time and cpu.

Is this a joke?

[yagaodirac]

Is this a joke?

In fact not. But it indeed looks like a joke.