Optimization idea: abstraction

[Cerus]

Once a factory reaches a certain size, you start repeating sets of structures over and over.

Once the blueprint system is in, what would you say to being able to place a blueprint for a "black box" style factory structure?

Would there be any significant performance or convenience benefits by analyzing the contents and workflow of the structures in a blueprint, and creating an equivalent structure that (presumably more efficiently) mirrored the function of the blueprint?

[ficolas]

You mean like the production stadistics when pressing P but just for a part of the factory to check how efficient that is to make it into a blueprint?

[Gammro]

I assume by a black box, you mean "single building for whole chain in the blueprint"?
Then I'd say no. Space is a valuable resource in this game, and optimizing designs yourself is part of the gameplay. I think the blueprints are there to help with repetitive building of structures, but shouldn't magically increase throughput and efficiency.

[Cerus]

The factory would be variably sized to match the space taken up by the original blueprint. You're still designing everything and space matters just as much.

This is mostly a scaling performance thing. Fewer rendered objects, simplified algorithm that reproduces the ultimate function of a multitude of complex separate objects.

[Cerus]

You mean like the production stadistics when pressing P but just for a part of the factory to check how efficient that is to make it into a blueprint?

That's kind of a part of it. It would analyze the production/consumption of the parts in the blueprint, as well as inserters/conveyors and where they entered and exited the blueprint. You'd then place the blueprint as a factory, and it would be built with the normal requirements, but as a single object that simplifies the redundant rendered and calculated components such as conveyors and inserters.

Take a blueprint that was a simple line of conveyor belts.

Code: Select all

> > > > > > > > > > > > > > > > > > > >

The factory version of this would be:

Code: Select all

>[                                   ]>

The cost to build the blueprint is 20 conveyor belts in either case. But in the case of the factory, instead of calculating movement across 20 conveyor belts, it would be calculating 2 conveyor belts (entering and exiting the factory), and condensing 18.
Whether or not this is more efficient depends entirely on how things are being calculated now, which isn't something I have any knowledge of. It's possible that this example could be less efficient, but other combinations of components would see a benefit.

[ssilk]

I suggested that a while ago ( https://forums.factorio.com/forum/vie ... f=5&t=1016 ), but nowadays I tend to say, like the devs, that a "what you see is how it works" is much better.

The sense of your idea is maybe at some point, when we have such big maps/factories, that it makes sense to simulate parts, which are very far away. In that case, blueprints may help, but that is a very complicated problem and it must hurt very strong, before it is really needed.

[Cerus]

Glad to see a version of it has been suggested before.

I completely reject the "what you see is how it works" argument, as my intuition greatly favors abstraction and summaries and that runs entirely opposite the repetitive mess of conveyors and inserters that we end up with now.

Creating this system would definitely be a lot of work though, and the benefits would likely be fractional. I might just wait to see how blueprints get implemented and try my hand at a mod.

[ssilk]

I think this repetition and not abstraction is one of the biggest points of factorio,
The object are real. Not like in sim city 4, which is only a simulation. Every single item, every entity is a real object In memory and not a formula. I would say it is a physics simulation.

Abstraction would mean to break with this concept. I would say: not needed yet.

[Cerus]

And I find that repetition meaningless and valueless. There's no comparison to Sim City 4's abstractions if it loses accuracy, that's a world away from the kind of simplification I'm talking about, not remotely comparable.

But ultimately yes, I was already agreeing with you. It's not needed right now, this is a "future consideration" kind of thing.

[ssilk]

Not that I want to have the last words, but repetition is part of the whole live. Music won't be possible without. The pixels on your monitor, on which you currently look, are very boring, but I know some people, which can occupy the half of the computer department in a big warehouse, when some of them are not light. And every abstraction looses it's details and that is sometimes a big difference.

There are for example this belts with the two lanes, but if you manage to lay items in the middle, then, sometimes, this can block the whole belt. Cannot be simulated, because it happens only under very seldom cases. And it was not foreseeable, that this will happen.

Factorio is very rich of those very minimal, but sometimes very important differences, which makes it so surprising - even for me - and happens only within this repetition of ever the same, but with changing cases.

[Cerus]

Culminating in a bizarre appeal to the supposed beauty of repetition.

I regret posting this.

[hoho]

Premature optimization is the root of all evil
- Donald Knuth


How big must a factory be to actually cause a slowdown on half-decent PC (newer than 10y) so that such optimizations become useful?

[kovarex]

Premature optimization is the root of all evil
- Donald Knuth


How big must a factory be to actually cause a slowdown on half-decent PC (newer than 10y) so that such optimizations become useful?

Optimisations are still important and with blue prints and possibly multiplayer it will always be. (https://www.factorio.com/blog/post/fff-18)

[hoho]

Optimisations are still important and with blue prints and possibly multiplayer it will always be. (https://www.factorio.com/blog/post/fff-18)

I was talking more about just blindly going in and trying to optimize things without even knowing what exactly slows the things down and to optimize before a clear picture of bottlenecks is seen. I've seen too much code in my hobby/professional programming years where people have started optimizing long before even the basic functionality was there. They ended up making future changes to the code way harder due to trying to do all sorts of "clever" micro-optimizations that pretty much did nothing besides making things harder to understand/change.

If bottlenecks are discovered and they have something obviously non-optimally coded then sure, go ahead and optimize away. While cache locality sure is important when dealing with tiny pieces of memory algorithmic stuff is nearly always WAY better to optimize.



Just a random question about the internals of the program, do you use any sort of linked lists for stuff that are somewhat performance critical and/or big? It's again somewhat of a micro-optimization but I've yet to see any place to use them instead of some sort of array/vector

[kovarex]

Optimisations are still important and with blue prints and possibly multiplayer it will always be. (https://www.factorio.com/blog/post/fff-18)

I was talking more about just blindly going in and trying to optimize things without even knowing what exactly slows the things down and to optimize before a clear picture of bottlenecks is seen. I've seen too much code in my hobby/professional programming years where people have started optimizing long before even the basic functionality was there. They ended up making future changes to the code way harder due to trying to do all sorts of "clever" micro-optimizations that pretty much did nothing besides making things harder to understand/change.

If bottlenecks are discovered and they have something obviously non-optimally coded then sure, go ahead and optimize away. While cache locality sure is important when dealing with tiny pieces of memory algorithmic stuff is nearly always WAY better to optimize.

I would like to assure you, that I launch profiler 30 times per day when I do optimisations, so yes, I'm doing all the stuff based on measurement of the profiler results. Things that are not critical don't get micro optimised.

Just a random question about the internals of the program, do you use any sort of linked lists for stuff that are somewhat performance critical and/or big? It's again somewhat of a micro-optimization but I've yet to see any place to use them instead of some sort of array/vector

Ha, we use many different data structures of different kinds and linked list is just the most simple example
When talking about linked list, we are using boost::intrusive_list A LOT (Nice explenation here http://www.codeofhonor.com/blog/avoidin ... nked-lists). It is not uncommon for an object to have more than one intrusive list hooks ...
But the best structure to be used depends on the use case a lot. On some places, where linked list would seem like common sense (big list and the need to remove from center quite often), it is better to use vector, to have cache locality, and solve the removing from center by swaping with last element.
The fun part is, when we need to serialise all these structures while we need to keep the order of everything to be exactly the same after save/load to keep determinism (For replays and multiplayer), we have to develop special utilities to make it possible in fast-to-use way.

[StephenLynx]

From reddit:
https://www.reddit.com/r/factorio/comme ... ssibility/
Recently a different post about abstraction made me think. What if the game could take isolated groups of assemblers and internally, abstract it? Only calculate what's coming in the area and how much it should come out.

Take this red circuit production area of my factory as an example: https://i.imgur.com/PjIDmPs.jpg

Plastic, copper plates and green circuits come in, red circuits come out in a steady pace. As far as production goes, it isn't related to anything else in between the train stations, aside from power supply. So couldn't the individual assemblers, inserters and belts be abstracted and just calculated what's in, what's out and the leftovers for the next cycle?

Now, I know this present some issues. What about when the player comes to look at it and expects to see individual wires going around and other stuff? And I agree, that do pose a challenge. I'm not talking about making the game behave any different as far as the player knows and sees. What's I'm talking about here is not about a hard suggestion, just a venue of thought that might produce some fruits if the developers think about it.

But if this is possible, think about how much UPS it would save. How larger factories could be or how better lower end machines could run it. I dare to say it might be the biggest optimization for a modular factories this game have ever seen.

[5thHorseman]

It would be crazy. Imagine an entire smelting array abstracted to just a few variables, like how many plates can be created per second (or tick or whatever), how long it takes a chunk of ore to turn into a plate, and how much power per plate is used. Then a train comes and drops off ore, which goes into the abstraction zone and disappears for x seconds, using y power, and a plate comes out of the zone on another belt.

I don't know how the game could properly do that on its own, but imagine something like Factorissimo where the player sets them up and then throws a black box over them intentionally.

[Koub]

The first thing that came to my mind reading OP's post is https://mods.factorio.com/mod/Factorissimo2 too

[eradicator]

What if the game could take isolated groups of assemblers

The impossibilty of this is the whole reason why factorio is in large parts not multi-threaded yet. As has been stated in various FFFs over the years. Everything interacts with everything. And imho one of the major selling points of Factorio is that it precisely simulates everything. If you put 20million ore in you get 20million plates out, and not "approximately 19.5million plates". So even if you could abstract it would be ugly.

[ssilk]

TL;DR
Parts of a factory can be simulated, if the conditions that influence that part have very specific values. E.g. input belts needs to be fully compressed, and many more. In that case it is safe to switch to a simulation.

Long read:
Well, I think this is a way we should think about deeper.

What I’m thinking here is also Factorissimo, but instead of just one building it should have the “real size”.

So what I see here are some possible solutions, which would simulate a realistic production process for very large quantities but switch back to the realistic simulation for low quantities.

How could that be done?

First I think about the concept of factory-streets. Like a gigantic smelting array. This can be simulated very exactly. See for example the max rate calculator https://mods.factorio.com/mod/MaxRateCalculator

But it needs a lot of preconditions to make sure, the simulation is always exact:
- all input belts are always compressed
- enough free space on output belts to put items on
- inserter are fast enough to feed/empty the smelters
- there is no complex belt stuff, like when it is possible that two different items could be on one lane
- same is with mixed input lanes. As soon as another item on a lane is imputed it needs to switch back to simulation.
- there are no circuits involved
- there is always 100% power available (well I think this can be simulated up to some degree)
- pipes are never filled below some level X
- player is not changing anything inside that factory street are
- the theoretical input/output is equal to the current input/output
- and of course some more preconditions.

You see it’s difficult to achieve those conditions so I would like to introduce a “factory-street-area”. Normally this area is hidden with some kind of industrial roof (game doesn’t need to simulate the inside), but when the player enters, the roof becomes transparent. When you want to built inside such an area the factory street switches to the normal mode and everything behaves as it should. You can then built inside the factory street, but you cannot built some constructions. For example you accidentally side-insert a different item into a belt-lane. You cannot do circuits, or you forgot to power a device.

But when you go outside and the above conditions are meet, then it switches to simulation mode after a while. After a while means: the theoretical input/output needs to be more or less equal to the current measured input/output.

And as soon as one condition isn’t meet anymore the factory street switches back to normal mode. That is likely, if the output is blocked or not enough input comes in, or if power fails. But for those cases that input is missing or output is blocked it’s already so, that the inserters and entities go to sleep. So not too much CPU used. The factory street covers only the case, when it is running with full speed.

So what does this bring?

- useless if you only have one factory street, because if you want to change the layout everything is as before. You need to built many streets.
- useless if you are not capable to supply a FS with a constant stream of material. This is an interesting task.
- And this is also a moment, where a factory street could be an advantage for the player: lets say the FS works and it is able to switch to simulation mode. What if the player then is able to increase the speed or productivity etc with modules? The modules work only in the simulated mode. ... not thought to the end but that could be the moment when you really want to build this.
- also useless, if you don’t need so much output...
- ... which is a quite interesting logistic task to build that so, that the factory street is either working in simulated mode or is turned off, because that takes much control from outside. This makes it also much easier to switch off the power, if nothing is produced anymore.


But for the case that the preconditions are meet, and you have many factory streets working in simulation, this should be a real bringer! And it would enable much, much bigger factories than now.

If this is also combined with some boxing mod, like deadlocks https://mods.factorio.com/mod/deadlock- ... es-loaders which enables to produce with boxed items, then I think factor 10-100 bigger factories should be possible.

I think for many players that would be a really good argument.