Friday Facts #260 - New fluid system

[Cobaltur]

Perhaps refactor it from floats to integers, which would make it work more cleanly, but would also make some calculations more complicated. TBD.

Oh that might solve a lot of other problems

liquid 0.x not fullfill "greater than 0" [0.16.24]

0.15.10 Fluid wagon leaving even though condition is not met

And what would be really cool: a filtering pump. One type would be enougth. E.g. only oil should be unloaded (in my oil factory outpost) so water would not got into it accidently

GUI and feature is already similar in the belt splitter filter

[Garlik30]

[...]

That's called barrels.

You obviously did not read. Or never used barrels.
The way I see it, barrels only use was when fluids wagons did not exist (in Vanilla). And they solve almost none (if any) fluid mechanic issue.
You still need tons of pipes in front of each building needing fluids, and in massive megafactories, this has a huge impact on UPS. While having bots transporting fluids DIRECTLY without needing the barreling/un-barreling mechanic would solve mega base builders issue with UPS. This would also allow Dominik to implement his solution (or any other that the Factorio dev team would find best) that would still be usefull for small base builders, or people who just like pipes.

I really see bots transporting fluids as a solution. Just like belts will never be as UPS friendly as bots, whatever change devs will do to pipes they will still drain tons of load on the CPU, except if they use the electric system, wich is overpowered for fluids IMO. Not saying it should not be done, just saying it will not solve mega base builders need for more UPS friendly late game solutions.

PS: this is the same discution as for nuclear. It will just never be as UPS friendly as solar, whatever great dev work is done on its optimisation. I love the mechanic, just cant use it in massive builds

[luc]

Hi Factorians,
This is Dominik, and my first FFF post ever! I will use this opportunity to talk to you about the exciting subject of pipes. Yeah, I know, right?

I'm only 2/3rds of the way but let me just say that this is a stellar Factorio Friday Facts! As an amateur (non-game) developer, I'm entirely wrapped up in the story and you really have me going on the topic of pipes . I really like that this FFF is longer and more in-depth. It really gets you into the topic, so I'm always happy to read those and often share those with some game developer friends. As another example, the netcode FFFs are also always very interesting. But if you have more to share about pipes or, heck, GUIs, please write it all up

[Matthias_Wlkp]

I'm reading these suggestions and I find this all way too complicated.

How about something REALLY simple:

1. Treat all connected pipes like a single storage tank.
2. Each pipe adds to total volume.
3. Measure two parameters of a single system - total volume and fill ratio (% of how much of it is full).
4. Any entity loading fluid into the system cares only if there is free space and adds instantly.
5. Any entity taking from the system has defined, how much fluid it can take per tick - say X
- per tick, take X * fill ratio - this way it would take more if the pipe is full and would gradually slow down until it balances itself if there is insufficient fluid stream
6. To avoid a situation when multiple machines take more fluid than there is in the system you have a couple of options:
a) don't allow the entities to take any fluid if there is less than Z% of the pipe full, assuming that X * 3 * Z% is less than volume of a single pipe
b) gameplay over physics - let the entities take more fluid than there is, forcing volume below zero, after 1 tick resupply to "0". Nobody (sane) will notice or care for a tiny surplus of a fluid for one tick...

Solves all your issues:
- tick effective (1 entity to calculate per chain of any pipes)
- splits fluids evenly
- allows for clearing a fluid type from a pipe system (with 6b)

Downsides - described in pt. 6.

Go simple!

[Ripshaft]

Very interesting update - I've got nothing to add regarding the process management, but regarding this line;

It abstracts away part of the realism and charm of the game. (While this is subjective at best, it does mean something to us.)

I just wanted to contribute a stern poke, reminding you of how incredibly important this is... whether the game is good or not is largely a subjective assessment - the appeal of the game is largely subjective. There's far more to the game than its mechanics, and it counts for quite a lot!!

If you dig the aethetic and feel of something, that may be more important than modifying the mechanics to be more computationally optimal if it comes at the expense of that... the mechanics and presentation both work together to create the experience that's so valuable and that contributes to the charm of the game. Synergy!

[mrudat]

From the point of view of getting maximum simulation done with minimum CPU time, would something like the following work:

• for each connected pipe network (on construction) calculate (and record for later use and display to player):
  • maximum possible flow through the pipe network (I believe that this was one of the possibilities already investigated)
    • boilers, steam engines and steam turbines count as a sink, but also as a pipe with 0 storage capacity if another boiler/steam engine is connected. (their internal buffer counts as the input buffer for the steam/water sink, rather than as additional fluid network capacity)
  • maximum possible flow from/to each source/sink
  • total pipe network fluid capacity (including tanks)
• during simulation
  • calculate total possible input fluid flow: sum(min(fluid at source,max flow from source))
  • calculate total possible output fluid flow: sum(min(space at sink,max flow to sink))
  • total input flow = min(total possible input flow, space in buffer + total possible output flow)
  • total output flow = min(total possible output flow, fluid in buffer + total possible input flow)
  • flow rate (for each source/sink) = min(fluid/space and source/sink, total input/output flow rate / total possible input/output flow rate * max flow rate to/from source/sink) -- I think; in any case, you want to evenly distribute flow across all sinks/sources in proportion to their max possible flow rate
  • display to player
    • input fluid flow, output fluid flow, maximum possible fluid flow for pipe network as a whole and for each source/sink; the flow rate for the source/sink should appear on the entity itself
    • total/max fluids held in the pipe network
• logically a fluid network that is 50% full has each item holding 50% of its maximum capacity, and that amount of fluid will be destroyed should the entity be deconstructed.

Possibly, as an additional restriction, calculate the amount of fluid immediately adjacent to the fluid sinks (based on the amount of buffered fluid) and limit max flow for each sink to at most that amount of fluid, so fluid doesn't teleport from source->sink the moment they're connected.


Also from the point of view of the fluid UI, can we please have fluids and fluid flow reported in Litres rather than units (given 1 unit seems to be somewhere between 50-100mL depending on fluid type); you could then flag that the circuit network signal is in 100mL increments (or better, allow a choice of unit to be read out of a fluid network).

[meganothing]

Nobody likes barrels. Sorry

Objection!

Interesting FFF! Great write up.

We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc.

Please do add this feature, flow control is a mod which adds these and it truly does improve fluid handling, especially in the later part of the game where the pipework turns in to the proverbial spaghetti. We've all been there - you need to sneak a pipe through a gap but can't because it'll connect to the pipe next to it and different fluids would mix..

The autoconnecting pipes separate pipe building from belt building. If pipes work largely like belts you don't have to think differently, don't have other topological problems to solve.

[foodfactorio]

hi, the new pipe system sounds like it could save lots of ups, but if you do your testing scripts, could you please also add some storage capacity variations, as some mods have large 500k capacity like Yuoki Industries, or 80k - 1500k in other mods (angel/py) and maybe what works fine in vanilla, might not correctly work with flow using larger capacities?

also, one of the main reasons i try to place pumps, is that i think a pump will help pull any liquids/gasses etc from earlier in the segment, and push it out to the destination, but more often than not, the pumps do not seem to work that way. (maybe it is because of the bugs you mentioned with the current system, but maybe pumps could have a special code that adds an additional overide to the basic algorithm, so that they will always pull in from the previous segment(s) at an additional speed/flow?) (and is so, possibly scaling up for each tier of pump?)

[Paul17041993]

The 'speed and volume' is pretty much what I would use (or do actually, technically), myself, though the term 'speed' also refers to 'pressure', 'momentum' and 'flow', which are all essentially identical terms in this context. The things that would control the 'speed', and in turn make the pipes behave realistically and without loop shenanigans, are; internal friction, internal pressure (in this case a vacuum) and direction change. So as the fluid traverses the pipe, the internal friction and any direction change eats away at the potential 'speed', limiting the maximum flow rate over long distances or around bends (including junctions, the flow will want to flow straight), but not prevent pipes and containers from being filled to the max. A pump (or container, or active conversion device such as a boiler) has the opposite effect at either end, sucking fluid in and accelerating the flow speed at both ends, thus using two pumps (even end-to-end) will drastically increase the maximum flow against the internal resistances (and mainly why my IRL watercooling uses two pumps as opposed to one).

I'd refactor liquids to use integers as well anyway, as it allows for atomic operations that are rather useful for high performance multithread or GPU/HSA acceleration, remember that modern mainstream CPUs are getting increasingly wide and laptops and low-end desktops almost always have an integrated GPU present of some sort, which is essentially usable as a simplified array of CPU cores.

[grig]

What it does not allow though is to limit the flow - one wire can, theoretically, run any current,

You can limit the current with resistance. Make each segment of the pipe have some resistance. This will also limit how far you can transport fluids with pipes and encourage use of trains with fluid wagons.

"Connect" sources (e.g. offshore pumps) to a high potential and assume they have internal resistance. This limits how much current they can output (this is very analogous to an electric battery).

"Connect" sinks (e.g. boilers) to ground (zero potential) and also assign some internal resistance to them. This makes them analogous to an electric load.

This limits the flow through a single pipe to V/(d * r + R1 + R2) where V is the potential the offshore pump is connected to, d is distance, r is the unit resistance of a pipe, R1 is the internal resistance of the offshore pump and R2 is internal resistance of the boiler.

If you only have offshore pumps, pipes and sinks (boilers etc) then your electric network basically consists of resistors. This means that if you write down the Ohm's law for each segment and Kirchhoff's law for every junction, you get a sparse system of linear equations. You could use an open-source solver library to crack that very efficiently. You can optimize it significantly by joining multiple pipe units into segments.

The tricky bit is the electric pumps. At a bare minimum, you want them to conduct only in one direction (basically, making them a diode). Unfortunately, this introduces non-linearity into your system: the Ohm's law doesn't work for diodes, and so you can't use a time-proven sparse linear solver anymore. There must be ways around it though.

[WeirdConstructor]

From a software dev with 20 years of experience: To all people that think they know or can estimate the performance of the result: You have no clue if you haven't run the benchmarks. Especially on current CPUs it's hard to impossible to predict. And you forget, that the algorithm Dominik implemented and debugged and rooted out edge cases from already saves computational overhead on 2 piece long pipes - and no base has only junctions in the pipes. You forget that the current system runs calculations for every little piece. Without knowing exactly how much more overhead the new implementation has, you have no base to make conclusions on.

[toketsu_puurin]

On the topic of handling mixed fluids I think the best solution (from a game play perspective) would be to create a new tech: distilling. It would give you something like a specialized refinery that can be set to separate water and oil (fast) or separate different miscible liquids (the different oils. This would be extremely slow.)

The key would be adding a sort of placeholder "fake" contaminated fluid I'll call sludge. Sludge would know what quantities of original fluids it contains, but the game otherwise treats it as a single substance. If you add two batches of this fluid together, it just adds them all together. Splitting into two batches would proportionally split the contained liquids.

Sending sludge into a distillery would give you a number of recipes depending on the products you want to get back. You could start with a recipe that produces petroleum (as a gas it should be easier to recover), water sludge (containing water and sulfuric) and oil sludge (which would have everything else.) The assumption here is that this will be a lossy process. Some percentage of sludge will be wasted to recover useful products.

[toketsu_puurin]

I just wanted to contribute a stern poke, reminding you of how incredibly important this is... whether the game is good or not is largely a subjective assessment - the appeal of the game is largely subjective. There's far more to the game than its mechanics, and it counts for quite a lot!!

If you dig the aethetic and feel of something, that may be more important than modifying the mechanics to be more computationally optimal if it comes at the expense of that... the mechanics and presentation both work together to create the experience that's so valuable and that contributes to the charm of the game. Synergy!

While I would generally agree that charm and visual appeal are important (I'm an artist,) there also comes a point where it doesn't matter how pretty it is because the game is unplayable. (PC Port of Arkham Knight ring a bell?) While you might not be hitting that point, other people are.

Stylistic flavor is important. All those tiny little details that most people never notice blend together to take the experience of a game from good to fantastic. But keeping something in the game just because it's pretty should never come at the expense of the game failing to work at an acceptable level. If it is failing that badly something has to change. It may be that some art has to go, or code has to be rewritten. Whatever it is will likely be horrible headache for the devs, but a game that doesn't work can't fulfill its primary purpose of being fun.

Fun first, function second, style third.

Kudos to the devs for not saying "yeah it's good enough," and pushing to fix it.

[fendy3002]

IMO, adding 2 limitations can help to greatly reduce the problem:

1. Fluid flow in integer, or better in 10 (or 2) units of fluid
2. Limit junctions to one way. For T junction there will be 1 input, 2 output each 50%. For cross junction there will be 1 input, 3 output each 33% or 2 input, 2 output each 50%. Configurable like splitter. That way balancing single pipe segment can be isolated and no circular.

[Alice3173]

Might be a bit late on this since there's already five pages but:

Would a simple solution for junctions like checking the difference between the current pipe's liquid and the neighboring pipes, averaging that out (ie: if the current pipe has 100 liquid, one neighbor has 70, and the other two have 30 you'd do ((100-70)+(100-30)+(100-30))/3), then splitting it between the neighboring pipes (obviously disregarding pipes that are fuller than the current one) not work for a simple but adequate solution? I don't know how the performance would actually compare to the current method though.

Another point left to consider is how to solve accidentally connecting pipes and tainting your whole oil system with water.

Wouldn't a simple solution be checking the contents of surrounding pipes and dissipating one of the liquids according to which one is the primary liquid in that system?

The autoconnecting pipes separate pipe building from belt building. If pipes work largely like belts you don't have to think differently, don't have other topological problems to solve.

That's a completely artificial challenge though. There's no logical reason to prevent a player from putting a pipe between two other pipes other than not wanting to limit the smart connection system.

[Rinin]

...
The tricky bit is the electric pumps. At a bare minimum, you want them to conduct only in one direction (basically, making them a diode). Unfortunately, this introduces non-linearity into your system: the Ohm's law doesn't work for diodes, and so you can't use a time-proven sparse linear solver anymore. There must be ways around it though.

It's not tricky, it's one more source and one more drain.

[zOldBulldog]

Adding my vote to some comments posted by others:

- Nuclear option: yes please. It is the good old kiss principle. I understand the "prettyness" of the more complex solutions, but they add little value to game play and cost too much in UPS. The other options just don't have enough payback for the cost.
- Better circuit network integration: yes please.

As to preventing fluid collision/contamination, why not model it as a "blowup" : Some visual effect similar to an explosion and the pipe or other component where the two fluid touched ending up disconnected and on the ground like when you try to pick it up and you have a full inventory. That way you completely eliminate contamination, but there is still a price (a minor one) to pay for the mistake.

[Zlutz]

Now we just need like 3x3 tank with 8 exits and/or similars so it'll be easy (and UPS friendly) to evenly split the flow!

Perhaps the best solution is not to create 8-exit tank, but to program the normal 4-exit tanks so they are combined to "one big tank" when they are planted one next to another, that would probably be very UPS friendly and enable us to have a "liquid splitter" with any number of exits!

[Dune]

Honestly, thanks for the update. Any help on fluids is much appreciated. Fluids is the biggest challenge in factorio, such that I find putting fluids on trains and transferring them that way, much easier than trying to pipe them.

Fluid mechanics, be damned. This isn't a simulation on real world models. Does nuclear fuel reprocessing exist? No! Power put on the electric network, is accessible anywhere else that/next tick. Realistic? Hardly!

Let's not get too realistic here either. I hate the "slosh" factor in the current system. It might be realistic, but oh so annoying. Let's also remember, this is a futuristic game. Our guy crash landed on planet using advanced space science to get there. Let's believe they came up with a solution to handle fluid smart in the future too.

"Eye-candy" is a nice feature, but it doesn't have to be the actual mechanics that is being used.

For power production, solar is 1 entity no matter how many panels are built.

Nuclear is a million entities, (not counting the requirement of all sorts of combinators to make work) and will never work for megabases because of the hundreds of (thousands?) of calculations needed to make it flow. Times the number of reactor setups.

The only way to make fluid UPS friendly is to limit the entities updated in the system. A suggestion, make connected pipes of a certain size(count of fluid containers or within a single chunk/region) one entity. Not just "straight," but all junctions and entities that connect to it. One of your bullet points was permitting instant transfer. This is it.

You can simulate the eye-candy when a player views it, if you'd like.

[TheDagmaar]

I am not sure if anybody will read this....

What about a "vacuum" system. If a consumer needs a liquid, it sets a vacuum++, vacuumMax == fluidMax, so if there is 0 fluid, then there will be vacuum = vacuumMax.
The fluid allways go where the vacum > other vacum. So the vacum go through the pipe system like a liquid BUT it will be only a line to the source.

factorio_pipe.jpg (34.04 KiB) Viewed 7112 times

The vacuum:
-always the greatest at the source
-when you go further from the source there will be a percent of the vacuumMax
-after a long section, there will be no vacuum

The graphics problem: there are small windows on the pipe. So what if every segment of the pipe have only one window on it and a segment can't be longer than 5-7 pipe, so there will be a nice flow effect, and nobody knows whats behind it.