Friday Facts #260 - New fluid system

[csdt]

Physicist here. I happened to bring up the wonky fluid behaviour about a year ago, but it didn’t seem to be a high priority at the time. Back then I came up with some alternative algorithms and ran some simulations in an Android app I hacked together for this purpose (can send APK+source if interested).

Based on what I came up with, the proposed model seems needlessly complicated to me. So I think my findings are worth sharing.

In my simulation I used the Euler equations (assumes zero viscosity) because they are computationally cheap and good enough for our purpose.
...

Hi Yeast,
thanks you for your proposal, so far it seems to be the most reasonable, although I am still unsure about the electricity model which also is interesting.
I admit that my knowledge of physics is rather limited, I had to rely on my common sense. About the “speed kinda models the pressure”, I suppose I could have put more thought into it when writing it. When I think about it, pressure describes it best, it's just that it developed from speed originally and is still called that and I never took the time to rethink what it is.
The model actually is not that dissimilar. The pipe segment has volume (same to level), and the pressure, and connections store the speed - which is just the last step flow. The speed only plays some role in the update and does not control it fully.
From what you propose I really like the idea to push towards only using the last state. I will have to think about it, might steal it. Right now I am thinking about allowing going over the pipe volume limit which would allow this and simplify things a lot.
Your equations are a simplified version of what I currently have. Unfortunately I don't think it would work very well in this form.
The main issue I see is a very low reaction speed. For example - have a straight pipe from source to drain, source is pumping, drain is closed, pipe is full. The drain opens. What I think your model does is that in first tick, only the last segments moves 1/4 of its fluid into the drain (assuming dt=1, which is about what we have) and all the rest stays in place. In next step, one more segments moves a bit, 1/16. Etc. You need "length of pipe" iterations for the wave to reach the source and quite many more to get speed. And then when you get the momentum and you close the pipe, how does the flow stop?
Even so, your ideas are inspiring, I think I will use them, if you don't mind
Thank you, D.

Be careful on some points: there is no relation between speed of the fluid and the pressure of the fluid. What you actually have is a (linear) relation between the speed and the difference of pressure. It is really crucial to understand this.
Even more important to realize: a pipe might have a maximum flow per tick higher than its own volume (if the fluid is really fast).

Also, I still think the electricity model is the way to go. I wrote a post about it a few pages ago: viewtopic.php?f=38&t=62489&start=120#p378993

[TheYeast]

Hi Yeast,
thanks you for your proposal, so far it seems to be the most reasonable, although I am still unsure about the electricity model which also is interesting.
I admit that my knowledge of physics is rather limited, I had to rely on my common sense. About the “speed kinda models the pressure”, I suppose I could have put more thought into it when writing it. When I think about it, pressure describes it best, it's just that it developed from speed originally and is still called that and I never took the time to rethink what it is.
The model actually is not that dissimilar. The pipe segment has volume (same to level), and the pressure, and connections store the speed - which is just the last step flow. The speed only plays some role in the update and does not control it fully.
From what you propose I really like the idea to push towards only using the last state. I will have to think about it, might steal it. Right now I am thinking about allowing going over the pipe volume limit which would allow this and simplify things a lot.
Your equations are a simplified version of what I currently have. Unfortunately I don't think it would work very well in this form.
The main issue I see is a very low reaction speed. For example - have a straight pipe from source to drain, source is pumping, drain is closed, pipe is full. The drain opens. What I think your model does is that in first tick, only the last segments moves 1/4 of its fluid into the drain (assuming dt=1, which is about what we have) and all the rest stays in place. In next step, one more segments moves a bit, 1/16. Etc. You need "length of pipe" iterations for the wave to reach the source and quite many more to get speed. And then when you get the momentum and you close the pipe, how does the flow stop?
Even so, your ideas are inspiring, I think I will use them, if you don't mind
Thank you, D.

Thank you for your reply Dominik.

First of all, it would not be stealing. As I said, i wrote my simulation for the purpose of improving Factorio, and I even gave a short demonstration at your office when I visited Prague last year, but back then fluid realism wasn't a top priority Anyway I would be disappointed if you didn't consider using it.

Now about the reaction speed issue. I'm absolutely sure that there is a delay in reaction, and I'm equally sure that this is not a problem. In the situation you describe it indeed takes some time before the source 'becomes aware' that something is draining the pipe. In fact, this awareness will travel even slower than 60 tiles per second!

The general solution for the one-dimensional wave equation is

Code: Select all

u(x,t) = F(x-c*t) + G(x+c*t).

Here u(x,t) is the amount of fluid at time t and position x. F and G can be any two continuous functions. They represent two waves of arbitrary shapes travelling to the right (F) and travelling to the left (G). Their shape is dependant on the circumstances (sources/drains) but because of their arguments ( x+c*t and x-c*t ) they travel exactly at speed c and -c. And c simply depends on the flow update step:

Code: Select all

connection.flow += (connection.pipe0.level - connection.pipe1.level) * c^2;

(This can easily be derived from the wave equation)

As I discussed in response to BlueTemplar, I was using more realistic physical constants than Factorio is:

Code: Select all

            acceleration   momentum preservation
            
Water:      0.01	   0.999
Steam:      0.2            0.99999
Oil:        0.014          0.9

Factorio:   0.4            0.59

With my water acceleration, c^2 = 0.01, giving c = 0.1 but since we update 60 times per second, we get an effective speed of 6. So waves travel at 6 tiles/second. With (presumably) Factorio constants, we get c^2 = 0.4, giving c = 0.6325, and a speed of 38 tiles per second. Yes, that's right. Factorio already seems to have a 'reaction speed issue'. In sixty iterations, changes in supply and demand travel 'only' 38 tiles as it is. But no worries, now comes the 'not a problem' part.

(As an aside: considering a tile is roughly the size of a Factorio engineer, 6 tiles per second is pretty decent for a wave in a liquid)

Please be aware that this fixed speed of waves has nothing to do with throughput. The flow speed and thus throughput can be huge: the speed of the wave just models how long it takes for changes in flow speed to propagate through the system.
To be more precise: the solution to the wave equation, u(x,t) = F(x-c*t) + G(x+c*t), only tells us how much fluid is at a certain time and place, not in which direction the fluid is moving, or how fast. Even though a particular composition of the F and G may give the appearance of a wave travelling left, the actual liquid may flow to the right. The wave is just a manifestation of changes in pressure travelling through a liquid that itself may be moving in the opposite direction. Yet it's exactly these travelling changes in pressure that you mean by 'reaction speed'. The reaction speed however is not so much a result of the 60 UPS, but a direct result of the wave equation.

Now in the case you describe where the pipe is already full and a drain opens: this causes a 'negative' wave (caused by the sudden removal of fluid) to travel towards the source at exactly the speed calculated above (depending on physical constants). For my water constants, the 'gap' would reach the source after 10 seconds for a pipe of 60 tile length. Only then will the source be able to start replenishing the pipe. So yes, there is a reaction speed. And this is completely natural, and not a problem. There is no problem, because the 60 tile pipe acts as a reservoir to supply the drain until the source gets the message. The longer the pipe, the longer it takes the source to realise what's going on, but the bigger the reservoir will be as well.

When the drain is then suddenly closed, the incoming fluid will bounce against the end of the pipe, and send a wave back towards the source. The wave travelling backwards will effectively bring all incoming fluid to a halt as it passes it. The incoming flow plus the bounced wave will indeed rise the level temporarily to at most twice the normal level. When the reflected wave reaches the source, the source will of course stop supplying. When the system settles, the level may be slightly elevated above normal. For short pipes this can be up to twice the normal level, but for longer pipes the effect will be mitigated by friction.


Now I realise that this exact dependence of the speed of waves (or reaction speed as you call it) on a physical constant in the flow update step might be met with disbelief. In fact, it is true that as each tile responds to changes in its neighbours in the previous frame, some minute effect will travel at 60 tiles/second. But this is an approximation error! Were we to use an infinite amount of infinitely small pipe pieces and model at infinitely small time steps, this effect would vanish. Another (more obvious) approximation error is that even very large values for c^2 would not allow us to exceed a wave speed of 60 tiles/second.


I hope I succeeded in bringing this complicated matter across. If anything remains unclear, don't hesitate to ask!


TheYeast

PS. In relation to the above:

That's a real thing; If I remember correctly it's related to the speed of sound in the liquid. Looking at the nearest neighbours on the previous tick gives you a speed of 60 tiles per second. Google gives me real speeds of 1000-1500 m/s for common fluids however.

The speed of sound would indeed be what you would get if the pipe was completely filled at all times.
In Factorio, the level of fluid in pipes is allowed to rise and fall. This means that adding 25% extra to a pipe at 50% would only raise the pressure in so far that gravity is pulling the additional fluid at the top downwards. The small increase in pressure yields the small value for c^2 above, and thus a low wave speed.
In reality, the fluid content inside a fixed length of pipe hardly changes: there are no 'air holes' to replace drained fluid with a lighter gas. So a pipe is always at 100%. When a pump tries to force in an additional 1%, the only way this can happen is by compressing the fluid. And this causes the pressure to rise very sharply, which translates into a high value for c^2, and thus a high speed of the wave.

I'm sure a physicist will be along shortly to tell me how wrong I am.

Sorry about that

No really you were not wrong, but that speed of sound thing applies to the case where the pipes are always filled with no air holes. This may be a bit more realistic, but makes the pipes rather boring to watch in the game. Also, high values for c make simulation a bit more fickle, because there are always approximation errors because of the discrete pipe and time slices, and more extreme values tend to bring these out.
Also, it would bring the complication of having to model the transition of being empty to fully filled up. Not worth it in my opinion.

[Zavian]

So will these changes fix the fact that you practically need a pump literally every 3 pipes just to get enough fluid throughput to feed large fully beaconed setups? I'm currently designing a mega base sized oil processing factory and dealing with putting enough pumps in is just a nightmare. There's no way to avoid the massive amounts of pipe spaghetti.

Treat a large refinery just like a belt based smelter or any other belt based setup. Only so many machines can output to a belt, and you can only fit so many belts into a beaconed smelter array before you need to duplicate the smelter row.

In the same way you can build a refinery row that takes about 850 crude and 1100 water per sec and produces about 1130 petroleum. eg 12 refineries, 2.6 chem plant for heavy to light and 19 chem plant for light to petroleum (those numbers on the basis of 12 beacons per refinery, 8 beacons per chem plants) will take 850 crude, and 1113 water and produce 1133 petrol. If you need more refining, then duplicate the layout.

Such an layout won't need huge numbers of pumps. I'd probably use 1 pump for crude where the crude leaves the storage tank, 1 pump for light after the refinery and heavy to light crackers, 1 pump for petrol after the last refinery and another after the last light to petrol cracker. Even at 1130 petrol flow, that should be good for roughly 20 pipe segments between pumps, and with that layout I might use separate pipes for the petrol output from the refineries and the petrol output from the crackers which means each pipe will be good for over 100 pipe segments.

tl;dr Factorio is a game where you need to think through possible solutions to scaling problems, rather than just attempting to brute force increased throughput.

[Lubricus]

Hi Yeast,
thanks you for your proposal, so far it seems to be the most reasonable, although I am still unsure about the electricity model which also is interesting.
I admit that my knowledge of physics is rather limited, I had to rely on my common sense. About the “speed kinda models the pressure”, I suppose I could have put more thought into it when writing it. When I think about it, pressure describes it best, it's just that it developed from speed originally and is still called that and I never took the time to rethink what it is.
The model actually is not that dissimilar. The pipe segment has volume (same to level), and the pressure, and connections store the speed - which is just the last step flow. The speed only plays some role in the update and does not control it fully.
From what you propose I really like the idea to push towards only using the last state. I will have to think about it, might steal it. Right now I am thinking about allowing going over the pipe volume limit which would allow this and simplify things a lot.
Your equations are a simplified version of what I currently have. Unfortunately I don't think it would work very well in this form.
The main issue I see is a very low reaction speed. For example - have a straight pipe from source to drain, source is pumping, drain is closed, pipe is full. The drain opens. What I think your model does is that in first tick, only the last segments moves 1/4 of its fluid into the drain (assuming dt=1, which is about what we have) and all the rest stays in place. In next step, one more segments moves a bit, 1/16. Etc. You need "length of pipe" iterations for the wave to reach the source and quite many more to get speed. And then when you get the momentum and you close the pipe, how does the flow stop?
Even so, your ideas are inspiring, I think I will use them, if you don't mind
Thank you, D.

Thank you for your reply Dominik.

First of all, it would not be stealing. As I said, i wrote my simulation for the purpose of improving Factorio, and I even gave a short demonstration at your office when I visited Prague last year, but back then fluid realism wasn't a top priority Anyway I would be disappointed if you didn't consider using it.

Now about the reaction speed issue. I'm absolutely sure that there is a delay in reaction, and I'm equally sure that this is not a problem. In the situation you describe it indeed takes some time before the source 'becomes aware' that something is draining the pipe. In fact, this awareness will travel even slower than 60 tiles per second!

The general solution for the one-dimensional wave equation is

Code: Select all

u(x,t) = F(x-c*t) + G(x+c*t).

Here u(x,t) is the amount of fluid at time t and position x. F and G can be any two continuous functions. They represent two waves of arbitrary shapes travelling to the right (F) and travelling to the left (G). Their shape is dependant on the circumstances (sources/drains) but because of their arguments ( x+c*t and x-c*t ) they travel exactly at speed c and -c. And c simply depends on the flow update step:

Code: Select all

connection.flow += (connection.pipe0.level - connection.pipe1.level) * c^2;

(This can easily be derived from the wave equation)

As I discussed in response to BlueTemplar, I was using more realistic physical constants than Factorio is:

Code: Select all

            acceleration   momentum preservation
            
Water:      0.01	   0.999
Steam:      0.2            0.99999
Oil:        0.014          0.9

Factorio:   0.4            0.59

With my water acceleration, c^2 = 0.01, giving c = 0.1 but since we update 60 times per second, we get an effective speed of 6. So waves travel at 6 tiles/second. With (presumably) Factorio constants, we get c^2 = 0.4, giving c = 0.6325, and a speed of 38 tiles per second. Yes, that's right. Factorio already seems to have a 'reaction speed issue'. In sixty iterations, changes in supply and demand travel 'only' 38 tiles as it is. But no worries, now comes the 'not a problem' part.

(As an aside: considering a tile is roughly the size of a Factorio engineer, 6 tiles per second is pretty decent for a wave in a liquid)

Please be aware that this fixed speed of waves has nothing to do with throughput. The flow speed and thus throughput can be huge: the speed of the wave just models how long it takes for changes in flow speed to propagate through the system.
To be more precise: the solution to the wave equation, u(x,t) = F(x-c*t) + G(x+c*t), only tells us how much fluid is at a certain time and place, not in which direction the fluid is moving, or how fast. Even though a particular composition of the F and G may give the appearance of a wave travelling left, the actual liquid may flow to the right. The wave is just a manifestation of changes in pressure travelling through a liquid that itself may be moving in the opposite direction. Yet it's exactly these travelling changes in pressure that you mean by 'reaction speed'. The reaction speed however is not so much a result of the 60 UPS, but a direct result of the wave equation.

Now in the case you describe where the pipe is already full and a drain opens: this causes a 'negative' wave (caused by the sudden removal of fluid) to travel towards the source at exactly the speed calculated above (depending on physical constants). For my water constants, the 'gap' would reach the source after 10 seconds for a pipe of 60 tile length. Only then will the source be able to start replenishing the pipe. So yes, there is a reaction speed. And this is completely natural, and not a problem. There is no problem, because the 60 tile pipe acts as a reservoir to supply the drain until the source gets the message. The longer the pipe, the longer it takes the source to realise what's going on, but the bigger the reservoir will be as well.

When the drain is then suddenly closed, the incoming fluid will bounce against the end of the pipe, and send a wave back towards the source. The wave travelling backwards will effectively bring all incoming fluid to a halt as it passes it. The incoming flow plus the bounced wave will indeed rise the level temporarily to at most twice the normal level. When the reflected wave reaches the source, the source will of course stop supplying. When the system settles, the level may be slightly elevated above normal. For short pipes this can be up to twice the normal level, but for longer pipes the effect will be mitigated by friction.


Now I realise that this exact dependence of the speed of waves (or reaction speed as you call it) on a physical constant in the flow update step might be met with disbelief. In fact, it is true that as each tile responds to changes in its neighbours in the previous frame, some minute effect will travel at 60 tiles/second. But this is an approximation error! Were we to use an infinite amount of infinitely small pipe pieces and model at infinitely small time steps, this effect would vanish. Another (more obvious) approximation error is that even very large values for c^2 would not allow us to exceed a wave speed of 60 tiles/second.


I hope I succeeded in bringing this complicated matter across. If anything remains unclear, don't hesitate to ask!


TheYeast

PS. In relation to the above:

That's a real thing; If I remember correctly it's related to the speed of sound in the liquid. Looking at the nearest neighbours on the previous tick gives you a speed of 60 tiles per second. Google gives me real speeds of 1000-1500 m/s for common fluids however.

The speed of sound would indeed be what you would get if the pipe was completely filled at all times.
In Factorio, the level of fluid in pipes is allowed to rise and fall. This means that adding 25% extra to a pipe at 50% would only raise the pressure in so far that gravity is pulling the additional fluid at the top downwards. The small increase in pressure yields the small value for c^2 above, and thus a low wave speed.
In reality, the fluid content inside a fixed length of pipe hardly changes: there are no 'air holes' to replace drained fluid with a lighter gas. So a pipe is always at 100%. When a pump tries to force in an additional 1%, the only way this can happen is by compressing the fluid. And this causes the pressure to rise very sharply, which translates into a high value for c^2, and thus a high speed of the wave.

I'm sure a physicist will be along shortly to tell me how wrong I am.

Sorry about that

No really you were not wrong, but that speed of sound thing applies to the case where the pipes are always filled with no air holes. This may be a bit more realistic, but makes the pipes rather boring to watch in the game. Also, high values for c make simulation a bit more fickle, because there are always approximation errors because of the discrete pipe and time slices, and more extreme values tend to bring these out.
Also, it would bring the complication of having to model the transition of being empty to fully filled up. Not worth it in my opinion.

If there only is flow between pressure differences and pressure = c*fluid_level.
Could it then be flow between full pipes?
And wouldn't pressure travel at the speed of the liquid?
And then the throughput be to limiting compared to the amount of fluid stored in the pipe?
Reaction times as on belts would be neet but the throuput should be to small compared to all the fluid stored in the pipes .

Then i think some sort of electricity model could be better where we always count the pipes as full and fluids "teleport" from source to drains with constraints governed by how the pipe network is built. It would more simulate how pressure differences in the pipes travel almost instant instead of with the fluid.

[mrvn]

Currently pumps only have an on/off switch.

Would it be possible to instead give them a throughput signal? So when they receive V=100 then the pump will pump 100 units of water per second.

[Dominik]

Hi Yeast,
thanks you for your proposal, so far it seems to be the most reasonable, although I am still unsure about the electricity model which also is interesting.
I admit that my knowledge of physics is rather limited, I had to rely on my common sense. About the “speed kinda models the pressure”, I suppose I could have put more thought into it when writing it. When I think about it, pressure describes it best, it's just that it developed from speed originally and is still called that and I never took the time to rethink what it is.
The model actually is not that dissimilar. The pipe segment has volume (same to level), and the pressure, and connections store the speed - which is just the last step flow. The speed only plays some role in the update and does not control it fully.
From what you propose I really like the idea to push towards only using the last state. I will have to think about it, might steal it. Right now I am thinking about allowing going over the pipe volume limit which would allow this and simplify things a lot.
Your equations are a simplified version of what I currently have. Unfortunately I don't think it would work very well in this form.
The main issue I see is a very low reaction speed. For example - have a straight pipe from source to drain, source is pumping, drain is closed, pipe is full. The drain opens. What I think your model does is that in first tick, only the last segments moves 1/4 of its fluid into the drain (assuming dt=1, which is about what we have) and all the rest stays in place. In next step, one more segments moves a bit, 1/16. Etc. You need "length of pipe" iterations for the wave to reach the source and quite many more to get speed. And then when you get the momentum and you close the pipe, how does the flow stop?
Even so, your ideas are inspiring, I think I will use them, if you don't mind
Thank you, D.

Thank you for your reply Dominik.

First of all, it would not be stealing. As I said, i wrote my simulation for the purpose of improving Factorio, and I even gave a short demonstration at your office when I visited Prague last year, but back then fluid realism wasn't a top priority Anyway I would be disappointed if you didn't consider using it.

Now about the reaction speed issue. I'm absolutely sure that there is a delay in reaction, and I'm equally sure that this is not a problem. In the situation you describe it indeed takes some time before the source 'becomes aware' that something is draining the pipe. In fact, this awareness will travel even slower than 60 tiles per second!

The general solution for the one-dimensional wave equation is

Code: Select all

u(x,t) = F(x-c*t) + G(x+c*t).

Here u(x,t) is the amount of fluid at time t and position x. F and G can be any two continuous functions. They represent two waves of arbitrary shapes travelling to the right (F) and travelling to the left (G). Their shape is dependant on the circumstances (sources/drains) but because of their arguments ( x+c*t and x-c*t ) they travel exactly at speed c and -c. And c simply depends on the flow update step:

Code: Select all

connection.flow += (connection.pipe0.level - connection.pipe1.level) * c^2;

(This can easily be derived from the wave equation)

As I discussed in response to BlueTemplar, I was using more realistic physical constants than Factorio is:

Code: Select all

            acceleration   momentum preservation
            
Water:      0.01	   0.999
Steam:      0.2            0.99999
Oil:        0.014          0.9

Factorio:   0.4            0.59

With my water acceleration, c^2 = 0.01, giving c = 0.1 but since we update 60 times per second, we get an effective speed of 6. So waves travel at 6 tiles/second. With (presumably) Factorio constants, we get c^2 = 0.4, giving c = 0.6325, and a speed of 38 tiles per second. Yes, that's right. Factorio already seems to have a 'reaction speed issue'. In sixty iterations, changes in supply and demand travel 'only' 38 tiles as it is. But no worries, now comes the 'not a problem' part.

(As an aside: considering a tile is roughly the size of a Factorio engineer, 6 tiles per second is pretty decent for a wave in a liquid)

Please be aware that this fixed speed of waves has nothing to do with throughput. The flow speed and thus throughput can be huge: the speed of the wave just models how long it takes for changes in flow speed to propagate through the system.
To be more precise: the solution to the wave equation, u(x,t) = F(x-c*t) + G(x+c*t), only tells us how much fluid is at a certain time and place, not in which direction the fluid is moving, or how fast. Even though a particular composition of the F and G may give the appearance of a wave travelling left, the actual liquid may flow to the right. The wave is just a manifestation of changes in pressure travelling through a liquid that itself may be moving in the opposite direction. Yet it's exactly these travelling changes in pressure that you mean by 'reaction speed'. The reaction speed however is not so much a result of the 60 UPS, but a direct result of the wave equation.

Now in the case you describe where the pipe is already full and a drain opens: this causes a 'negative' wave (caused by the sudden removal of fluid) to travel towards the source at exactly the speed calculated above (depending on physical constants). For my water constants, the 'gap' would reach the source after 10 seconds for a pipe of 60 tile length. Only then will the source be able to start replenishing the pipe. So yes, there is a reaction speed. And this is completely natural, and not a problem. There is no problem, because the 60 tile pipe acts as a reservoir to supply the drain until the source gets the message. The longer the pipe, the longer it takes the source to realise what's going on, but the bigger the reservoir will be as well.

When the drain is then suddenly closed, the incoming fluid will bounce against the end of the pipe, and send a wave back towards the source. The wave travelling backwards will effectively bring all incoming fluid to a halt as it passes it. The incoming flow plus the bounced wave will indeed rise the level temporarily to at most twice the normal level. When the reflected wave reaches the source, the source will of course stop supplying. When the system settles, the level may be slightly elevated above normal. For short pipes this can be up to twice the normal level, but for longer pipes the effect will be mitigated by friction.


Now I realise that this exact dependence of the speed of waves (or reaction speed as you call it) on a physical constant in the flow update step might be met with disbelief. In fact, it is true that as each tile responds to changes in its neighbours in the previous frame, some minute effect will travel at 60 tiles/second. But this is an approximation error! Were we to use an infinite amount of infinitely small pipe pieces and model at infinitely small time steps, this effect would vanish. Another (more obvious) approximation error is that even very large values for c^2 would not allow us to exceed a wave speed of 60 tiles/second.


I hope I succeeded in bringing this complicated matter across. If anything remains unclear, don't hesitate to ask!


TheYeast

PS. In relation to the above:

That's a real thing; If I remember correctly it's related to the speed of sound in the liquid. Looking at the nearest neighbours on the previous tick gives you a speed of 60 tiles per second. Google gives me real speeds of 1000-1500 m/s for common fluids however.

The speed of sound would indeed be what you would get if the pipe was completely filled at all times.
In Factorio, the level of fluid in pipes is allowed to rise and fall. This means that adding 25% extra to a pipe at 50% would only raise the pressure in so far that gravity is pulling the additional fluid at the top downwards. The small increase in pressure yields the small value for c^2 above, and thus a low wave speed.
In reality, the fluid content inside a fixed length of pipe hardly changes: there are no 'air holes' to replace drained fluid with a lighter gas. So a pipe is always at 100%. When a pump tries to force in an additional 1%, the only way this can happen is by compressing the fluid. And this causes the pressure to rise very sharply, which translates into a high value for c^2, and thus a high speed of the wave.

I'm sure a physicist will be along shortly to tell me how wrong I am.

Sorry about that

No really you were not wrong, but that speed of sound thing applies to the case where the pipes are always filled with no air holes. This may be a bit more realistic, but makes the pipes rather boring to watch in the game. Also, high values for c make simulation a bit more fickle, because there are always approximation errors because of the discrete pipe and time slices, and more extreme values tend to bring these out.
Also, it would bring the complication of having to model the transition of being empty to fully filled up. Not worth it in my opinion.

The theory is nice, but the theory is not what your algorithm in fact does. Big C^2 does not fix it if you limit the flow by 0.25.
What I see is that your flow updates incrementally by volume difference -> 1st iter. last segment moves 1/4 to the drain; 2nd iter last segment moves some more and second to last moves 1/16... the negative wave exponentially losing strength. So the pipe will be emptying very slowly into the drain, not really capable to satisfy a large demand, and due to the exponential it would take about infinite number of iteration to get a flow of some 1/2 per tick from a source not even very far away. Changing the constants do not change much about it.

[TheYeast]

If there only is flow between pressure differences and pressure = c*fluid_level.
Could it then be flow between full pipes?

No, the pressure difference increases or decreases the flow. In a full pipe all the fluid could be stationary, or move really fast, or anything in between.

And wouldn't pressure travel at the speed of the liquid?

Certainly not. Suppose a river streams to the left. Then you close a flood gate so that there will only be a narrow opening. The pressure buildup would have to slow the river down, eventually over it's entire length. How could the pressure reach upstream to slow it down if it were to travel only to the left at the speed of the river?

And then the throughput be to limiting compared to the amount of fluid stored in the pipe?

No because of the previous answer. I will make a post about flow rate explaining this in detail.

[TheYeast]

The theory is nice, but the theory is not what your algorithm in fact does. Big C^2 does not fix it if you limit the flow by 0.25.
What I see is that your flow updates incrementally by volume difference -> 1st iter. last segment moves 1/4 to the drain; 2nd iter last segment moves some more and second to last moves 1/16... the negative wave exponentially losing strength. So the pipe will be emptying very slowly into the drain, not really capable to satisfy a large demand, and due to the exponential it would take about infinite number of iteration to get a flow of some 1/2 per tick from a source not even very far away. Changing the constants do not change much about it.

I am pretty sure my algorithm does exactly what the theory says. I used the same formulas for solving the wave equation as I used in my algorithm. Besides, I just ran my simulator again and can confirm it does exactly what I said.

I understand what you are saying about every iteration losing strength, but you forget that if the last segment would lose 25% of its content at each iteration, it would be nearly empty after only a few iterations. Then the same exchange starts between it and the fore last, and so on. So the wave does not lose strength at all (ignoring small friction for now).
In case this isn't clear: once a pressure difference has set the fluid in motion, it will continue to accelerate until the destination has the same level of fluid; It will maintain its velocity as long as the levels are equal; It will only start slowing down once the destination achieves a higher fluid level, and it may take some time to stop.
Therefore, as long as the sink keeps its level at zero, fluid will rush in.

As a recap of the algorithm:

Pass one:
Flow speeds are increased/decreased in with respect to differences in fluid levels:

Code: Select all

flow += (pipe0.level - pipe1.level) * c^2

Therefore, previous flow is maintained ( += ) even if the adjacent pipes are now at equal level! Flow will not be set back to zero at each pass!

Flow speed is slightly decreased to simulate friction:

Code: Select all

flow *= 1 - r;

Since r is very close to zero, this can mostly be ignored when reasoning about the system.
Now apply the sanity limit:

Code: Select all

flow = limit ( flow, pipe0, pipe1 );

Where the limit function limits the flow to 25% of the level in the source (depends on flow sign).

Pass two:
Update content of pipes:

Code: Select all

	pipe0.level -= flow; 
	pipe1.level += flow;

Because throughput seems to be a concern, I will do a separate post about flow rate.

[mrvn]

How will this play with circuit controlled pumps. E.g. say a pump is on every other tick.

Given a long enough pipe would that create a flow at the source of half the pump speed or would the shock waves propagate all the way to the source causing an on/off pattern there too?

[Lastmerlin]

Excellent FFF! As usual, the developer-player interaction of factorio is outstanding. Its also nice to see, that the pipes (as one of the most annoying aspects of the game so far) are in the focus now.

Concerning the issue itself, I will not claim to have the perfect solution (and there are already so many proposals floating around). But I would to give my personal preference, which targets of the rewrite are really important.

Fairness/Logic of the system: Perfect fairness is not even soo important in my opinion. What is important is a logical and coherent behaviour. Right now, a large pipe system is a nightmare to debug. Sometimes parallel paths reduce throughput, sometime some paths get huge preference, almost identical layouts create very different results. Factorio is all about finding and fixing flaws, so any system should give the player the information and tools to do exactly this. I believe your current plan is already a very good direction. Any algorithm, that uses global data (like electric current models) could create locally strange and unintuitive results. What the player sees is always local, so I believe the algorithm should operate on local data as well.

Realism: The nuclear option is to simple in my opinion. However, to much realism could by annoying as well. Example: Braess paradoxon (https://en.wikipedia.org/wiki/Braess%27s_paradox) - I believe its no benefit if you can model something like this, its just frustrating. Again, I like the current idea.

Throughput: I hope for two things: a) That the throughput drop over distance gets reduced a lot. It is so annoying, that one pipe is so very much better than two pipes. This leads to puzzles like *how to reduce this pipe section from three to two pipes and how can I fit in a pump every two pipe segments*. In general, the current excessive use of pumps is a clear sign that something is wrong. b) Possibility for mods to scale throughput. With the popular Bobs/Angels mods, one pipe can not even support a single modded machine (and I am talking about not-too-crazy speed boni, like 700%). It seems, that there is currently no way, to give the player better pipes. While everything else scales a lot, this leaves the pipes woefully inadequate. In fact, some different pipe types and scalability could be good idea for vanilla as well.

Handling: I fully support any plans, to make the fluid contamination less of a *ragequit and reload* scenario. Other than that: I strongly dislike any plans that introduce a big variety of pipes (tee junction and ellbow and whatever) to clutter up your inventory.

[mrvn]

Fairness/Logic of the system: Perfect fairness is not even soo important in my opinion. What is important is a logical and coherent behaviour. Right now, a large pipe system is a nightmare to debug. Sometimes parallel paths reduce throughput, sometime some paths get huge preference, almost identical layouts create very different results. Factorio is all about finding and fixing flaws, so any system should give the player the information and tools to do exactly this. I believe your current plan is already a very good direction. Any algorithm, that uses global data (like electric current models) could create locally strange and unintuitive results. What the player sees is always local, so I believe the algorithm should operate on local data as well.

I think this problem comes simply from the total design flaw of computing everything in a single step. Every pipe takes it's state, computes the next state and overwrites its own state with it. The order in which pipes are evaluates totally changes the outcome. So not only does fluid in a fork flow unfair. Taking a blueprint and building it somewhere else can totally change the flow.

The allocation/reservation step should avoid this.

[Zavian]

Sometimes parallel paths reduce throughput, sometime some paths get huge preference, almost identical layouts create very different results. Factorio is all about finding and fixing flaws, so any system should give the player the information and tools to do exactly this.

The simplest and most obvious way to simplify debugging complex pipe networks is to not build complex pipe networks with multiple. Don't build loops, don't try to increase throughput by using duplicated pipes that cross-connect somewhere, don't feed assemblers from both ends. Build a network where there is always exactly one path from each source to each sink. Then all you need to worry about is making sure that supply equals or exceeds demand, that you aren't trying to move too much fluid through any section of pipe (better data on fluid flow rate in a pipe could be useful) and that all recipes with multiple outputs aren't going to backup. (Note that only one of those 3 is actually a pipe issue).

If you need more throughput than one pipe can provide, then duplicate the design the same way you would for a row of smelters.

Edit:

I think this problem comes simply from the total design flaw of computing everything in a single step. Every pipe takes it's state, computes the next state and overwrites its own state with it. The order in which pipes are evaluates totally changes the outcome. So not only does fluid in a fork flow unfair. Taking a blueprint and building it somewhere else can totally change the flow.

You can also avoid this by just making sure that supply exceeds demand. When that happens pipes backup, and the backpressure ensures that every branch gets as much fluid as it will take.

[mrvn]

If you need more throughput than one pipe can provide, then duplicate the design the same way you would for a row of smelters.

I needed more throughput than one pipe can provide for the ONE fluid output of ONE washing plant (Angels + Bobs mod) with beacons mk3. Had to add the high pressure pipe mod to get bigger pipes to make it work at full beaconed speed. Vanilla pipes using pipe/pump/pipe/pump/pipe pattern couldn't handle the throughput.

[Zavian]

If you need more throughput than one pipe can provide, then duplicate the design the same way you would for a row of smelters.

I needed more throughput than one pipe can provide for the ONE fluid output of ONE washing plant (Angels + Bobs mod) with beacons mk3. Had to add the high pressure pipe mod to get bigger pipes to make it work at full beaconed speed. Vanilla pipes using pipe/pump/pipe/pump/pipe pattern couldn't handle the throughput.

That is not a vanilla problem, but a mod issue. Pump->tank->pump->tank has higher throughput than a vanilla pipe, if you want to use that. Another way is to pump into a tank, then pump out of the tank's 3 other outputs. If all three need to go to the same destination you can pump them into a common tank at the destination. The fact that you are using pumps makes sure that fluid can't flow in a loop, so you always know which way the fluid is trying to flow when calculating throughput.

There is also the alternative of simply not adding a full set of beacons, if the output is going to exceed the flow capacity of the pipe.

[just_dont]

I really hope that TheYeast ideas will make it into the game in some way - they seem to be really well-thought and from my (admittedly very simple) attempts to simulate it in various scenarios, I can say that nothing about it feels "unintuitive" with regards to how the liquid moves around pipes.

[Engimage]

I do greatly support TheYeast's concept. Not only it is reasonable from CPU standpoint but also this has a great educational sense.

Also this leads to a point of making hard cap on a pipe absolete replacing it with say volume and friction giving the throughput a soft cap like it would in real life. If a player can design a system which will push the limits over common values it would really be a cool challenge.

This also leads to a pump redesign as normally pumps have both limited pressure and throughput so overall it would become pretty fun designing effective setups.

With this new design a flow becomes measurable value within a pipe opening up circuit connection possibility to just any pipe which is, again, pretty cool!

[pleegwat]

I'm sure a physicist will be along shortly to tell me how wrong I am.

Sorry about that

No really you were not wrong, but that speed of sound thing applies to the case where the pipes are always filled with no air holes. This may be a bit more realistic, but makes the pipes rather boring to watch in the game. Also, high values for c make simulation a bit more fickle, because there are always approximation errors because of the discrete pipe and time slices, and more extreme values tend to bring these out.
Also, it would bring the complication of having to model the transition of being empty to fully filled up. Not worth it in my opinion.

No worries. So basically, this is modelling the pipes like open-air channels (such as the Roman aqueducts), instead of like a pressurised pipe system (such as the modern water mains). And we ignore that underground pipes only work for a pressurized system.

And now I'm thinking that may be an interesting graphical direction as well.

[silkroadalice]

As a recap of the algorithm:

Pass one:
Flow speeds are increased/decreased in with respect to differences in fluid levels:

Code: Select all

flow += (pipe0.level - pipe1.level) * c^2

Therefore, previous flow is maintained ( += ) even if the adjacent pipes are now at equal level! Flow will not be set back to zero at each pass!

Flow speed is slightly decreased to simulate friction:

Code: Select all

flow *= 1 - r;

Since r is very close to zero, this can mostly be ignored when reasoning about the system.
Now apply the sanity limit:

Code: Select all

flow = limit ( flow, pipe0, pipe1 );

Where the limit function limits the flow to 25% of the level in the source (depends on flow sign).

Pass two:
Update content of pipes:

Code: Select all

	pipe0.level -= flow; 
	pipe1.level += flow;

Because throughput seems to be a concern, I will do a separate post about flow rate.

Nice model! I have an idea to optimize it, but don't know if it's feasible.

The model you mentioned could be based on a structure like this:

Code: Select all

    T1-j-p-j-..-j-p-j-T2

T1 & T2 stands for storage tanks , pumps, pipes or whatever; j stands for junction in your model; and p for pipe. T, j and p update in every iter, the time cost will be proportional to the number of j(or p)s connecting the Ts. Such a model, let's call it the distributed elememt model.

A lumped element model should be based on a structure like:

Code: Select all

    T1-j1-S-j2-T2

S stands for the segment of straight pipe between T1 & T2. According to the FFF, treating the straight pipe as a single segment would improve the perfomance.The pressure change on T1 propagates through S to T2, and increases or decreases the flow in j2 with a time delay
D = segmentLength/soundSpeed, and so does T2 to T1. The time delay can be simulated by using a circular queue, that makes the time cost of updating a segment being constant, compared with it is proportional to the number of j(or p)s in the distributed elememt model.

My question is: Is it possible to transform the distributed elememt model to a lumped element model? If trnasforming is possible, would the dispersion in the continous model have something to do with D in the discrete model here?

[TyrHeimdal]

Looks great, and as long as it fixes circuit network "errors" where tanks read as 0, but still has fluid in them... I'm good!

[TheYeast]

Because throughput seems to be a concern, I will do a separate post about flow rate.

And here it is


Using once again:

Code: Select all

connection.flow += (connection.pipe0.level - connection.pipe1.level) * c^2;

or for notatinal convenience:

Code: Select all

f += (p0-p1) * c^2;

(I'll keep the c^2 form, where c is the speed of waves travelling through the fluid)


In the absence of friction, the throughput is (theoretically) unbounded. Any difference in fluid level will be counteracted by acceleration until the difference is gone. As long as the flow cannot replenish a drain fast enough, it will speed up until it can. (In the simulation it will not be unbounded, because of the discrete time slices, but let's ignore the edge cases)

When adding the friction step we get:

Code: Select all

connection.flow *= (1.0 – liquid.friction);

for short:

Code: Select all

f *= 1 - r;
or
f = f * (1-r);
or
f = f - r*f;

with 0 < r << 1 the friction.

At every update, the flow is thus reduced by r*f. Therefore, to prevent the flow from slowing down, this must be counteracted by a level differential that adds exactly r*f back to the flow speed.

So when the flow is adjusted in the first pass, we want this step

Code: Select all

f += (p0-p1) * c^2;

to have (p0-p1) * c^2 equal to r*f, adding back the friction loss.
Let's use d to denote p0-p1.
We then rewrite our condition like this:

Code: Select all

(p0-p1) * c^2 == r*f
d * c^2 == r*f   	(substitute d)
f == d * c^2 / r  	(move f to the left)

This is the equilibrium flow rate between two segments with fluid level difference d, where d exactly overcomes friction.

Suppose we want to know the maximal flow speed through a pipe of n segments. The highest flow speed would be obtained by keeping the source at max level (say 100) and the drain at 0. The level in segments in between will drop linearly, such that each step has the same difference d (as each one needs to overcome the same friction). This distribution maximises each step d. However, our simulation is imperfect, and we take at most 25% from the current content in a pipe to ensure it doesn't go below zero. Therefore, the last segment with the lowest level must maintain a level of 4*f, to allow us to take out one f each frame. We end up with a level linearly declining from 100 to 4*f. This will result in the highest flow rate f that can be achieved.

So d is simply the entire decline divided by the number of steps down: d = (100-4*f) / n.

Substituting in the condition derived above, we get:

Code: Select all

f == d * c^2 / r			(first condition)
d == (100-4*f) / n			(second condition)
f == 100 * c^2 / ( n*r + 4*c^2 )	(solving for f)

Multiply by 60 UPS to get the Flow Per Second:

Code: Select all

fps == 6000 * c^2 / ( n*r + 4*c^2 )

Note the theoretical maximum of fps = 1500 for n = 0, caused by the 25% outflow sanity limit.

So with the values I used in my toy simulation, c^2=0.01 and r=0.001 and n=10, we get fps = 1200. I can confirm this exactly matches what is being simulated when I run my model.

However, when substituting the values that I read in a post above for current Factorio physics, c^2=0.4 and r=0.41, I calculate no more than fps = 58 for a 100 segment pipe. This is nowhere near what I can pump through such a pipe, so I now distrust these values, or there must be something else radically different in the current implementation. I could not ascribe it to an update ordering artifect.

Anyway, the formula above allows you to pick a friction r that for a given reasonable length of n pipe segments provides the desired flow rate limit (before pumps are needed to reinforce the flow).

The formula also allows you to pick a replacement friction r2 when merging x segments (without junctions) into one (optimisation). Calculate a new r2 for the merged segment that results in the same max flow rate with n=1 as the true friction r does for all x separate pieces combined.

So in summary, c will be the rate at which supply/demand induced pressure differences travel through pipes as visible waves of increased fluid level.
Larger values for friction r will dampen these waves over time, and will limit the flow rate.
To get a higher flow rate, c^2 could be raised, but to keep wave speed esthetically pleasing my first choice would be to lower the friction r.


That concludes the flow rate calculation.


I have one further remark. In a post above I noticed these different pressure min/max for different components (pipe 0/100, pump 0/200, boiler -100/100).

When, as i suggest, the max level of fluid in a pipe is no longer enforced, having a pump go up to 200 is obviously no longer right. But the boiler going down to -100 is quite pointless as well in my opinion: as I calculated above, the flow rate is dependent on the small level differences between adjacent pipe segments over the entire length of n segments. Having an extra artificial difference of 100 at a source or sink will not speed up the intermediate flow. I see no reasonable purpose for this at all. The only possible reason to want this could be to have a sink collect a reserve for when the next production cycle begins, but you then might as well take out what enters the sink and lock that reserve in a different variable until satisfied. So as you are trying to simplify and optimise fluids, consider taking this out entirely. Or instead I would be very interested to hear what I am missing here

Only one more post coming up!