Friday Facts #82 - Optimisations

[Martc]

The original idea here was to simulate different chunks in different threads. However this is quite tricky to do properly - namely because overlaps and special cases. Another approach we have been thinking about recently is to do some "thematic" parallelization. Things like electric energy propagation, path finder, rail finding.

I think you can simulate factories that aren't directly connected (only with trains or robots) in different threads. But I know, finding "not connected" factories can be quite challenging, because wagons can be used as storage box.


And I have question about Steam integration, do you plan support Steam Workshop for mods? And maybe for blueprint sharing?

[McRib]

The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.

You mean version 12.01 or you mean a stable version 12.xx?

[FishSandwich]

The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.

You mean version 12.01 or you mean a stable version 12.xx?

0.12.0 will be the first version. Stable 0.12 is a lot longer away(4-5+ months?)

[kovarex]

The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.

You mean version 12.01 or you mean a stable version 12.xx?

12.0, after the experience we have, we can't guess the stable ....

[ssilk]

I guess for you ( was nearly right the last time ): Stable 0.12 in September.

<crossing thumbs>

[Wolfwaffe]

Steam integration
This is not technically part of 0.12 as we plan to go to the steam when the 0.12 is stable, but this time we decided to postpone it no more.

Finally! Been waiting for this forever. One question though, will there be byers tiers like game has right now (belt repair man and others). Kinda wanted to get highest tier, 'cause i like the game, but i'm not really fond of messing around with paypal >>

[cpy]

There is no such thing as stable multiplayer
Especially with mods, and i don't play without them.

[roy7]

Yes. The goal is to make it playable even with 1s latency. However the latency hiding won't be perfect. This involves some dirty internal tricks, therefore we decided to go for most common actions. Things like moving, building, mining, etc. They will be (unless there is a collision) instantaneous. However in non-latency hidden actions the lag will still be felt. More on this will be in the fff.

That sounds fantastic. It's the basic actions like moving around and chopping down a tree that make a slow connection feel laggy. With those being instant, it'll feel snappy regardless of what the lag is. Very happy to hear about this.

[ssilk]

In the FFF the whole crew is mentioned.
The whole?
No!
A little hedgehog is missing! See https://www.factorio.com/team

So, what has happened to the hedgehog?

[FishSandwich]

So, what has happened to the hedgehog?

He got squashed trying to cross the road..

Why? He wanted to show his girlfriend he had guts..

[slpwnd]

In the FFF the whole crew is mentioned.
The whole?
No!
A little hedgehog is missing! See https://www.factorio.com/team

So, what has happened to the hedgehog?

That webpage needs some serious update=) But the hedgehog is still with us! Well originally there were three. Now only one is left in the office - the small one and he is sitting on my monitor watching me if I do the job well=)

[bulldog98]

I wonder if main thread threading is planned, because you could build 4x ish bigger factories on quadcore cpu.

The only thing that comes into my mind for this is the following:

[slpwnd]

I wonder if main thread threading is planned, because you could build 4x ish bigger factories on quadcore cpu.

The only thing that comes into my mind for this is the following:

Brilliant

[cpy]

Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.

[orzelek]

Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.

Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.

[kovarex]

Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.

Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.

More than tricky

[ratchetfreak]

Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.

Threading and determinism don't go hand in hand.

Allowing parallel loops would be nice though, and it would allow better parallelization. For example ticking all assemblers

Code: Select all

void tick(float powerfactor){//is the % of power available to allow slower progress
    if (inprogress){
        //update progress bars depending on power factor

        if(progress==100){
            //add result to output
        }
    }
    if(hasEnoughIngredients && outputNotFull){
        //remove the inputs

        inprogress = true;
    }
}

This is independent of all other assemblers and inserters and power shortage is handled through the powerfactor parameter after precalculating the power for all machines.

In the lua engine you could say, given this collection of entities run this code once for each of them. With the restriction that the function is only allowed to access that entity and read-only basic values. Adding a reduce (adding the results of the function together) on the results is a simple followup.

[kovarex]

In the lua engine you could say, given this collection of entities run this code once for each of them. With the restriction that the function is only allowed to access that entity and read-only basic values. Adding a reduce (adding the results of the function together) on the results is a simple followup.

This is quite abstract already even without talking about the details, and preparing the restriction specification would almost surely take more time than the time saved by the loop here. In lua it would be even more complicated, as the lua engine is not prepared for parallelism even remotely. And even if it would, the modders would make faulty parallelism code almost for sure, which would lead to huge amounts of desync reports that we would have to solve.

The result for me is, that custom parallelism written in C++ (by us) might happen in the future for specific tasks, but lua parallelism isn't really an option.

[ratchetfreak]

Depending the the setup that's used I can see something like the following for the parallel foreach:

Code: Select all

parallelFor(entitiesToProcess, 
            [TempEnginePool]()=> TempEnginePool->pop(),
            [&Staticinputs](Pair<Entity*, Vector<LuaCode> e, LuaTempEngine* le)=>{
                le->prepEngine(e.first, Staticinputs); 
                foreach(LuaCode code in e.second)
                    le->run(code);
               le->propagateChangesToEntity(e);
            }, 
            [TempEnginePool](LuaTempEngine* le)=>TempEnginePool->release(le));

where

• the first parameter is the collection of entities,
• the second is the per thread init to get a temporary lua engine from a pool,
• the thirds is the per entity code to set the engine's globals as needed and run the code and finally propagate the changes to the entity back
• the final one is per thread cleanup to release the engine back into the pool

This will need a global lock on the map though.

In lua code you can submit the list of entities and the fileID with the code to run (to reinforce that it's not executed in the same scope), this list may not contain doubles and optionally a reduce operation+callback. The code will then be appended to the vector paired with the entity in the entitiesToProcess list.

The loop will not be executed until all mods' events for this tick have run and the results of the parallel code will be propagated back through the callback the next tick.

The order in the LuaCode vector will be deterministic (based on order of execution of the mods, and the order in which they pass the code), the setup disallows propagation of changes to other entities.