Smart Train Deliveries with Combinator Magick [0.13/0.14]

[siggboy]

Since you're storing requester/provider as `amt of type`, what happens if more than one provider/requester have/want the same type of item? Do you simply ignore them until the first gets used and you poll them again?

I'm answering on behalf of him: as he has described it, he matches one pair of requester/provider per resource type and stores those. So the memory could have one valid coal/coal pair, one valid iron/iron pair etc. The trains are then send to the appropriate routes and the pairs are removed from the memory. When the coal/coal pair is being serviced, another coal/coal pair could enter the memory immediately, and so on.

Actually his system is very similar to mine, with the difference that he does have a buffer of several routes that can be given to trains immediately as they arrive. In my implementation I only look for a valid route when a train enters the station (i.e. on demand). This does introduce some minimal additional delay but is less complex.

Counting trains in some way is useful, but it should not be overengineered because the typical case amounts to simply locking a station while a train is on the way. It is very rare that you actually need to count several trains in many places, because a well designed system under full load will simply not have the buffers for this to even be possible (if you have 3 train loads of ore buffered then there is not a lot of demand, and all the train counting will not really improve performance anyway).

What you want to avoid are two situations:

• Trains piling up at a requester because as soon as the request comes in, every available provider is matched to the requester. This can be done by either locking the requester entirely (train count = 1 means requester is locked) or discounting inbound resources (either by counting trains if you always ship full loads or by counting incoming resources if you want to ship partial loads).
• Trains going to a provider that does not have a full load of goods (or less than the minimum amount that you're willing to ship). I think this case can usually be handled by simply locking the provider, ie. only allowing one train at a time to the provider, because it normally takes a while for the buffer at the provider to fill up again, and the train latency is not high enough that you actually need to account for that. So by the time the provider could again be polled it still does not have a full buffer anyway. If that makes sense. The situation where you actually profit from counting trains at the provider is when the outpost is so far away that the buffer will actually fill up during the time that it takes a train to get there -- so you would want a train on the way even before the buffer is full, to avoid wasting bandwidth. In my opinion that situation is too rare to warrant a complex solution.
  In the other case where stuff keeps piling up at the providers because there is no demand you really do not need to queue up trains at the providers, since there is no demand anyway.

[DemiPixel]

Since you're storing requester/provider as `amt of type`, what happens if more than one provider/requester have/want the same type of item? Do you simply ignore them until the first gets used and you poll them again?

I'm answering on behalf of him: as he has described it, he matches one pair of requester/provider per resource type and stores those. So the memory could have one valid coal/coal pair, one valid iron/iron pair etc. The trains are then send to the appropriate routes and the pairs are removed from the memory. When the coal/coal pair is being serviced, another coal/coal pair could enter the memory immediately, and so on.

Actually his system is very similar to mine, with the difference that he does have a buffer of several routes that can be given to trains immediately as they arrive. In my implementation I only look for a valid route when a train enters the station (i.e. on demand). This does introduce some minimal additional delay but is less complex.

Counting trains in some way is useful, but it should not be overengineered because the typical case amounts to simply locking a station while a train is on the way. It is very rare that you actually need to count several trains in many places, because a well designed system under full load will simply not have the buffers for this to even be possible (if you have 3 train loads of ore buffered then there is not a lot of demand, and all the train counting will not really improve performance anyway).

What you want to avoid are two situations:

• Trains piling up at a requester because as soon as the request comes in, every available provider is matched to the requester. This can be done by either locking the requester entirely (train count = 1 means requester is locked) or discounting inbound resources (either by counting trains if you always ship full loads or by counting incoming resources if you want to ship partial loads).
• Trains going to a provider that does not have a full load of goods (or less than the minimum amount that you're willing to ship). I think this case can usually be handled by simply locking the provider, ie. only allowing one train at a time to the provider, because it normally takes a while for the buffer at the provider to fill up again, and the train latency is not high enough that you actually need to account for that. So by the time the provider could again be polled it still does not have a full buffer anyway. If that makes sense. The situation where you actually profit from counting trains at the provider is when the outpost is so far away that the buffer will actually fill up during the time that it takes a train to get there -- so you would want a train on the way even before the buffer is full, to avoid wasting bandwidth. In my opinion that situation is too rare to warrant a complex solution.
  In the other case where stuff keeps piling up at the providers because there is no demand you really do not need to queue up trains at the providers, since there is no demand anyway.

Second one is pretty intense, I doubt I'd ever have anything like that

Still wondering how we're going to do this in 0.13. Obviously we don't have all the information, but it seems we have to disable train stations or something like that. If this is the case, we would have to tell the train station to allow the train to come in instead of telling the train to go to a specific station. Just a bit more of a headache!

[AccidentalChef]

I'm answering on behalf of him: as he has described it, he matches one pair of requester/provider per resource type and stores those. So the memory could have one valid coal/coal pair, one valid iron/iron pair etc. The trains are then send to the appropriate routes and the pairs are removed from the memory. When the coal/coal pair is being serviced, another coal/coal pair could enter the memory immediately, and so on

That's really close, but to clarify, I store the requests and the offers in main memory before a match is found. Immediately after the match is found, the pair is cleared out of memory and sent out over the network. Different stations deal with that information in different ways. At the main depot, a buffer stores the station numbers of the providers so that trains can be immediately dispatched. The oil barrel loading station does basically the same thing, but only for oil trains. Requesters and providers watch to see if they have trains coming to prevent the problems siggboy listed.

[siggboy]

Second one is pretty intense, I doubt I'd ever have anything like that

What do you mean, I didn't quite get that.

Still wondering how we're going to do this in 0.13. Obviously we don't have all the information, but it seems we have to disable train stations or something like that. If this is the case, we would have to tell the train station to allow the train to come in instead of telling the train to go to a specific station. Just a bit more of a headache!

It's not going to be possible without SmartTrains. The features announced for 0.13 do not allow a single train line with logical station numbers, but we need that for these kinds of systems. Disabling the train stations will only be useful for much simpler setups.

[DemiPixel]

Second one is pretty intense, I doubt I'd ever have anything like that

What do you mean, I didn't quite get that.

Still wondering how we're going to do this in 0.13. Obviously we don't have all the information, but it seems we have to disable train stations or something like that. If this is the case, we would have to tell the train station to allow the train to come in instead of telling the train to go to a specific station. Just a bit more of a headache!

It's not going to be possible without SmartTrains. The features announced for 0.13 do not allow a single train line with logical station numbers, but we need that for these kinds of systems. Disabling the train stations will only be useful for much simpler setups.

I'm sure XKnight will find some clever hack.

[siggboy]

I'm sure XKnight will find some clever hack.

XKnight probably finds this entire thing completely boring and could do it with 2 constant combinators in two hours

[kann_]

...Explanation...

Thanks, that was really well explained. Maybe the decentralization is the better way to do it. My current system works the same and although I send demand and supply numbers, the processor makes decisions on a train to train basis and anything above 1 is neglected anyway. Anyway, I will try to make it more simple by centralizing a few functions.

Could you also give a small description of the polling communication principle?

For my part: The multiplexing network connects all stations with one wire. A counter iterates through all station numbers in a predefined order. If a station see its own number on the line it answers two ticks later.

[siggboy]

I would like to add multi-station support for the depot circuit and I could use some suggestions on how to do it. I have a few ideas but maybe I've missed something obvious.

Assume that the signal for the train (destination) is in a register, and that we have to trigger an action when the train leaves. So the "train-at-station" signal needs to be visible from the station (and only the station) that is actually launching the train.

Not really sure what a stable solution is that doesn't use too many combinators. It should also work if more than one train arrives at a depot station simulatenously (ie. train-at-station goes to one for all of them during the same tick). It should not break even if the system is enabled when there are already trains at the station.

So the circuit should not rely on train arriving in succession.

[AccidentalChef]

...Explanation...

Thanks, that was really well explained. Maybe the decentralization is the better way to do it. My current system works the same and although I send demand and supply numbers, the processor makes decisions on a train to train basis and anything above 1 is neglected anyway. Anyway, I will try to make it more simple by centralizing a few functions.

Could you also give a small description of the polling communication principle?

For my part: The multiplexing network connects all stations with one wire. A counter iterates through all station numbers in a predefined order. If a station see its own number on the line it answers two ticks later.

I think decentralizing costs more combinators overall, but for me it's easier to design and test multiple small circuits and link them together than to design one bigger one that does everything. It would be really cool imo to have one monolithic circuit that ran everything and required very little to be done at each station but I'm not going to be the one to design it at this point.

In my system, all stations are connected to the cpu with the red wire. A counter iterates through the stations, and is shuffled by another circuit to make sure two stations of the same type aren't polled in a row. That shuffled number is pulsed out as a yellow signal to all stations. Any station whose number matches the pulse will reply if it either needs or can provide at least one full train load of material. Stations reply with their station number in the material they need or have, and the reply is positive for providers and negative for requesters. Stations that need a delivery of empty barrels add 1000 to their reply so the system knows to route them through the barrel loading station on their way to the outpost. Examples: Iron ore provider at station 37 replies with 37 iron ore if it can provide a full train. Steel requester at station 52 replies -52 steel if it needs at least a train load. Oil outpost 75 replies 1075 filled barrels if it needs more empty barrels.

[AccidentalChef]

I would like to add multi-station support for the depot circuit and I could use some suggestions on how to do it. I have a few ideas but maybe I've missed something obvious.

Assume that the signal for the train (destination) is in a register, and that we have to trigger an action when the train leaves. So the "train-at-station" signal needs to be visible from the station (and only the station) that is actually launching the train.

Not really sure what a stable solution is that doesn't use too many combinators. It should also work if more than one train arrives at a depot station simulatenously (ie. train-at-station goes to one for all of them during the same tick). It should not break even if the system is enabled when there are already trains at the station.

So the circuit should not rely on train arriving in succession.

Here's how I'm doing it now (ignoring extra combinators that deal with oil barrels)... Each lane of the depot has two combinators, and works with a unique signal. For this example, let's say lane 1 in the depot uses the "1" signal. One combinator holds the destination in memory. It's configured as "1" * "train at station" -> "1" and the output is connected to the input using the green wire. This clears the memory as soon as the train leaves. The other combinator is "train at station" > "1" (input from memory combinator) -> "1" (output to red wire). That way each lane shows as available on the red wire only if there's a train at the station but not a destination stored in memory yet.

In your system, you could start polling stations once any train is available, and while you're doing that you pick which of the available lanes is going to be sent out. I use a counter that checks each lane in order but is paused rather than reset in between trains. Once you know the destination and which lane you're sending it to, send a 1 tick pulse of the destination on that lane's unique signal down the green wire that hits all the memory cells and you're done.

[kann_]

In my system, all stations are connected to the cpu with the red wire. A counter iterates through the stations, and is shuffled by another circuit to make sure two stations of the same type aren't polled in a row. That shuffled number is pulsed out as a yellow signal to all stations. Any station whose number matches the pulse will reply if it either needs or can provide at least one full train load of material. Stations reply with their station number in the material they need or have, and the reply is positive for providers and negative for requesters. Stations that need a delivery of empty barrels add 1000 to their reply so the system knows to route them through the barrel loading station on their way to the outpost. Examples: Iron ore provider at station 37 replies with 37 iron ore if it can provide a full train. Steel requester at station 52 replies -52 steel if it needs at least a train load. Oil outpost 75 replies 1075 filled barrels if it needs more empty barrels.

Sounds like there is no difference to multiplexing except the shuffling of the order, right? I guess handling the tick delay is a little different with a random order.
I count trains to allow as many trains per supply station as resources are available. And I send trains to a fixed requester station.

[DemiPixel]

It's not going to be possible without SmartTrains. The features announced for 0.13 do not allow a single train line with logical station numbers, but we need that for these kinds of systems. Disabling the train stations will only be useful for much simpler setups.

Still true?

[siggboy]

Sounds like there is no difference to multiplexing except the shuffling of the order, right?

We're down to how we define "multiplexing".

In my book, it's only really "multiplexing" if you also do demultiplexing. We don't do demultiplexing, but you do, effecively copying the various values from the outpost into a storage in your CPU.

If you don't assign the responses to the corresponding polling signals (which is what demultiplexing means), then it's not really a multiplexing protocol. And neither AccidentalChef nor me do that in our designs; we just fire the polling signals at the stations and then pick up the responses that we like, and discard the rest.

It's also the reason why my outposts respond with their own station number, because that saves me the trouble of having to synchronize the responses with the polling signals.

If you allow a broader definition of the term, then this is all multiplexing in some way or other.

[siggboy]

It's not going to be possible without SmartTrains. The features announced for 0.13 do not allow a single train line with logical station numbers, but we need that for these kinds of systems. Disabling the train stations will only be useful for much simpler setups.

Still true?

Answer the question yourself. How will you make a train go from the depot to a specific station with the new features in 0.13?

[DemiPixel]

It's not going to be possible without SmartTrains. The features announced for 0.13 do not allow a single train line with logical station numbers, but we need that for these kinds of systems. Disabling the train stations will only be useful for much simpler setups.

Still true?

Answer the question yourself. How will you make a train go from the depot to a specific station with the new features in 0.13?

I haven't gotten a chance to play. Is it possible to disable train stations? If so, could you have every station on every train, disable every station (except the main "meetup" one), and then enable a station and trigger the train when you want it to go to a specific station? If you disabled the train station immediately after, would the train stop or continue until it got there?

[siggboy]

I would like to add multi-station support for the depot circuit and I could use some suggestions on how to do it. I have a few ideas but maybe I've missed something obvious.

Here's how I'm doing it now (ignoring extra combinators that deal with oil barrels)... Each lane of the depot has two combinators, and works with a unique signal. For this example, let's say lane 1 in the depot uses the "1" signal. One combinator holds the destination in memory. [...]

In my case it's actually even simpler, but your suggestion of using unique signals for each lane certainly put me on the right track. Thanks a lot for the inspiration.

(Assume that the lanes use signals "A", "B", "C" and so on.)

I only need 1 arithmetic combinator per lane. It outputs "train-at-station * N" with the signal type of the lane, where N is the lane number. So it would output "A=1" if a train is at lane 1, "B=2" for lane 2 etc. All these outputs are put on a green wire bus which is then used as the input for the lane selection.

(I have a question about how you do the lane selection further down.)

Most of the combinators that I need are used for the selection of the lane. I need 2 for the counter, one for the selection of the signal, one to subtract the polling signal and 3 more which are my write-once register. That way I get a signal like "B=1" in the register which would select lane 2 (B). I can multiply the destination onto this and just put it on all the train stations directly (no need for a register at each lane since I only do one train at a time anyway).

Then I need 2 more to filter the "train-at-station" signal from the active lane. I do this by multiplying the "B=1" (from the above example) by 1000, adding all the train-at-station signals from the green bus and then just extracting all that are greater than 1000 (which is just one signal, namely the active one). This is then the effective "train-at-station" signal that I can simply feed into my existing depot circuit.

I'm very happy with the result, but I'm still interested in how you do the lane polling specifically in your implementation.

In your system, you could start polling stations once any train is available, and while you're doing that you pick which of the available lanes is going to be sent out. I use a counter that checks each lane in order but is paused rather than reset in between trains.

Can you give us the specifics how the counter that checks the lane works and how you hold it in between trains?

[siggboy]

Quick update:

• Choumi has made the first release of SmartTrains for 0.13, so I'll soon move to 0.13 and stabilize the project for the new Factorio.
• I've added resource bookkeeping for the requester stations. It works by sending the amount and type of resource to the requester when the train leaves the depot (the provider knows how much it's going to load on the train, and there's only one train at a time per provider). When the train arrives at the requester for unloading, then the cargo content of the train is subtracted again from the bookkeping register. The amounts in the bookkeeping register (plus what's in the train proper) are pretended to be in the buffer of the requester.
  This nicely streamlines the deliveries. I've tried it in my medium-size base (1 rocket per 3 minutes) and it works really, really well. Actually I'm loving it.
• I've added multi-lane support for the depot. Works really well and does increase train throughput.
• I'm now handling oil on a separate train line that does basically use the "version 1" of the setup for delivering empty barrels to the pumping outposts (and bringing full barrels on the way back to the oil depot). This works really well, too, and it's very simple. Can be used on the same wire along with the main depot.
  (Actually I've come to realize that you can even handle oil as a part of the main line, and it's what I'll be doing soon.)

If anybody is interested in the blueprints (and maybe instructions) for the updated stations I can post them here. Otherwise there won't be any new updates until everything works in 0.13 -- that might be a while, depending on how things go on the mod front and depending on any bugs in 0.13.

Roadmap for 0.13:

• Provider stations that offer more than one resource type (just like requesters can already request more than one resource type). This will require using filter inserters for the loading and those will be told what resource to load for the current train.
• Making the circuits smaller / optimization.
• Using the belt counter to determine average throughput of resources at providers, and possibly using this information to tighten the schedule (by indicating ready ahead of time).

Other than that I really feel that it's quite feature-complete now. The multi-lane capability and requester bookkeeping have really made the setup capable of handling high loads very efficiently. In my current base (which is medium sized at most), I build about 1 rocket per 3 minutes with only 20 trains handling everything (including a lot of arbitrary shipping of intermediate products, i.e. it's a truly decentralized setup). I think you can build a decentralized mega-base (1+ rockets/minute) easily with way fewer than 100 trains required to support a consistent load. We'll see. (My current base is at 50 outposts now.)

[Nexela]

Well now I get to cry in a corner until I can figure this out.

My stations on a 12.35 - 13.3 migrated save ignore circuit conditions in train stop rules! resetting smart trains or removing doesn't fix the issue. I used a custom migration script from .12 to .13 to replace all smart train stops with regular train stops (as well as replacing some other items from mods not updated yet.)

It works like it should in a new .13.3 world with smart trains. The mod set is different though so its possible somehow a mod is screwing with it but my guess is a migration bug. Time to dig in it further and hopefully figure it out.

[siggboy]

You should post in the SmartTrains thread about this, maybe Choumiko can help you. (I certainly can't because I only use the mod, I've never done any development work on it.)

I'm also not sure yet if I will migrate my current base to 0.13 or just make a new one. Updating the existing base won't be easy anyway.

[Nexela]

Oh guess I did post it in the wrong one! had both tabs open because they are the only 2 topics I subscribe too!