Smart Train Deliveries with Combinator Magick [0.13/0.14]

[siggboy]

The problem with that approach is that it assumes that you always have full trains of <stuff>. But <stuff> can be things like expensive intermediate products (e.g. speed modules) or simply things that you do not need to ship around in bulk (e.g. sulfur).

If you want to support these scenarios then it's not enough to only count "train loads", you need to count the actual resources.

Another problem is that even if you assume "full trains", the amounts vary due to differing stack sizes (4 wagons of coal is a different amount than 4 wagons of copper plates). If your receivers (requesters) can receive several goods -- and that's very useful unless you want to build countless receiver stations -- then you have to count the type of resource along with the trains. In this case it's easier to simply count the resource directly ("6k iron ore" is simpler to keep track of than "4 trains of iron ore", and in the latter case you also need to configure how much is in a train).

[kann_]

My current system is already based on train load and I use 2500 for Oil, 12k for ores for example and 4k uranium ore.
At each unloading/loading station the resources will get divided by the amount per train.

The requirement is just that each resource has a fixed train load, which is reasonable. Exactly because there are different stack sizes.
Although I never thought about speed modules... Usually I always have the leave if full/empty condition turned on. I guess that wont be possible if you want to send a single stack of items like speed modules. I think my system would work for speed modules if the supply stations don't have the full condition, but the supply stations need some modification.

Different resources per unloading station is not a problem at all since the station only requests full trains and unloads every train that arrives.

I thought you do it the same way. In what units do you request trains?
Think about it, requesting trains is really the best way to do it.

You only want a requester station to request if it needs a train load and you only want a provider station to provide when it has a full train load, otherwise the system will be slowed down with trains waiting to get unloaded/loaded. So you need to know how much fits in one train anyway at each station and every resource it uses.

I should check out your system more carefully

[siggboy]

I thought you do it the same way. In what units do you request trains?

In my system, the requesters do not request specific amounts. The requester says "I want resource of type X", and "X" can be several resources. The system will send a train to pick up "X" from a provider and send it to the requester when it is available. How much will be sent entirely depends how much the train will pick up at the provider. So you need to set up the providers accordingly. For example, I set up my "sulfur" provider to only load 1k sulfur into the train, because that's already more than enough. I also set up the providers to not buffer excessive amounts of resources (it would not make sense to buffer 20k sulfur, it takes eons until you've used that much, and it has downsides to create large buffers that you don't ever use up, e.g. if you want to move the station/buffer).

So the amounts shipped are determined at the provider side; the requesters take what they get.

BTW I also limit the buffers for typical bulk goods, such as metal plates, but in these cases I send full train loads (e.g. 12k copper plates) because usually these resources are used up quickly.

You only want a requester station to request if it needs a train load and you only want a provider station to provide when it has a full train load, otherwise the system will be slowed down with trains waiting to get unloaded/loaded.

Yes, it's true that you "waste" bandwidth if you send trains that are not full. But in many cases it's simply unreasonable to wait for a provider to buffer a full train load. See the "sulfur" example above, or some other high value goods such as the mentioned speed modules.

If you ferry smaller amounts it keeps the amounts in the buffers smaller and moves the entire system closer to a "lean production" facility where you only produce (and ship) the amounts needed at the time.

The train capacity should not be the measuring stick that dictates how much you need to ship.

I should check out your system more carefully

I'm not saying my system is better than yours, but it does take a different approach philosophy compared to yours in some areas.

[AccidentalChef]

I've been working on a similar train system for a while now, inspired by your original post in this thread. I'm not quite done but am close enough that it's starting to seem like it's worth discussing. I'd been trying to figure it out for a few days before you posted this, but hadn't thought of polling the outposts, and that was the idea that got everything going for me. My system is certainly over engineered in at least a few ways, but I'm also trying to put together a truly enormous factory before 0.13 launches. I'll need to be able to launch slightly more than one train per second to meet my goals. I know one of your major goals is simplicity and a lot of what I've come up with is serious overkill, but hopefully it'll be interesting anyway.

In order to keep the system easy to actually use and be able to send out trains fast enough, I had 3 main goals. First of all, the factory has to run on one train line for all providers and requesters, including oil. Second, if at all possible the depot has to know where to send trains before they arrive at the station rather than starting to search once the train arrives. Third, the system has to work with multiple depot stations (or, in practice, one multi-lane depot).

The base's CPU starts off by polling each station to see what it can request or provide. I does "shuffle" the order of the polling for reasons that should soon be clear. Each station can respond one of 3 ways. Providers will reply with positive station number of the resource they provide (eg, 37 iron ore), requesters will reply with a negative number (eg, -37 iron ore), and providers that also need barrels will reply with station number plus 1000 (eg, 1037 oil barrels). The CPU stores these replies into one of two memory cells, one for providers and one for requesters. Each cell can store data on multiple signals, so as it polls each station it can remember exactly who asked for which resources. As soon as a match is found (for example, both memory cells have a value for green circuits) those values are pushed out over the network (red for the provider station, blue for the requester) and removed from memory. The result of this is that one scan through all the stations can send out dozens of trains, rather than performing one scan for each train. The downside is that the scans have to be a bit slower. I think polling one station every 3 ticks is the limit. Because the system can only store one provider and one requester for each material, it shuffles the order of its polling so it doesn't poll two stations of the same type in a row.

Two things happen with these signals. First, the red signal goes into a FIFO buffer at the depot. If the buffer fills up, it pauses the CPU polling stations until space is available again. Farther away, each provider station listens for these broadcasts and if the red signal matches its station number, it stores the blue signal in memory. The blue signal tells each station where to send the train once it's full. These stations are designed so that basically all it takes to enable pipelining trains is adding a FIFO buffer and reconnecting a couple of wires, but I'm currently undecided whether my default station will allow one incoming train or two.

While there's at least one provider station's address stored in the buffer at the depot, the CPU is scanning each lane of the depot to see which have available trains. If a train is available, its first destination (the provider) is dequeued from the buffer and sent to that lane's memory cell. If that value is greater than 1000, the lane knows it's picking up oil and loads the train with empty barrels. The extra 1000 is subtracted and the train's destination is known. Otherwise, the train is sent out empty. If it's an oil train, it brings back empty barrels from the refinery station to restock the depot at the end of its trip.

I also haven't decided if it's worth the effort for the requester stations to be able to adjust their desired buffer sizes based on the latency of the trains or not. It could be as simple as just adding 10% to the desired amount each time the station's buffer hits zero, but since this system is already so complicated I'm hesitant to add even more combinators. I'm also considering having the requester stations listen for their station number so they can stop requesting (or at least adjust for expected incoming resources) until the train arrives. I'm unsure if this will provide any real benefit due to the stations already being polled out of order, which prevents too many trains from going to one requester at the expense of others.

I'm hoping to have everything done and tested this coming week, but would be interested to hear what you think about it.

[siggboy]

AccidentalChef, your design sounds great. I'd like to try it when you've got a working version, please make a demo map.

Couple of thoughts:

I think the key to having a high train throughput (you want more than 1 train/second, which is a lot) is having multiple depot lines; polling the stations in advance so the result is ready before a train arrives does not make much difference (but it does not matter for your design because you get that feature for free anyway).

I already thought about adding multiple lane support for my depot as well, and I would do it in much the same way as you did. Your design seems very sound in that area.

The oil handling is neat, but you'll need to add oil loaders to every lane in the depot (and wire the inserters everywhere so you can enable them only when needed). Shouldn't be too much of a problem, of course there need to be plenty of barrels so you can guarantee that every lane in the depot will have them when needed (or you need to make a more complicated loading scheme but that will introduce delays).

How do you shuffle the polling and guarantee that way that no two stations of the same type are not polled in a row? How is this helping with the problem of only having one station per resource type in the buffer? This part is not clear to me.

I think limiting the requesters in some way, while a train is on the way, is useful. Not for the reason that one requester gets favored at the expense of others, but because you overfill orders at the requesters, and you will have lots of trains queuing up, and that can cause problems on the railroad. In the general case you won't have a number of requesters for each resource so that the system will somehow even out the deliveries. Even then, the requesters will usually not all be requesting at the same time. For example, one of the iron ore requesters (if you have several) will start requesting ore, and immediately all your available iron ore providers will be activated to serve that requester. That might be 10 or more trains queueing up a minute later at that location. You don't want to have huge waiting bays everywhere for that reason.

The requesters should be limited. I still need to add that to my system as well.

[nuhll]

I love it. Please can someone make a tutorial how to do it exactly? Or maybe a video?

[siggboy]

I'm not a maker of videos .

If you can describe your scenario I can write a howto for you. You should probably use version 2 of the setup, since version 1 is quite limited (but it's good enough for smaller maps).

You should also download the demo map for version 2 and have a look at it. You need the latest pre-release version of SmartTrains.

Using the blueprints is not very difficult, but you need to be somewhat familiar with SmartTrains, otherwise it's possible that you cause problems for yourself if you configure your train stations incorrectly.

[nuhll]

Maybe with pictures, step by step? Oo

[siggboy]

I still need to document the version 2 anyway. I will also make a howto with screenshots. Until then, please download the demo map and you will see how it works in practice (it's why I made the demo map, it's supposed to be the documentation of how to use the system in a real game).

[AccidentalChef]

AccidentalChef, your design sounds great. I'd like to try it when you've got a working version, please make a demo map.

Couple of thoughts:

I think the key to having a high train throughput (you want more than 1 train/second, which is a lot) is having multiple depot lines; polling the stations in advance so the result is ready before a train arrives does not make much difference (but it does not matter for your design because you get that feature for free anyway).

It would be nice to eliminate the FIFO buffer so I may have to see if only polling when trains are at the station makes much difference. I think I'll probably end up keeping it, since the worst case time to find a matching provider and requester is pretty bad (though the average should be quite fast).

I already thought about adding multiple lane support for my depot as well, and I would do it in much the same way as you did. Your design seems very sound in that area.

I think it would work nearly as well (certainly for small to medium factories) to just see if there is a train at any station and just send the signal to the first station in the line. If that station is free to send a train, it does. If not, it just passes the signal to the next station. This would have the downside of favoring some lanes in the depot over others, and because of the oil barrel loading I decided against it.

The oil handling is neat, but you'll need to add oil loaders to every lane in the depot (and wire the inserters everywhere so you can enable them only when needed). Shouldn't be too much of a problem, of course there need to be plenty of barrels so you can guarantee that every lane in the depot will have them when needed (or you need to make a more complicated loading scheme but that will introduce delays).

I'm actually reconsidering how to handle the barrels today. It might make life a lot easier to have a barrel loading station next to the main depot. I'd stick with the current system where oil providers respond to the poll with their station number plus 1000, but now instead of a signal greater than 1000 telling the depot to load barrels, it would tell the depot to send the train to the barrel loading station. The barrel station would listen for signals greater than 1000 and store them so it could send oil trains on to their next destination. Simplifying and speeding up everything but oil is probably worth adding an extra stop to the oil trains.

How do you shuffle the polling and guarantee that way that no two stations of the same type are not polled in a row? How is this helping with the problem of only having one station per resource type in the buffer? This part is not clear to me.

Getting the throughput I want requires that the system can send out more than one train for some materials per polling cycle. That means I need to alternately poll providers and requesters for each material. This is obviously most important for iron and copper. It would be possible to do that by manually setting the order of the stations, but that seems tedious to me for a large base. My solution is to number the stations in blocks and poll them out of order so that for each requester, I poll a few providers before I poll the next requester. This doesn't completely eliminate any problems but it's far better than polling all the requesters or providers in a row and therefore ignoring most of them on each cycle.

To shuffle the order of the polling, I take the output of the counter, multiply it by a number relatively prime to the number of stations, then modulo the number of stations, sof we call the timer output T, the multiplier P, and the number of stations S, the output is (T*P) mod S. The numbers being relatively prime ensures that every station will be polled before the cycle restarts. I make sure P is at least as large as the number of iron ore requesters, which makes sure that no two requesters for any material are polled in a row. Again, this is mostly just for iron and copper. Most other things are requested infrequently enough that it's not worth worrying about. I do have a more complicated version that alters the order of the polling each time through (though it also repeats eventually) but I don't think it's necessary and probably won't use it.

I think limiting the requesters in some way, while a train is on the way, is useful. Not for the reason that one requester gets favored at the expense of others, but because you overfill orders at the requesters, and you will have lots of trains queuing up, and that can cause problems on the railroad. In the general case you won't have a number of requesters for each resource so that the system will somehow even out the deliveries. Even then, the requesters will usually not all be requesting at the same time. For example, one of the iron ore requesters (if you have several) will start requesting ore, and immediately all your available iron ore providers will be activated to serve that requester. That might be 10 or more trains queuing up a minute later at that location. You don't want to have huge waiting bays everywhere for that reason.

The requesters should be limited. I still need to add that to my system as well.

I think that's still less of a problem with my system because of the order I poll the stations, but you've convinced me to implement it anyway. I'll probably increment a counter when a train is queued up for the requester station and decrement it when a train departs the requester. That will let me have multiple trains incoming when necessary (likely ore requesters waiting on trains from distant mining outposts) without overstocking anything.

I'll put together a demo map once it's all done, though it may be hard to make one that really stresses the system without a huge amount of work.

[nuhll]

Please, could someone who understands this take 5 min of time and write down how excatly to archive this logic train system.

Maybe i can do a video of it then.

[DemiPixel]

At the very start of the station, why do you need * 2G? Couldn't you just do `each + 0 -> * 1` and then in the next combinator check `each = 2`?

[siggboy]

At the very start of the station, why do you need * 2G? Couldn't you just do `each + 0 -> * 1` and then in the next combinator check `each = 2`?

"Each = 2" would not work because there might be more than 1 station offering a resource. For example, if 3 stations are offering Iron Ore, and Iron Ore is requested, then the sum on the wire is "4". On the other hand, if Iron Ore is provided twice, but not requested at all, the sum on the wire would be "2".

In both cases, comparing to "2" would give the wrong result (a false negative, and a false positive, respectively).

Multiplying the requester flag with 2G and then adding the provider flags to this number means that the result will be negative (due to overflow) if, and only if, at least one provider does offer that resource.

It's not necessary to use "2G" as the addend and then checking for negativity. You can use any large number; for example, you could simply add "100" and check for ">100". This would only fail if you actually had more than 100 outposts offering the resource.

Edit: Actually the above explanation only makes sense for the version 2, which is not documented. In version 1 (the one that is diagrammed) the simple comparison with "2" does not work for another reason: since we compare "any = 2" this would trigger a false positive if any signal does have the value "2" on the wire (and this always happens simply because the polling signal assumes this value). So we need to make a large number appear on the wire in the case of a positive match which can not appear spuriously; -2^31 is a very safe value in this regard.

I'm using "2G" to make it obvious that it's a mathematical trick to make the comparison easier.

[DemiPixel]

At the very start of the station, why do you need * 2G? Couldn't you just do `each + 0 -> * 1` and then in the next combinator check `each = 2`?

"Each = 2" would not work because there might be more than 1 station offering a resource. For example, if 3 stations are offering Iron Ore, and Iron Ore is requested, then the sum on the wire is "4". On the other hand, if Iron Ore is provided twice, but not requested at all, the sum on the wire would be "2".

In both cases, comparing to "2" would give the wrong result (a false negative, and a false positive, respectively).

Multiplying the requester flag with 2G and then adding the provider flags to this number means that the result will be negative (due to overflow) if, and only if, at least one provider does offer that resource.

It's not necessary to use "2G" as the addend and then checking for negativity. You can use any large number; for example, you could simply add "100" and check for ">100". This would only fail if you actually had more than 100 outposts offering the resource.

Edit: Actually the above explanation only makes sense for the version 2, which is not documented. In version 1 (the one that is diagrammed) the simple comparison with "2" does not work for another reason: since we compare "any = 2" this would trigger a false positive if any signal does have the value "2" on the wire (and this always happens simply because the polling signal assumes this value). So we need to make a large number appear on the wire in the case of a positive match which can not appear spuriously; -2^31 is a very safe value in this regard.

I'm using "2G" to make it obvious that it's a mathematical trick to make the comparison easier.

Ah, I see. Are you not polling one station at a time in version 2? I don't see why you would get more than one station responding? But I understand the second part

[siggboy]

Ah, I see. Are you not polling one station at a time in version 2? I don't see why you would get more than one station responding? But I understand the second part

In version 2 we also have requester stations, and we need to match requesters with providers. This is done by polling in two stages: in stage 1 we poll the requesters; while this is happening, all providers put their resource flag on the wire. So, if there are 2 coal providers ready, then the signal we see would be "Coal = 2" (each of the outposts puts "Coal = 1" on the wire simultaneously). This gets combined with the "2G" signal from the requester, and if we overflow then we know that the requester does have at least one provider. We can then select this requester for delivery (since we know that we can deliver from at least one provider).

In the second stage we now poll the providers. The resource that we want is already selected (along with the requester from stage 1). When we have chosen a provider we are done and can send the train.

So it's basically the same thing as in version 1, but it happens twice, once for each "direction", so to speak.

[kann_]

I think that's still less of a problem with my system because of the order I poll the stations, but you've convinced me to implement it anyway. I'll probably increment a counter when a train is queued up for the requester station and decrement it when a train departs the requester. That will let me have multiple trains incoming when necessary (likely ore requesters waiting on trains from distant mining outposts) without overstocking anything.

I'll put together a demo map once it's all done, though it may be hard to make one that really stresses the system without a huge amount of work.

Have a look into my train counter: viewtopic.php?f=8&t=19594&start=20
Btw. why do your outposts answer with their station number and not with the amount of available resources in units of trains? If you do it that way and substract the number of trains you can easily figure out if there are resources left.

I would like to see a map of your system as well. Sounds pretty cool.

[nuhll]

Could you guys atleast explain how to use the blueprints? What mods are needed? And how to "use" it in game?

[AccidentalChef]

I think that's still less of a problem with my system because of the order I poll the stations, but you've convinced me to implement it anyway. I'll probably increment a counter when a train is queued up for the requester station and decrement it when a train departs the requester. That will let me have multiple trains incoming when necessary (likely ore requesters waiting on trains from distant mining outposts) without overstocking anything.

I'll put together a demo map once it's all done, though it may be hard to make one that really stresses the system without a huge amount of work.

Have a look into my train counter: viewtopic.php?f=8&t=19594&start=20
Btw. why do your outposts answer with their station number and not with the amount of available resources in units of trains? If you do it that way and substract the number of trains you can easily figure out if there are resources left.

I would like to see a map of your system as well. Sounds pretty cool.

I'd actually read through the thread you linked a couple of times already. My original plan was to do something with multiplexing but in the end, the polling method seemed to work better for my goals. I'm finishing up my implementation now, in which every station knows how many trains are incoming at any time. I considered adding a part to track average travel times from each station to another but that seems like a lot of work for little to no payoff.

The outposts respond with their station number because I really don't care how much material is available, as long as it's a full train load. Each provider and requester adjusts its answer based on how many trains are inbound anyway, so I guess I've just decentralized any calculation based on resource quantities. Effectively, each station is just answering the question, "If you send another train right now, can I fill it up?" regardless of how many are on the way already. It's not meaningful information to me to know how many trains it could fill up, since I'll be asking it again soon anyway.

The reason for answering with station numbers is to simplify computations at the CPU. As implemented, the CPU only cares about one thing: matching a station that has a full train load of material with a station that needs a full train load. It's very easy to keep track of these stations using the station numbers instead of resource quantities. As an example, let's say station #31 needs iron ore, and station #73 has a full train load available. For this example we'll ignore any out of order scanning and just simplify things. the CPU starts polling stations and gets to 31. Station 31 needs iron ore, so it responds -31 iron ore. The CPU sees that response, checks to see if the memory cell for requesters has an opening for iron, and if so, stores the signal -31 iron ore in memory. From this point forward, it will ignore any requests for iron ore until it finds a provider for station 31.

The polling continues and a similar process takes place for any other station, whether requesting or providing. It doesn't matter if we find the provider or the requester first, since either will be stored until their match is found. In this example, perhaps station 42 requests solid fuel, 55, provides green circuits, and 63 requests steel. Those values are all stored until a match is found. Now we get to station 73, which can provide a full train load of iron ore. The CPU doesn't have an iron provider stored, so it stores 73 iron ore to the provider memory cell. Immediately the CPU sees that an iron ore value is stored in both the requester and provider memory cells, so it knows it has a match and it's time to send a train. It extracts those signals from memory, removes those signals only (leaving all other data intact), converts the requester station number to blue and the provider to red, and pulses that signal out over the network. The Iron ore provider (station 73) sees the signal 73 red and knows to store whatever is on the blue signal (in this case 31 blue) because a train is coming and it needs to know where to send it. The requester (station 31) sees signal 31 blue and knows a train is incoming and to subtract that from how much iron ore it wants.

So, I guess the tl;dr version of this is that each station only needs to reply with basically a "yes" or "no" answer, rather than a value, because it already knows how many trains are in the way and can adjust based on that, and it replies with its station number because that makes the circuitry simpler everywhere else in the system.

[DemiPixel]

Bla

Since you're storing requester/provider as `amt of type`, what happens if more than one provider/requester have/want the same type of item? Do you simply ignore them until the first gets used and you poll them again?

[AccidentalChef]

Since you're storing requester/provider as `amt of type`, what happens if more than one provider/requester have/want the same type of item? Do you simply ignore them until the first gets used and you poll them again?

I'm not storing amount of type, I'm storing which station either requests or can provide a full train load of each type. Storing 37 Iron Ore means station 37 has at least one train load of iron ore available, and storing -52 would mean that station 52 needs at least one train load of iron ore. If more than one train load is needed or can be supplied, the station will have to wait until its polled again to ask for or offer another train load.

I do ignore further responses of the same type until a match is found for the first one. This isn't really a big problem because I can poll 20 stations per second, and I poll the stations in an order that makes sure the high volume resources (iron ore and iron plates especially) always alternate between polling providers and requesters. In actual practice it should be pretty uncommon for a requester to be ignored because the cpu is still trying to find a match for another one. Providers, especially mines, should be overbuilt just enough that there's usually a provider ready before a requester needs the resources.