Fast reading for inventory slots

[darkfrei]

Hi devs!

Is it possible to read inventory faster than now? The code like

Code: Select all

local inventory = entity.get_inventory(defines.inventory.chest) -- defines.inventory.chest == 1
if inventory then
  for n = 1, #inventory do
    local item_stack = inventory[n]
    if item_stack and item_stack.valid_for_read then
      game.print(serpent.line (item_stack))
    end
  end
end

Every this reading needs a lot of UPS, for example just player's inventory with 60 slots needs almost 3 ms, but 20 slots inventory needs 1 ms.

It looks like the C++ part is much faster, I've tested it with this mod, but there is very tiny UPS improvement.

But If we use this mod EmptySlotsReader and read slots one-by-one, it needs too much UPS.

Can you please check if it can be easier just with

Code: Select all

local inventory = entity.get_inventory(defines.inventory.chest)
local slots = inventory.get_slots() -- read only; nil if no inventory

Where slots is just one Lua table such as

Code: Select all

slots = {
  [1] = {item_name = "iron-plate", count = 4}, 
  [2] = nil, 
  [3] = {item_name = "copper-plate", count = 16},
  [4] = nil,
  [5] = {blocked = true} -- etc.
  }

2020-02-04T15_55_48-Factorio 0.18.3.png (14.56 KiB) Viewed 3662 times

and

Code: Select all

slots = {
  [1] = {item_name = "iron-plate", count = 4}, 
  [2] = nil, 
  [3] = {item_name = "copper-plate", count = 16},
  [4] = {filter = "steel-plate"},
  [5] = {blocked = true} -- etc.
  }

2020-02-04T15_56_20-Factorio 0.18.3.png (16.8 KiB) Viewed 3662 times

UPD: replaced "contents" to "slots".
Reason: The Inventory.get_contents already works.

[Honktown]

I think in your case the print is chewing up the ups:

Code: Select all

/c

local function on_tick()
  local player = game.players[1]
  local selected = player.selected

  if not selected then return end

  local inventory = selected.get_inventory(defines.inventory.chest)
  local slots = 0

  if inventory then slots = #inventory else return end

  local str = ""
  local item = nil
  local i = 1

  for i=1,slots do
    item = inventory[i]
    if item.valid_for_read then str = str .."\nitem: " .. item.name .. " count: " .. item.count end
  end

  log(str)
end


script.on_event(defines.events.on_tick, on_tick)

1250.590 Script local function on_tick() local player = game.players[1] local selected = player.selected if not selected then return end local inventory = selected.get_inventory(defines.inventory.chest) local slots = 0 if inventory then slots = #inventory else return end local str = "" local item = nil local i = 1 for i=1,slots do item = inventory if item.valid_for_read then str = str .."\nitem: " .. item.name .. " count: " .. item.count end end log(str) end script.on_event(defines.events.on_tick, on_tick):1:
item: steel-chest count: 4
item: small-electric-pole count: 2
item: small-electric-pole-2 count: 2
item: medium-electric-pole count: 1
item: medium-electric-pole-2 count: 1
item: medium-electric-pole-3 count: 1
item: medium-electric-pole-4 count: 1
item: big-electric-pole count: 1
item: big-electric-pole-2 count: 1
item: big-electric-pole-3 count: 1
item: big-electric-pole-4 count: 1
item: substation count: 1
item: substation-2 count: 1
item: substation-3 count: 1
item: substation-4 count: 1
item: transport-belt count: 2
item: fast-transport-belt count: 1
item: express-transport-belt count: 1
item: high-speed-transport-belt count: 1
item: regenerative-transport-belt count: 1
item: underground-belt count: 2
item: fast-underground-belt count: 2
item: express-underground-belt count: 2
item: high-speed-underground-belt-structure count: 2
item: regenerative-underground-belt-structure count: 2
item: splitter count: 1
item: fast-splitter count: 1
item: express-splitter count: 1
item: high-speed-splitter count: 1
item: regenerative-splitter count: 1
item: chute-miniloader count: 1
item: miniloader count: 1
item: fast-miniloader count: 1
item: express-miniloader count: 1
item: filter-miniloader count: 1
item: fast-filter-miniloader count: 1
item: express-filter-miniloader count: 1
item: burner-mining-drill count: 1
item: stone-furnace count: 1
item: assembling-machine-1 count: 1
item: wood count: 1
item: iron-plate count: 8
item: pistol count: 1

https://drive.google.com/open?id=1bSLCe ... QzkxZO05P4

Going over a chest with 48 slots adds ~ .35 to the script update timer, and that's doing a single log print after going over the inventory. Doing it on an empty chest with 65530 slots pushes the script update to 135ms

Edit: I'm not against a whole inventory contents table, which would be super useful.

[Honktown]

Related, wanted to know for a while:

Does putting local var inside the loop cost more than putting it outside?

Answer: no. Same performance on the 65530 slot chest with local outside vs inside.

[Rseding91]

I'm fine with adding some utility functions to LuaInventory.

Although I'm a little confused as to what you actually want?

inventory.get_contents() is already a thing?

[eradicator]

As far as i remember the "empty slot reader" mod thread the main problems that needed solving were:
LuaInventory .find_empty_stack() --new
LuaInventory .count_empty_stacks() --new
LuaInventory .can_insert() --new second return value: item count

I.e. faster methods to determine if and how much space is free in an inventory.

[Rseding91]

"how many more items can fit in this inventory" is a super loaded question and heavily depends on several factors.

Things like item health, tool durability, items that hold other items, and filtered slots effect those and make it super complicated to give a direct number.

[Oktokolo]

"how many more items can fit in this inventory" is a super loaded question and heavily depends on several factors.

But a can_insert wich returns the actual count of insertable items would be really helpful.

[Rseding91]

"how many more items can fit in this inventory" is a super loaded question and heavily depends on several factors.

But a can_insert wich returns the actual count of insertable items would be really helpful.

As I said that's not something that I think I can realistically do because it depends on too much. For example: can you insert "1 repair tool": if it's at 100% durability then no, but if it's <= 59% then yes; because there's no room except there's an item stack of "repair tool" at max count but with durability of 41%. So, the <= 59% repair tool can be "added" to the "full" stack of existing repair tools without actually changing the count in the inventory; it just 'refills' the repair tool.

Repeat that for an entire inventory and there's just no simple way to say "this many can be inserted" if I have to account for all that. If I just ignore all of that; then sure it's easy. If count < max_stack_size then count the difference and that's the number you get.

If that's "ok" then I can do that and just stick a big disclaimer on it about items with health, durability, and things like armor with equipment grids not stacking.

[Oktokolo]

But a can_insert wich returns the actual count of insertable items would be really helpful.

As I said that's not something that I think I can realistically do because it depends on too much.

The most naive way to implement it, is:
1. Clone the inventory.
2. Perform an insert on the clone, getting the count of actually inserted items.
3. Delete the clone.
4. Return the count of inserted items from step 2.

This should still be way faster and definitely more reliable than anything i could come up with in LUA.

P.S.: Can i clone an Inventory in LUA so that it stil has a working insert method?

[Honktown]

But a can_insert wich returns the actual count of insertable items would be really helpful.

As I said that's not something that I think I can realistically do because it depends on too much.

The most naive way to implement it, is:
1. Clone the inventory.
2. Perform an insert on the clone, getting the count of actually inserted items.
3. Delete the clone.
4. Return the count of inserted items from step 2.

This should still be way faster and definitely more reliable than anything i could come up with in LUA.

P.S.: Can i clone an Inventory in LUA so that it stil has a working insert method?

The inventory is a Lua object, hence why it has valid. When you get the inventory from the entity, you don't need any valid check at all if you can be certain it exists (maybe part of why the one script eradicator was testing had terrible ups). valid_for_read doesn't mean the same thing obviously.

If that's "ok" then I can do that and just stick a big disclaimer on it about items with health, durability, and things like armor with equipment grids not stacking.

I wish more people would ask/share because I have a mod which already does just that for train wagons, in I believe one of the most efficient ways possible. Unfortunately I've gotten no feedback over performance, but there was a small spike in downloads...

[Optera]

As far as i remember the "empty slot reader" mod thread the main problems that needed solving were:
LuaInventory .find_empty_stack() --new
LuaInventory .count_empty_stacks() --new
LuaInventory .can_insert() --new second return value: item count

I.e. faster methods to determine if and how much space is free in an inventory.

The original question asked was "How many slots are empty and how many are (partially) occupied?".
viewtopic.php?p=473658#p473658
Back then I said it was far to expensive to iterate through every slot of an inventory.

What I'd need to implement the original question in Inventory Sensor is a single function "count_empty_slots" returning only slots neither filtered, blocked or filled with any item.

I wouldn't bother with filtered slots.
"count_empty_slots(ItemPrototype)" -> returns free slots for this item
Might be nice to have, but too expensive compared to a simpler version treating filtered slots as blocked.

[Oktokolo]

But a can_insert wich returns the actual count of insertable items would be really helpful.

As I said that's not something that I think I can realistically do because it depends on too much.

The most naive way to implement it, is:
1. Clone the inventory.
2. Perform an insert on the clone, getting the count of actually inserted items.
3. Delete the clone.
4. Return the count of inserted items from step 2.

This should still be way faster and definitely more reliable than anything i could come up with in LUA.

P.S.: Can i clone an Inventory in LUA so that it stil has a working insert method?

The inventory is a Lua object, hence why it has valid. When you get the inventory from the entity, you don't need any valid check at all if you can be certain it exists (maybe part of why the one script eradicator was testing had terrible ups). valid_for_read doesn't mean the same thing obviously.

I double-checked but have still no idea, what you tried to say here.

[Honktown]

I double-checked but have still no idea, what you tried to say here.

P.S.: Can i clone an Inventory in LUA so that it stil has a working insert method?

https://lua-api.factorio.com/latest/Lua ... ory.insert

The inventory is it's own recordable LuaX thing

Code: Select all

/c

local players_inv = {}
local player = nil
local pid = 1
local inventory = nil

for pid, player in pairs(game.players) do
  if player.valid then players_inv[pid] = player.get_main_inventory() end
end

for _, inventory in pairs(players_inv) do
  inventory.insert({name = "iron-plate", count = 5})
end

.insert(itemstack) also exists for the entity itself:

https://lua-api.factorio.com/latest/LuaEntity.html

Inherited from LuaControl: get_inventory, get_main_inventory, can_insert, insert...

[Oktokolo]

P.S.: Can i clone an Inventory in LUA so that it stil has a working insert method?

https://lua-api.factorio.com/latest/Lua ... ory.insert

The inventory is it's own recordable LuaX thing

Code: Select all

/c

local players_inv = {}
local player = nil
local pid = 1
local inventory = nil

for pid, player in pairs(game.players) do
  if player.valid then players_inv[pid] = player.get_main_inventory() end
end

for _, inventory in pairs(players_inv) do
  inventory.insert({name = "iron-plate", count = 5})
end

.insert(itemstack) also exists for the entity itself:

https://lua-api.factorio.com/latest/LuaEntity.html

Inherited from LuaControl: get_inventory, get_main_inventory, can_insert, insert...

Inserting items into an inventory is not the problem nor is getting the inventory.
I need a way to check, how much items of a stack i could insert into an inventory. I want to do the check without actually performing the insert on the real target inventory.
Sadly, can_insert is crippled to only return whether any item could be inserted - it does not return the count of items that could be inserted.
The idea is to clone the inventory and perform a real insert on the copy (not altering the original inventory) because insert does return the count of actually inserted items.
But how to clone an inventory in a way that the copy still has the insert method with the secret code capable of handling all the edge cases of item insertion?

I actually don't think, that it is possible in reasonably performant way (want to do it every couple of ticks). So i will probably end up doing can_insert followed by an insert and then a remove to undo the insert - naively assuming that remove is perfectly symmetric to insert (wich it probably isn't - i'll just hope that the resulting bugs are minor enough to be ignorable)...

[Rseding91]

The entire base game works without needing to know the exact count of items that could be inserted so I find it weird that mods would now need to know that?

In cases where the base game wants to know "can I fit X into this" it just ignores all the durability stuff and "works fine". Enough that most nobody notices when an inventory is actually able to hold 5% more of a repair pack.

[Honktown]

Inserting items into an inventory is not the problem nor is getting the inventory.
I need a way to check, how much items of a stack i could insert into an inventory. I want to do the check without actually performing the insert on the real target inventory.
Sadly, can_insert is crippled to only return whether any item could be inserted - it does not return the count of items that could be inserted.
The idea is to clone the inventory and perform a real insert on the copy (not altering the original inventory) because insert does return the count of actually inserted items.
But how to clone an inventory in a way that the copy still has the insert method with the secret code capable of handling all the edge cases of item insertion?

I actually don't think, that it is possible in reasonably performant way (want to do it every couple of ticks). So i will probably end up doing can_insert followed by an insert and then a remove to undo the insert - naively assuming that remove is perfectly symmetric to insert (wich it probably isn't - i'll just hope that the resulting bugs are minor enough to be ignorable)...

I should test if using get_contents() is faster than inv.getsignal(i) (probably is, but not by that much), but I do a search for empty spaces and do stack - item_in_space for my mod:

Code: Select all

              local items = {}
              local item = nil
              local count = 0
              E = 0
              inv = c.car.get_inventory(defines.inventory.cargo_wagon)
              for i=1,#inv do
                if not inv[i].valid_for_read then E = E + 1
                else
                  if items[inv[i].name] then
                    items[inv[i].name] = items[inv[i].name] + game.item_prototypes[inv[i].name].stack_size - inv[i].count
                  else
                    items[inv[i].name] = game.item_prototypes[inv[i].name].stack_size - inv[i].count
                  end
                end
              end

Insertable item count is game.item_prototypes[name].stack_size * E + items[name]

At big numbers, the only speedups may be caching game.item_prototypes, and using get_contents() instead of 1-by-1. I'd be willing to make a script thing to shove a stack of every item into a chest with 65530 slots (Infinite Buffer Chest) until it's full and see what times are like when you try to check free space. : P

Checking each slot for valid_for_read was about 130ms per tick on my computer.

Behind the scenes would be much faster. With that 65530 slot chest, it's 130ms/65530, or 1983 nanoseconds per slot's valid_for_read. In C++ code it could get that down 3-5x for many slotted chests which were empty at least (a guess), but it depends on a lot of things.

Doing insert and remove is way more sloppy, but I can't blame people for using it. There's not many options.

@Rseding
I did notice one day I stacked science packs that the green bar filled on the icon. Never thought about it before. Neat.

[Honktown]

I'm posting this separately because wall-of-text:

Wow, get_contents() can't even be used for empty spaces. I thought it returned a thing like LuaConstantCombinatorControlBehavior.parameters, which returns an array for the slots. get_contents() returns table[name] = count, which loses the awareness of distribution (e.g. is a count of two for a single item taking 1+1 space, or a single space with 2 in it).

Also caching the inventory slot itself is possible, I wonder if that could speed things up after the first acquisition tick.

[Oktokolo]

In cases where the base game wants to know "can I fit X into this" it just ignores all the durability stuff and "works fine". Enough that most nobody notices when an inventory is actually able to hold 5% more of a repair pack.

Would probably work for me too. Please make the function doing this check for a given stack on a given inventory callable from LUA.

[Rseding91]

Ok, I added LuaInventory::find_empty_stack(), count_empty_stacks(), and get_insertable_count().

From testing the "count number of non-empty slots" on a 48,000 slot inventory it went from 78 MS to 0.16 MS.

[darkfrei]

Ok, I added LuaInventory::find_empty_stack(), count_empty_stacks(), and get_insertable_count().

From testing the "count number of non-empty slots" on a 48,000 slot inventory it went from 78 MS to 0.16 MS.

You are awesome!