If every mod has its own lua state, does that not duplicate a lot of data?

[PTree]

Hello!

I consider Factorio having one of the best modding interfaces / APIs ever... so I'd like to learn about that aspect a bit to perhaps offer something like that in a project I might make one day.

Thus, I am curious...how is the issue of having event handling and all that solved for separate states?

My guess is userdata that gets put into table references, or something.

It's fine to be vague here, after all, I am kind of just here to learn about such things...my Factorio gameplay needs are very light, and I doubt I have much need to make any threads about it.
(Finished the basegame many a time, and mods, well, they're beyond the scope of support, for gameplay purposes)

[Pi-C]

Hello!

I consider Factorio having one of the best modding interfaces / APIs ever... so I'd like to learn about that aspect a bit to perhaps offer something like that in a project I might make one day.

Thus, I am curious...how is the issue of having event handling and all that solved for separate states?

Code: Select all

/c for name, version in pairs(game.active_mods) do game.print(name) end

This will list the active mods in the order they've been loaded by the game. If an event is raised, the game calls out to all mods which have registered a handler for that event. The game provides the event data in a table. Then it loops over the mods from game.active_mods. If any mod has registered an event handler, the handler will be called with the global event data. When that mod's event handler is done, the game continues the loop. If another mod has registered an event handler, that one will be called as well. However, the event data may have been changed by the first mod (e.g., event.entity of on_entity_damaged may have been healed). Also, events may be nested: If one of the active mods were to set event.created_entity.color in response to on_built_entity, this would immediately raise on_entity_color_changed and the game would start another loop over game.active_mods, this time trying to find mods that listen to the new event.

[PTree]

Hello!

I consider Factorio having one of the best modding interfaces / APIs ever... so I'd like to learn about that aspect a bit to perhaps offer something like that in a project I might make one day.

Thus, I am curious...how is the issue of having event handling and all that solved for separate states?

Code: Select all

/c for name, version in pairs(game.active_mods) do game.print(name) end

This will list the active mods in the order they've been loaded by the game. If an event is raised, the game calls out to all mods which have registered a handler for that event. The game provides the event data in a table. Then it loops over the mods from game.active_mods. If any mod has registered an event handler, the handler will be called with the global event data. When that mod's event handler is done, the game continues the loop. If another mod has registered an event handler, that one will be called as well. However, the event data may have been changed by the first mod (e.g., event.entity of on_entity_damaged may have been healed). Also, events may be nested: If one of the active mods were to set event.created_entity.color in response to on_built_entity, this would immediately raise on_entity_color_changed and the game would start another loop over game.active_mods, this time trying to find mods that listen to the new event.

Ah, yes, that does seem to go with the advisory warning that mods are loaded by dependency and then by alphabet, as well as the nested events.
I think I'm on the same page as that.

I'm trying to conceptualize the nature of having actually separated Lua states, which I assume Factorio uses, instead of coroutines or environments or whatever.
(I know of those concepts but have not tried them myself)

I assume that Factorio keeps one internal, C/C++ side Lua registry, or at least some sort of manager, that knows all the separate states (to iterate over them) and works with them...
The mystery to me is the data sharing, if there is indeed any.

Without thinking deeply about it, my first impression was that it would just be a lot of duplication of functions, like, let's say the utility functions the game provides, table.deepcopy and whatnot.
I have not done enough research to actually know if such functions are actually available to the separate Lua states, but if they were, then, without any C Lua API shenanigans, every since state would have to have these functions as their own copy, which adds to memory usage.

Not much... hell, probably quite insignificant amounts for even loosely contemporary machines, but I'm still curious how they actually implement this clean, yet highly customizable modding interface.

I don't want it spoonfed, but ...the general workings would be nice. The only other game I find has similar amounts of customization flexiblity is Arma (3) by BiS...
Perhaps the secret is to be Czech... but yeah
I want to be transparent about this: I have no commercial project running or anything (and since I'm a Linuxer, I don't think I ever will go for that), but I still kind of want an idea about how to do something like this.

[Pi-C]

I'm trying to conceptualize the nature of having actually separated Lua states, which I assume Factorio uses, instead of coroutines or environments or whatever.
(I know of those concepts but have not tried them myself)

I assume that Factorio keeps one internal, C/C++ side Lua registry, or at least some sort of manager, that knows all the separate states (to iterate over them) and works with them...

The following is just how I suppose things may work, in the oversimplified way of somebody who's lacking the theoretical background and is not a professional programmer (I've studied humanities, not computer science).

The mystery to me is the data sharing, if there is indeed any.

I guess the game doesn't care about states, but entities, tiles, items etc. Mods don't do really much: Basically, a mod with a control script won't do anything on its own. The file control.lua will be read and will pull in any files loaded with "require()" when a new game is started or when an existing game is loaded. Depending on what the game signals during bootstrapping, the mod will then run the handlers for script.on_init, script.on_load, and/or script.on_configuration_changed. Usually, handlers for all events the mods needs to listen have been registered at this point. Apart from that, the mod will idle and do nothing until one of the events it's listening to is raised. At that point, the event handler will act like a remote function that has been called by the game with the event data as an argument.

In my last post, I've made a mistake, by the way: Apparently, the same event data will be passed on to all mods:

Code: Select all

====================================================================================================
Entered event script for on_entity_damaged
Event data: tick = 475379, entity = car "tank" (714), cause = character "character" (9), force = LuaForce 1 ("player"), damage_type = LuaDamagePrototype "explosion", original_damage_amount = 1160, final_damage_amount = 343.5, final_health = 1656.5
(@__autodrive__/scripts/event_handlers.lua: 1654)
====================================================================================================

 294.162 Script @__autodrive__/libs/debugging.lua:161: event.entity: car "tank" (714)
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.original_damage_amount: 1160
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.final_damage_amount: 343.5
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.final_health: 1656.5
 294.162 Script @__autodrive__/libs/debugging.lua:161: Healed entity!
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.original_damage_amount: 1160
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.final_damage_amount: 343.5
 294.162 Script @__autodrive__/libs/debugging.lua:161: event.final_health: 1656.5
 294.163 Script @__autodrive__/libs/debugging.lua:161: 
====================================================================================================
Leaving event script for on_entity_damaged
Event data: tick = 475379, entity = car "tank" (714), cause = character "character" (9), force = LuaForce 1 ("player"), damage_type = LuaDamagePrototype "explosion", original_damage_amount = 1160, final_damage_amount = 343.5, final_health = 1656.5
(@__autodrive__/scripts/event_handlers.lua: 1668)
====================================================================================================

 294.163 Script @___debug__/control.lua:36: Entered script for on_entity_damaged!
 294.163 Script @___debug__/control.lua:40: event.entity: tank (714)
 294.163 Script @___debug__/control.lua:41: event.original_damage_amount: 1160
 294.163 Script @___debug__/control.lua:42: event.final_damage_amount: 343.5
 294.163 Script @___debug__/control.lua:43: event.final_health: 1656.5
 294.163 Script @___debug__/control.lua:44: event.entity.health: 2000

As you can see, both mods get the same event data, but event.entity has been changed (health: 1656.5-->2000). Now suppose the first mod wouldn't heal the damaged entity, but kill it for good:

Code: Select all

1801.866 Script @__autodrive__/libs/debugging.lua:161: 
====================================================================================================
Entered event script for on_entity_damaged
Event data: tick = 474961, entity = car "tank" (714), cause = character "character" (9), force = LuaForce 1 ("player"), damage_type = LuaDamagePrototype "explosion", original_damage_amount = 1160, final_damage_amount = 343.5, final_health = 1656.5
(@__autodrive__/scripts/event_handlers.lua: 1654)
====================================================================================================

1801.866 Script @__autodrive__/libs/debugging.lua:161: event.entity: car "tank" (714)
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.original_damage_amount: 1160
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.final_damage_amount: 343.5
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.final_health: 1656.5
1801.866 Script @__autodrive__/libs/debugging.lua:161: Destroyed entity!
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.original_damage_amount: 1160
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.final_damage_amount: 343.5
1801.866 Script @__autodrive__/libs/debugging.lua:161: event.final_health: 1656.5
1801.867 Script @__autodrive__/libs/debugging.lua:161: 
====================================================================================================
Leaving event script for on_entity_damaged
Event data: tick = 474961, entity = LuaEntity (invalid), cause = character "character" (9), force = LuaForce 1 ("player"), damage_type = LuaDamagePrototype "explosion", original_damage_amount = 1160, final_damage_amount = 343.5, final_health = 1656.5
(@__autodrive__/scripts/event_handlers.lua: 1670)
====================================================================================================

As event.entity has become invalid, there is no point in raising on_entity_damaged for the mods that haven't seen the event yet, so the log file ends with the output of the first mod.

Without thinking deeply about it, my first impression was that it would just be a lot of duplication of functions, like, let's say the utility functions the game provides, table.deepcopy and whatnot.
I have not done enough research to actually know if such functions are actually available to the separate Lua states, but if they were, then, without any C Lua API shenanigans, every since state would have to have these functions as their own copy, which adds to memory usage.

Absolutely speculating: I imagine the game providing a kind of super-environment which makes libraries like util, gui etc. available to all other mods. Mods can "require()" a library (actually, a table of functions indexed by function name), and I'd really be surprised if that would place a copy of the library or its functions in their library -- I'd rather bet they all will get a pointer to the same function instead. There shouldn't be any issues with two different mods locking the function for each other because Factorio is an event-based game. Thus, whenever something happens to an entity, a GUI element, players, etc., the game will raise the appropriate event in the manner described: Call first mod that listens to the event, wait until it is done, check for stop condition (e.g. "event.entity is no longer valid) and proceed to the next mod if possible …

Not much... hell, probably quite insignificant amounts for even loosely contemporary machines, but I'm still curious how they actually implement this clean, yet highly customizable modding interface.

If you're lucky, perhaps one of the devs may give you a more technical explanation.

[Tertius]

The mystery to me is the data sharing, if there is indeed any.

If you mean the game state with data sharing, most objects the API is providing are not native Lua objects or arrays that contain copies of the game data. Instead, they are userdata objects whose properties and methods are directly accessing the internal game data. So if one mod changes some game property in an API object, it's not changing some Lua variable cached in the API object. Instead, a userdata handler is called and the change is performed directly on the game data and whatever it does to the game is done immediately. This way, any change is also directly visible to other mods, because there is no duplication or caching of data.
Your mod can of course make explicit copies of API object properties to native LUA variables, but this doesn't usually make sense on a larger scale.

[Rseding91]

Every mod gets is own independent lua state and all the machinery and libraries associated with it. The simplest answer is: it just doesn't use that much memory to have each one independent. It's a few kilobytes to a megabyte for each mod. Compare that to a single high resolution texture loaded into memory (50 megabytes or more) and it just isn't significant.