Midi in-/output for Factorio

[The_Real_DrexXxacK]

Hello everyone,

I'd like to hear some thoughts about implementing midi in- and output capabilities in Factorio.

So the idea would be as simple as: have midi-notes available as signals to either be recieved or sent via regular circuit network (eiher simply as signal, or over new 'midi combinator' ) - available on any midi port outside of Factorio.

As a result there'd be an infinite number of possibilities to have your factory interact with the real world; or vice versa... i dont even know where to beginn with examples of how to utilize that!

Simplest idea: have a midi touchpad with led feedback: you could output anything (levels, states) to be visualized via led colour. Control trains, power switches, literally any circuit contition via hardware buttons. Set levels or logistic requests via faders or potis.

Engineering more complex midi setups could enable us to interact with our factory remotely via the mobile phone (eg. with touchOSC), any midi controller, any other midi-capable software (have a lightshow controlled by your factory ^^). With tools like bomes midi translator we could do even more complicated circuting than yet possible - still following the original intention of Factorio: automate and remote


So the one main question is: How complicated would it be to give Factorio access to midi devices, and could this be done in a mod without having to implement big bunches of stuff in the game code (as to say: without bothering the devs )

Cheers!

[AngledLuffa]

It's not exactly what you're looking for, but here's an option for midi blueprints:

http://miditorio.com/

It would be great if the already implemented instruments had more range to allow for more notes

[ssilk]

So the one main question is: How complicated would it be to give Factorio access to midi devices, and could this be done in a mod without having to implement big bunches of stuff in the game code (as to say: without bothering the devs )

I like the idea to control stuff via midi, cause that's the way to connect the virtual world of Factorio with the real world.


Technically speaking: Midi and Factorio's signal network have many commons. Indeed it should be possible to have MIDI-channels (signals), and on the channels notes (values). In Factorio itself the speaker device is ready to receive that values and play tones. But because you cannot have multiple values in one signal an because note-release must be translated into a "zero"-value you cannot play more than one note on a signal at one time. So for one signal you can play only monophonic. But that could be resolved with more signals...


What is technically difficult are two (or three) things:

a) Any input into Factorio needs to be deterministic.
This means: In Factorio you cannot input anything from outside, which is not guaranteed, that it is equal on all other instances of the game.
An instance are other players computers here. This is a very deep concept to ensure that Factorio is able to handle multplayer sessions.

Ok, what does that mean? You cannot directly put midi notes into the game without bringing the game out of sync or break all the rules to prevent that.

The only way to do that is to handle any input as an event which can be reflected to all instances of the game. Which means you need a mod, which simulates a "player". A player is, what you as player control with your mouse/keyboard. A player ensures that all his "actions" (=events) are transferred to all instances (the other players computers) and BEFORE the events trigger actions. An action is in that frame some mod, that waits on those events and converts the action to midi-notes on a circuit.

In Other words: The central part of the software needs to read the input (a midi file), and converts that to actions of a player. In this case this "player" doesn't control a so called "character", it is just a virtual player and is only there to make sure all actions are transferred in 100% sync to all instances, before the second part of the software comes into game: That part gets the player inputs and controls some signal networks (change the signals in a circuit network very fast).

So until here this seems to be easy, but - and maybe I need to be corrected here, cause my knowledge is now nearly 2 years old - there is no way to input something into Factorio that way. There is no function to read files, to read a network stream to read anything from outside of Factorio-context. There is no function to read the system time, because of the determinisim.

So this is the first thing you need to solve: How to input midi-files into Factorio.
There is the miditorio website (see link above), which converts a midi-file to a blueprint with chests as a "memory". Or which seems to be also easy: You can read the pressed key on keyboard and map notes to it (a-key is C, w is C#, s is D, e is D# and so on)...

I think there are converters, that can convert midi signals into keyboard events. That way it should be possible to input stuff.

b) how to convert midi channels into a Factorio signal?
Well, when you solve the problem to input non-deterministic data into factorio, the remaining part seems to be really simple: For one channel read note-height and length and convert it to value and microsenconds. Then "play" the note by sending the value to a signal on the circuit for N ticks. For more channels you need just more signals. A good starting point seem to https://mods.factorio.com/mod/shortwave

That's it.

So: Because of this determinism I don't think it's possible without bothering the devs. But implementing that would be a relieve for many, many suggestions, first of all the always wanted "I want to show the local time in the game in multiplayer".

[Chao]

So until here this seems to be easy, but - and maybe I need to be corrected here, cause my knowledge is now nearly 2 years old - there is no way to input something into Factorio that way. There is no function to read files, to read a network stream to read anything from outside of Factorio-context. There is no function to read the system time, because of the determinisim.

There may not be a function but there is already a way to get data from outside the factorio context that can already handle signals https://github.com/clusterio/factorioClusterio.

Admittedly it's not going to be a simple mod to implement and means you're going to have to at least run a server locally but could be a fun project if someone want to go deep into things...

[darkfrei]

Recipes and items for notes;
player inventory as holder for input / output values;
constant combinator that can read items from player inventory or set items to it.

Now you can click recipes with notes, send the virtual signal and play it with in-game speakers.

[The_Real_DrexXxacK]

Hey,
first of all: many thanks for your opinions and hints!

So: Because of this determinism I don't think it's possible without bothering the devs. But implementing that would be a relieve for many, many suggestions, first of all the always wanted "I want to show the local time in the game in multiplayer".

I agree - Devs are needed indeed ^^
Figured out that (pretty sure ^^) ingame lua cant access midi devices without the 'application Factorio' being able to select a midi in- and one outputdevice.
Since its only generic drivers for the very most midi devices: hopefully it'd be just like no work at all for the guys at Wube - if theres an interrest


Probably I am getting something wrong (im neiter a programmer, nor a native english-speaker ), but just to clarify:

My interpretation of midi-interaction is not aiming at playing back notes/sounds via midi, and not reading midiFILES (predefined midi-notes and timings) but live midi input from virtual or hardware midi controllers - and output (eg to give feedback by setting led colors or levels).

Technically speaking: Midi and Factorio's signal network have many commons. Indeed it should be possible to have MIDI-channels (signals), and on the channels notes (values). In Factorio itself the speaker device is ready to receive that values and play tones. But because you cannot have multiple values in one signal an because note-release must be translated into a "zero"-value you cannot play more than one note on a signal at one time. So for one signal you can play only monophonic. But that could be resolved with more signals...

(...)

b) how to convert midi channels into a Factorio signal?
Well, when you solve the problem to input non-deterministic data into factorio, the remaining part seems to be really simple: For one channel read note-height and length and convert it to value and microsenconds. Then "play" the note by sending the value to a signal on the circuit for N ticks. For more channels you need just more signals. A good starting point seem to https://mods.factorio.com/mod/shortwave

Serving the purpose of my idea (control/get feedback from your factory via external hardware) theres no real translation needed; also my interpretation of signal/value i s a bit different: Inputting " 'note on' on channel 1 with note c1 with velocity 50" should appear as (eg) Signal "CH1_Nc1_on" with value "50"
Any "note_off" should remove the corresponding signal from the signalcloud (=> midi signals should be present all over the map; or broadcasted to 'midi combinators' similar to ->shortwave)

The only way to do that is to handle any input as an event which can be reflected to all instances of the game. Which means you need a mod, which simulates a "player". A player is, what you as player control with your mouse/keyboard. A player ensures that all his "actions" (=events) are transferred to all instances (the other players computers) and BEFORE the events trigger actions. An action is in that frame some mod, that waits on those events and converts the action to midi-notes on a circuit.

Why create a virtual player? Just 'use' the real one who has the midi stuff running.. only if the ones with the midi controls are online there is a midi input to be synced. "Hold last value on input loss" and "Latest value takes precedence"-behavior should take care of things in absence of midi inputs. Thou there might be a need to dedicate and regulate midi controls; 3 players sending one note with different values at the same time might mess things up.. so probably needing the permission of an admin to transmit midi on speciffic channels might be a solution.
But this should come with an acceptable limitations.
For the moment i would even pay money for a buggy integration in my sense that would only work in singleplayer ;D

What is technically difficult are two (or three) things:

a) Any input into Factorio needs to be deterministic.
This means: In Factorio you cannot input anything from outside, which is not guaranteed, that it is equal on all other instances of the game.
An instance are other players computers here. This is a very deep concept to ensure that Factorio is able to handle multplayer sessions.

The deterministic part of that:
Midi in- and output must not be the same! if i set the (imagined ) midi signal CH1Nc1 to x in factorio, that willl only be outputed to the midi output, and not affect the same channel/note on the input. (feedback prevention)
if (as mentioned above) either a midi channel or a midi-range could(must) be assigned exclusively to one single player - wouldnt that make the input/signal deterministic? its just one source and one reciever broadcasting it to everyone (server).

In Other words: The central part of the software needs to read the input (a midi file), and converts that to actions of a player. In this case this "player" doesn't control a so called "character", it is just a virtual player and is only there to make sure all actions are transferred in 100% sync to all instances, before the second part of the software comes into game: That part gets the player inputs and controls some signal networks (change the signals in a circuit network very fast).

As mentioned in the beginning: its about reading a midi stream, not about reading files;
That means the maximum speed of changes is limited to the midi stream: ~32kbit/s (i dont know how to calculate further - but im guessing thats not quite much compared to factorios internal update rates ^^)
But reading the midi input just every .5 seconds would be fine enough to set certain things - call in trains, set logistic requests oder switch power. So "very fast" is not really needed.


So far - hoping that i didnt miss anything or get things totally wrong (- and hoping on eventually succeeding in getting things implemented )

DrexX


[Edit: added plural 's' where i forgot it]

[ssilk]

To clarify stuff:

- I'm 90% sure devs will never program a midi-input into Factorio. What I meant with "But implementing that would be a relieve for many, many suggestions" is some kind of API, that allows programmers to insert any non-deterministic data into Factorio in a way, that is compatible to multiplayer and doesn't need so much overhead as in Clusterio https://github.com/clusterio/factorioClusterio

- With a "player" I mean in this case this: https://lua-api.factorio.com/latest/LuaPlayer.html ; "character" is one type of controller (https://lua-api.factorio.com/latest/def ... ontrollers), but I would mean "ghost" or "god".
And why the mess? There is really no way to go around this "player" layer. It is needed. Factorio forbids programming around it and it will desync in multiplayer, if you find a way to go around it. And if you find a way it is not sure, if this "bypass" is fixed with the next Factorio version.

- You cannot create new signal types "on the fly". "CH1_Nc1_on" needs to be created during the startup of Factorio. You either need to use existing signal types ("wood", "stone", "belt", "A", "1" etc) or create new ones. "CH1_Nc1_on" seems to be a bit too special.
What I mean is more like: You can build a modded constant combinator with a name. In this example you create a constant combinator "CH1", which means "I'm the combinator for channel 1". Into that you place stone at the first space.
The mod recognises this combinator and now and "plays" the notes by changing the value of the stone very fast (can change every tick), according to the midi-signals. The notes on CH1 are then played on stone-signal (legato!) . "c1" is value 24 and the 24 is hold as long, as the midi-release signal comes.
For ployphony you need to insert more signals - it works like in any electronic instrument that has a limited number of voices, so if you play for example a C-Dur-Accord and you insert stone, wood and copper into the constant combinator, then it would set "stone" to 24, "wood" to 28 and "copper" to 31 as long as you hold the accord.

- Input/Output: Output of midi/data is much simpler than input. But of course would it be cool, if the same Midi-interface can be used for input and output. But that's not target of what I wrote. But kind of suggestion: If there is will be any kind of standardized interface to Factorio, then it should include input AND output.


Here the schema how I think it could work in a way, that is future compatible:


MIDI-File or some Midi-Input/Keyboard
|
Midi2Factorio (handles input and errors)
|
Midi2Factorio local Server (API)
|
Factorio-Midi-Api-Mod (reads from API and converts midi to signal stream)
|
Factorio "Player", type god; spreads inserted data to all instances of Factorio
|
Factorio-Midi-Player Mod (reacts to events from the player)
|
Constant Combinator with matching channel names
|
Wires (green, red)
|
Some Factorio device which does things with it


Well, that's my - very naive, not much thought about - software-architecture, of how that might work. It looks more complicated as it might be; there are more or less 2 big parts: The Factorio2Midi (program, server, API, whatever), some program working in the background on your computer. And the Factorio-Midi-Mod, which constists of two parts: The Midi-Api, which reads from Factorio2Midi and converts that to player actions. And the Midi-Player, which reacts on player actions, searches for combinators with a channel name and puts that data into the combinator.

The problem (why that many layers?) is still what I said in the first post: Determinism. Not sure if some layers can be stripped away, but this is how I would try to develop it first.