Friday Facts #408 - Statistics improvements, Linux adventures

[Haighstrom]

Finally the accumulator charge graphs! Fabulous.

I'm not totally convinced about the new production filter UI. The sizing and brightness of controls gives more prominence for fiddling around with quality views than global vs surface view, when the likely relative importance to the player is the other way around. I was staring at "All (Separate)" and trying to guess its meaning for ages before seeing the dropdown - it was unintuitive for me at first glance.

I would make the quality filtering less prominent (add a little settings cog and stick in it a submenu perhaps?), and the Global box bigger and brighter. Why not a drop down with the list of surfaces, with one saying "ALL" ? I agree with others saying the term "global" is confusing.

[Kallinger]

Very nice FFF, never expected to see a post about Wayland in a FFF, you guys always surprise me!

Talking about non-blocking save, the only problem i ever had was on heavily modded games, with big saves (SE-K2) that my PC ran out of both RAM and Swap. (Firefox stealing like 10GB did not help)
As a result my games crashed more often on save than at any other point, which is pretty much the worst possible timing, so I don't know if that's possible but could the game check if enough RAM is available for the non-blocking save before it gets started, and if not fall back to the default save operation?

[FuryoftheStars]

Talking about non-blocking save, the only problem i ever had was on heavily modded games, with big saves (SE-K2) that my PC ran out of both RAM and Swap. (Firefox stealing like 10GB did not help)
As a result my games crashed more often on save than at any other point, which is pretty much the worst possible timing, so I don't know if that's possible but could the game check if enough RAM is available for the non-blocking save before it gets started, and if not fall back to the default save operation?

You may want to take a look here and/or add to it:
viewtopic.php?f=182&t=112884

[youROCK]

This is all really nice work and I'm increasingly frustrated with being stuck with 1.1 hahahaha
(Still playing it though )

Windows also has mechanisms of shared memory combined with copy-on-write, however it's very Windows specific and definitely not as easy to use as a simple fork() with a little bit of process synchronization. So I guess it's not feasible to get this background saving to Windows as well. However, it would be a really great QoL feature if implemented nonetheless.

I use WSL a lot on my day job, and while I knew about the Windows lack of a good fork(), after reading today's FFF this morning I started wondering how WSL does accomplish what clearly is the correct semantics for it. I ended up finding this article that does a short but informative dive into the subject. So my conclusion is "cool, all we need to do is write a kernel-level driver, that's not gonna be a nightmare at all!"

I think what you need isn't a regular kernel driver either, it has to be a "native api" level driver (native meaning using NT kernel directly). It's probably relatively simple code but getting it right is probably very tricky

[aka13]

Since wsl 1 got ditched, I doubt that any advances in that direction will be made in wsl 2.

[mmmPI]

I had no particular suggestions for the statistics, it sound already good to me that there is a possibility to show each planet stats or "global"(galactic ?/total ?/combined ?) stats.

I think re-ordering the more produced/consumed item so that the second column start with the 2nd most produced/consumed instead of an item in the middle of the list as suggested would be a good idea.

I found the second part of the FFF sounding like a whole unknown territory, interesting to read about things completly unfamiliar to me.

[valneq]

So what do you think? Are there any other statistics improvements you can think about for 2.0?

As has been requested many times: Please add the option to display statistics on a logscale! More specifically a semilog plot.

And why do wee need this? It allows to meaningfully display data that spreads over several orders of magnitude:
copper cable production can easily peak in the hundreds of thousands, while production of modules is much lower and satellites even less.

What would that look like? This is a random exampled pulled off Wikipedia, note how the y-axis increases by factors of ten:


source

[leoch]

Any chance you can make the colours match on production and consumption graphs?

[Xunie]

Are there any other statistics improvements you can think about for 2.0?

Yes. Min/max (as well as "last") tracking of accu-charge/production/pollution/circuit-network statistics data.
(See https://oss.oetiker.ch/rrdtool/doc/rrdtool.en.html and RRDtool in general and search for "consolidation" to see what I mean!)

Example:
It's very handy to know what the min and max accumulator charge level has been for each of the last 20 days. It may also be very handy to know what the min/max production speed has been for copper plates. Maybe every day, copper production automatically shuts off due to power constraints Or maybe my lasers completely drained my accumulators and maxed out my power production for a minute? Right now that data is measured, but eventually consolidated into average values that you can view in the various graphs. So you can't know this for certain. This may just not be clear in the data shown in the 10 hour graph because it's consolidated into average values! Not ideal! I can try to stare at single pixel graph lines, but it won't necessarily give me better insight into the data. You just can't tell minimum and maximum values from 'averaged' data! See here:

Example:

factorio suggestion.png (75.27 KiB) Viewed 2151 times

(I have to commend the rendering of the graphs. luv me some pixels at 1080p. Simple as)

Solutions:

• Keep min/max and last data as well.
• Allow viewing time series data as candlesticks/OHLC.
• Allow creation of complex 'graphs' with multiple time series and synthetic data such as total_kills/pollution.
• A circuit network debugger that also logs min/max/avg/last data in the same manner. (allows tracking of individual production lines as well as regular circuit network debugging)
• Allow viewing and cross referencing of multiple disparate data sources such as circuit network data with power, production or pollution data.
• Dashboard views with draggable windows and widgets!
• Draggable always visible mini graphs in the HUD so you can track statistics while expanding.
• Dashboards visible on a second monitor!

Perhaps you create a single graph with multiple lines for viewing the kills per day for each planet surface as shown by a line graph.
Drawing multiple candlestick time series on a single graph is visually very noisy. But this is not a problem when observing a single time series.
Drawing additional lines over candlesticks is quite doable and I'd love this feature as well.
If tracking both min/max values, just being able to get a candlestick going in the power statistics window is already a big bonus.

Dashboards and debuggers:
Imagine attaching a 'debugger' to a circuit network on some planet somewhere and remotely keeping track of that value with a candlestick chart.
But next to that chart on that custom dashboard are two other graphs: production of plastics on that particular planet as well as the charge status of accumulators on that planet.
The ability to bookmark chart views and make custom 'dashboard' views for yourself/faction where you can have multiple charts of particular time series data would be an incredibly powerful thing and would be really fun to look at together.

Presentation and open/close values:
'Candlestick' and 'OHLC' charts are useful for showing both min&max values during a 'time slot' as well as showing a 'closing' and 'opening' value. However, because the factory operates continuously, there is no strict "closing" and "opening" time like in markets. Times for sunrise/sunset may differ per planet/season/mod. For 'daily' power statistics, these can be computed/measured right in the middle of the sunrise and/or sunset period of the day. Whatever makes sense, I guess. I have no sage advice here. Don't let timezones bite your butt, I guess.

Note: candlestick charts are probably a little less useful for regularly sampled time series data. However, for the circuit network with timing, it may be very important to know what the value at the 'opening' tick is! Let the user decide what graph they want I guess?

What do y'all think? Is this overkill or 'just right'?

[Tertius]

Improvement for the statistics view?
Often, I want to inspect the actual performance of some production line. Not the performance of the whole surface but only some line. For example the production line for red circuits, including their precursor items like green circuits and plates. I want the actual behavior, not the "should be" statistics from a tool like rate calculator. I don't want to include that other green circuit line far away used for a different thing.

So I'd like to be able to select a factory part and get the statistics for that part only.

It's part of the lean cycle: build, measure, learn.

Currently I'm tapping belts to measure and display as numbers with nixie tubes, but that's a chore.

It's probably required to create a specific database for each selection, so these selections can be persistent per save. I create a selection, and from that point on all buildings in that selection are not only reporting into the global database but also into the selection database. The history reaches back to the creation of the selection.

[Amarula]

Thanks for all the linux love and support. Been using the non-blocking save for years without problems.
As I mention in my response to viewtopic.php?p=294841#p294841 I think the production stats deserve some love - looking forward to whatever improvements are made.

[Koub]

There have been several suggestions regarding the statistics graphs, and ways to improve on them. Production and consumption stats and graphs are surprisingly one of the parts of the game on which I've seen the most suggestions.

Here is a list of those I could find, and I'm sure I missed some :

viewtopic.php?f=6&t=9388 - Match colours in production and consumption graphs for the same item
viewtopic.php?f=6&t=38901 - Add more significant digits
viewtopic.php?f=6&t=44908 - Add a logarithmic scale on graphs
viewtopic.php?f=6&t=50655 - Add a section with the net difference between production and consumption
viewtopic.php?f=6&t=52326 - Use the same Y axis scale for production and consumption
viewtopic.php?f=6&t=53977 - Add peak consumption/production values to the graphs
viewtopic.php?f=6&t=60497 - Remember the time scale upon exiting the graphs
viewtopic.php?f=6&t=82586 - Add units to the mouse over tooltip on graphs
viewtopic.php?f=6&t=95640 - Add an option to smooth the curves in the production graph
viewtopic.php?f=6&t=110100 - Allow syncing production and consumption filters
viewtopic.php?f=6&t=111774 - Add a graph that shows demand

[Dev-iL]

@raiguard
Waaaait a second - if Allegro is gone, what's stopping you from properly supporting RTL languages (Hebrew, Arabic, Farsi, ...)?

[camela_camera]

desperately need option for mods to disable statistics graphs for specific electrical networks, since the performance cost of maintaining the graph for unused or hidden networks greatly limits the interesting things that can be done with them

[jockeril]

Well, not only am I super excited about those new statistics features (put them in now !!! ) I'm also happy to read about the Linux part of the game and happy it supports Linux natively (and with added perks and features) because this is only strengthening my pending decision to just install Linux instead of upgrading my Windows 7 x64 10 y/o machine to Windows 10, which I avoided doing for several years now and really don't want to do,

I've actually already confirmed other games I like will run on Linux with either Proton or Wine full support (as I don't play the "latest and greatest" AAA bs with all the bugs anyway) and now it's just a matter of choosing the distro which has become my biggest problem...

So I guess I'll see you on Linux soon !

[jockeril]

@raiguard
Waaaait a second - if Allegro is gone, what's stopping you from properly supporting RTL languages (Hebrew, Arabic, Farsi, ...)?

B A Z I N G A ! ! ! !

[valneq]

[…]
I've actually already confirmed other games I like will run on Linux with either Proton or Wine full support (as I don't play the "latest and greatest" AAA bs with all the bugs anyway) and now it's just a matter of choosing the distro which has become my biggest problem...

So I guess I'll see you on Linux soon !

I've been gaming on Linux for years now, have had only very minor issues with games. Praise be Steam's Proton!

When it comes to distro, it mostly will not matter for gaming, as long as you can run Steam.

[argbla]

Doesn't "global statistics" imply statistics over a globe, or in other words a planet?

Exactly what I was thinking when reading that word. Maybe something like "universal" (from universe) would be more fitting here?

"Galactic Statistics" also sounds cool.

[jockeril]

Are there any other statistics improvements you can think about for 2.0?

Yes. Min/max (as well as "last") tracking of accu-charge/production/pollution/circuit-network statistics data.
Solutions:

• Keep min/max and last data as well.
• Allow viewing time series data as candlesticks/OHLC.
• Allow creation of complex 'graphs' with multiple time series and synthetic data such as total_kills/pollution.
• A circuit network debugger that also logs min/max/avg/last data in the same manner. (allows tracking of individual production lines as well as regular circuit network debugging)
• Allow viewing and cross referencing of multiple disparate data sources such as circuit network data with power, production or pollution data.
• Dashboard views with draggable windows and widgets!
• Draggable always visible mini graphs in the HUD so you can track statistics while expanding.
• Dashboards visible on a second monitor!

[...]

Dashboards and debuggers:
Imagine attaching a 'debugger' to a circuit network

[...]

The ability to bookmark chart views and make custom 'dashboard' views for yourself/faction

[...]

Presentation and open/close values:
'Candlestick' and 'OHLC' charts are useful for showing both min&max values during a 'time slot' as well as showing a 'closing' and 'opening' value.

[...]

What do y'all think? Is this overkill or 'just right'?

Somebody, Please, build this guy a mod that exports these statistics in real time so he can create a dynamic web page or an excel spreadsheet to put on a second monitor,

then again, maybe Wube would like that Idea and we (those of us who are looking for real time data tracking, which might or might not include me) can have extended statistics on a second monitor next to my CPU and GPU real time info

Just sayin'

[jockeril]

Finally the accumulator charge graphs! Fabulous.

I'm not totally convinced about the new production filter UI. The sizing and brightness of controls gives more prominence for fiddling around with quality views than global vs surface view, when the likely relative importance to the player is the other way around. I was staring at "All (Separate)" and trying to guess its meaning for ages before seeing the dropdown - it was unintuitive for me at first glance.

I would make the quality filtering less prominent (add a little settings cog and stick in it a submenu perhaps?), and the Global box bigger and brighter. Why not a drop down with the list of surfaces, with one saying "ALL" ? I agree with others saying the term "global" is confusing.

I'd like to expand on that to have the cog-menu be like a filter for all the quality options (though I'd probably just want to normally see the highest to know how well I am at turning everything into the highest quality, but now that I thought about it, the journey would probably reveal if the ratios are good or not...