Factorio Forums

Disabling mods? Suggestion: disable, benchmark vanilla only

- I was not that happy to disable mods for benchmarking tests but it came up there
- How cooler is Editor Extension’s underground belt for creating ore / dumping plates? Yes, cooler than the vanilla exp loader & the inf chest
- But as measured there is a negative UPS impact
- To avoid further issues (like to run into mod which IS bad for benchmarking) I’d suggest testing vanilla. It is just simpler.
- Stability of the results are about the same: 1% (yellow boxes)

The Savegames

- ZM: Zero Mod - Only vanilla elements. Mods disalbed before save.
- EE: Editor Extensions on - same as ZM, so only vanilla elements
- EET: Editor Extensions on, using the Ungerground Belt from EE to produce ore/dispose plates
All attached here.

Results

- ZM and EE is the same savegane and just tested to prove that having EE alone has not impact on the results. That's proven
- Vanilla saves (ZM, EE) run at 187 UPS, EET runs at 166 UPS
- Results shows the same ~1% accurancy in both cases
- There is one outlier result, my bad: I've disabled AV and other tools and the test was already running, ignore that blue bar
- As a bonus you can see 10k, 25k and 50k tick runs - 50k runs give a bit better accurnacy but the differnce is really small
- Another graph from this dataset is posted here

Ha Belter,

Will you take this save in your bechmark?
It beats everything we made up unto now
It is from the ups master Stevetrov.

The one other concern I have with benchmarking with mod entities is that it is unclear whether it takes in the same ups as an equivalent batch of miners or even infinite chests+loader in vanilla. After testing, I know for a fact that there is a difference between those two vanilla methods, so I would expect similar differences to occur with modded entities.

The impact is a slight bias towards increasing or decreasing the input lane count to optimize the last few percent, as the UPS per item produced changes whether you are at 7.5 or 22.5 items per lane. As such, testing with different number of source items would produce different UPS results than a vanilla version of the same thing.

I would expect that other entities could have similar behaviour.

All ups measures are relative.
Hardware, OS, configuration, mods, all differ between all players.
So it doesn't matter if you use editor extensions or not. it is far more important that all the maps for testing have the same configurations, and the same amount of mod elements, whether it is the EE loader/unloader or the vanilla version.

Agreed, all UPS is relative to a number of variables. Let's assume for a moment that we are talking about the same design, on the same machine. What I mean is that when comparing varied designs, the impact of the mod elements such as ore sources, may* have different trends than vanilla elements with different item rates, causing variants with different amounts of the ore sources (though with the same total ore rate) to behave differently UPS-wise.

This is not an issue for the comparison at a conceptual stage when all players agree to use a common mod setup and a common ore source (ex: EE, infinite chests), but when translating the design back to the vanilla ore sources (ex: miners or train unloading), the UPS-optimal number of ore sources vs transport line designs maybe different. True, it's a very small effect, and can be neglected most of the time, but will be noticeable trying to get the last (fractions of a) percent out of very optimized designs in a factory settings. As evidence, I submit the following comparison of miners vs infinite chests (https://www.reddit.com/r/technicalfacto ... te_chests/)

*haven't tested EE, but I would expect some effect. How much depends on many factors, as you said. Since mods rely on lua scripts (is that correct?), I don't know what that means for performance.

disentius wrote: ↑

Mon Mar 21, 2022 1:18 pm

All ups measures are relative.

Absolutely, but for comparison purposes you want to:

• Minimise the overhead of things you are not testing (so the relative UPS of different but functionally equivalent designs is closest to their relative UPS effect).
• Maximise ease for others to repeat tests.
• Minimise potential sources of variation affecting consistency of results.

All these favour benchmarking without mods.

agreed:) that is why we stopped using EE.