[1.1.21] Minor Performance Regression from 1.1.19

[PLLovervoltage]

Platform

Windows 10
Ryzen 3600
Test map: flame_sla 10k map downloaded from https://factoriobox.1au.us/maps


Issue

I was testing reducing my memory timings and was benchmarking factorio ups with map downloaded from site above. I noticed a measurable drop in performance and after further testing trying to narrow it down I determined it was from the newer factorio version and NOT from my timing adjustments. Not sure if this is considered an issue but thought to bring it up just in case.


Data (4 runs of 5000 updates)

Version 1.1.19
31927 ms
31880 ms
31660 ms
31731 ms

Version 1.1.21
32661 ms
32525 ms
32574 ms
32540 ms

Which translates to about ~158 ups to ~153 ups drop.

sidenote: More variation in the 1.1.19 measurements as they were recorded from different memory timings. The 1.1.21 has less variance as it was repeated on the same memory timing.

[boskid]

I did quick benchmark of 1.1.19 and 1.1.21 using that save file and simple bat file:

Code: Select all

Factorio_x64_1.1.19\Factorio_1.1.19\bin\x64\factorio.exe --benchmark 4c5f65003d84370f16d6950f639be1d6f92984f24c0240de6335d3e161705504.zip --benchmark-ticks 10000 --benchmark-runs 1 > benchmark-19a.txt
Factorio_x64_1.1.21\Factorio_1.1.21\bin\x64\factorio.exe --benchmark 4c5f65003d84370f16d6950f639be1d6f92984f24c0240de6335d3e161705504.zip --benchmark-ticks 10000 --benchmark-runs 1 > benchmark-21a.txt
Factorio_x64_1.1.19\Factorio_1.1.19\bin\x64\factorio.exe --benchmark 4c5f65003d84370f16d6950f639be1d6f92984f24c0240de6335d3e161705504.zip --benchmark-ticks 10000 --benchmark-runs 1 > benchmark-19b.txt
Factorio_x64_1.1.21\Factorio_1.1.21\bin\x64\factorio.exe --benchmark 4c5f65003d84370f16d6950f639be1d6f92984f24c0240de6335d3e161705504.zip --benchmark-ticks 10000 --benchmark-runs 1 > benchmark-21b.txt

Just in case of cpu temperature dependant performance i was interleaving both versions. Results:

.19: Performed 10000 updates in 53350.269 ms | avg: 5.335 ms, min: 4.563 ms, max: 19.230 ms | checksum: 242813681
.21: Performed 10000 updates in 53545.973 ms | avg: 5.355 ms, min: 4.512 ms, max: 23.244 ms | checksum: 502419499
.19: Performed 10000 updates in 53731.172 ms | avg: 5.373 ms, min: 4.576 ms, max: 20.370 ms | checksum: 242813681
.21: Performed 10000 updates in 53998.415 ms | avg: 5.400 ms, min: 4.616 ms, max: 22.852 ms | checksum: 502419499

I am not seeing any performance regression.

-- edit:
Just in case i made 3 runs of every version interleaved with max cooling - still no significant differences:

.19: Performed 10000 updates in 53492.417 ms | avg: 5.349 ms, min: 4.434 ms, max: 20.331 ms | checksum: 242813681
.21: Performed 10000 updates in 53645.828 ms | avg: 5.365 ms, min: 4.556 ms, max: 23.226 ms | checksum: 502419499
.19: Performed 10000 updates in 53427.115 ms | avg: 5.343 ms, min: 4.579 ms, max: 19.497 ms | checksum: 242813681
.21: Performed 10000 updates in 53793.643 ms | avg: 5.379 ms, min: 4.589 ms, max: 22.821 ms | checksum: 502419499
.19: Performed 10000 updates in 53536.815 ms | avg: 5.354 ms, min: 4.578 ms, max: 19.270 ms | checksum: 242813681
.21: Performed 10000 updates in 53532.988 ms | avg: 5.353 ms, min: 4.587 ms, max: 22.886 ms | checksum: 502419499

[PLLovervoltage]

I repeated your methodology and got the same difference in perfomance. System wasn't thermal throttling before. But to eliminate randomness from Ryzen precision boost overdrive I set system to run on a manual fixed clock.

benchmark-19a: 61741 ms
benchmark-21a: 62805 ms
benchmark-19b: 61846 ms
benchmark-21b: 63090 ms

I've also attached the output files