[Linux PSA] Using Microsoft mimalloc results in ~50% reduced update time

Post all other topics which do not belong to any other category.
Post Reply
User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

[Linux PSA] Using Microsoft mimalloc results in ~50% reduced update time

Post by ptx0 »

Hi all,

Stumbled upon this: https://www.reddit.com/r/factorio/comme ... &context=3

Code: Select all

Did you notice the gradual drop in performance with each successive run in the two hugetlb cases?

This isn't just a testing artefact. If you monitor the huge page usage (AnonHugePages in /proc/meminfo) you will see that less and less huge pages are used with each successive run. For example on my system during the first run I see ~2.3GB huge page usage (pretty much the whole memory used by Factorio), second run ~1.6GB, third run ~1.2GB, fourth and fifth run ~900MB.

What is happening here is that after each run when Factorio frees all the memory used by the map data the glibc malloc calls madvise(..., ..., MADV_DONTNEED) on the freed memory. This is generally a good thing as it releases the physical memory backing those memory areas, however this destroys the huge page mappings, and they mostly don't get restored when the memory gets allocated again for the next run.

The same happens when running interactively with GUI when you exit a game to the main menu and load another save (or the same save) without restarting Factorio.

I've found no way to prevent this from happening when using the glibc malloc implementation. However, I had success with using mimalloc (https://github.com/microsoft/mimalloc) instead. libhugetlbfs isn't needed with that as mimalloc has huge page support built in that can be enabled through an environment variable. After installing it according to the documentation you can use it pretty much the same way as you'd use libhugetlbfs through the use of LD_PRELOAD:

env LD_PRELOAD=/usr/local/lib/libmimalloc.so MIMALLOC_LARGE_OS_PAGES=1 /path/to/factorio/bin/x64/factorio
which on my system (Ubuntu 20.04 on ZFS, Ryzen 3700x, ECC 2667MHz memory) reduces my update time from ~7ms to ~3.5-4ms.

it uses 25% of the memory it uses when not using mimalloc - major savings here.

Additionally, running the following:

Code: Select all

sudo renice -n -19 $(pidof factorio)
schedules the task as realtime, giving it priority over every other application/process other than the kernel itself, preventing some frame time spikes due to CPU wait.

blazespinnaker
Filter Inserter
Filter Inserter
Posts: 665
Joined: Wed Sep 16, 2020 12:45 pm
Contact:

Re: [Linux PSA] Using Microsoft mimalloc results in ~50% reduced update time

Post by blazespinnaker »

quote might be easier to read.
Did you notice the gradual drop in performance with each successive run in the two hugetlb cases?

This isn't just a testing artefact. If you monitor the huge page usage (AnonHugePages in /proc/meminfo) you will see that less and less huge pages are used with each successive run. For example on my system during the first run I see ~2.3GB huge page usage (pretty much the whole memory used by Factorio), second run ~1.6GB, third run ~1.2GB, fourth and fifth run ~900MB.

What is happening here is that after each run when Factorio frees all the memory used by the map data the glibc malloc calls madvise(..., ..., MADV_DONTNEED) on the freed memory. This is generally a good thing as it releases the physical memory backing those memory areas, however this destroys the huge page mappings, and they mostly don't get restored when the memory gets allocated again for the next run.

The same happens when running interactively with GUI when you exit a game to the main menu and load another save (or the same save) without restarting Factorio.

I've found no way to prevent this from happening when using the glibc malloc implementation. However, I had success with using mimalloc (https://github.com/microsoft/mimalloc) instead. libhugetlbfs isn't needed with that as mimalloc has huge page support built in that can be enabled through an environment variable. After installing it according to the documentation you can use it pretty much the same way as you'd use libhugetlbfs through the use of LD_PRELOAD:

env LD_PRELOAD=/usr/local/lib/libmimalloc.so MIMALLOC_LARGE_OS_PAGES=1 /path/to/factorio/bin/x64/factorio
OptimaUPS Mod, pm for info.

Post Reply

Return to “General discussion”