Friday Facts #115 - The power switch

[Yeol]

I've worked on projects that used in-house equivalents to the STL before, mostly to get around bad STL implementations on some platforms. If you're really only using a very small subset of the STL or boost, they can be fine, but sooner or later you need something that's not in your "STL-lite", and you have to add it. And that entails more work, testing and so on and so forth. It gets tiring after a while.

There is a build system called FASTBuild I've used on some past projects which massively cut into build times. I kind of hesitate to suggest it, however, because of a few things. The first is that I don't know if it's ever really been used outside of the company where I used it, which means that help might not be widely available for it. The second is that, well, it might be overkill for a game like Factorio with a fairly small team. It has features that let it transparently offload some compilation to other machines that have spare CPU time, that cache the results of compilations on a given machine so they can be automatically reused by other people on the project, and so on. I don't know how effective this would be for small teams.

On the bright side, when I used it with Visual Studio and it was properly configured, the build process would really be no different from normal: just hit build. It basically replaces the normal MSBuild process that Visual Studio uses by default to run its compiler/linker. It's also designed to target a whole bunch of platforms (mostly consoles), so it should be usable for the Linux side of things as well.

If you're interested, you can take a look at http://fastbuild.org. Hopefully you'll be able to tell at a glance whether it might be useful for Factorio. The Features section has some performance comparisons. I wish I could do more than just drop a link, but build systems have never been my specialty.

[kovarex]

I've worked on projects that used in-house equivalents to the STL before, mostly to get around bad STL implementations on some platforms. If you're really only using a very small subset of the STL or boost, they can be fine, but sooner or later you need something that's not in your "STL-lite", and you have to add it. And that entails more work, testing and so on and so forth. It gets tiring after a while.

There is a build system called FASTBuild I've used on some past projects which massively cut into build times. I kind of hesitate to suggest it, however, because of a few things. The first is that I don't know if it's ever really been used outside of the company where I used it, which means that help might not be widely available for it. The second is that, well, it might be overkill for a game like Factorio with a fairly small team. It has features that let it transparently offload some compilation to other machines that have spare CPU time, that cache the results of compilations on a given machine so they can be automatically reused by other people on the project, and so on. I don't know how effective this would be for small teams.

On the bright side, when I used it with Visual Studio and it was properly configured, the build process would really be no different from normal: just hit build. It basically replaces the normal MSBuild process that Visual Studio uses by default to run its compiler/linker. It's also designed to target a whole bunch of platforms (mostly consoles), so it should be usable for the Linux side of things as well.

If you're interested, you can take a look at http://fastbuild.org. Hopefully you'll be able to tell at a glance whether it might be useful for Factorio. The Features section has some performance comparisons. I wish I could do more than just drop a link, but build systems have never been my specialty.

Thank you for the info, it looks interesting, I actually stumbled across fastbuild some time ago, but I didn't have the time to fiddle with it long enough to make it work. Now when I inspected the page, I states that it can be integrated with visual studio, but I couldn't find a step by step way how to do it for dummies. Is there such a guide?

[Yeol]

Thank you for the info, it looks interesting, I actually stumbled across fastbuild some time ago, but I didn't have the time to fiddle with it long enough to make it work. Now when I inspected the page, I states that it can be integrated with visual studio, but I couldn't find a step by step way how to do it for dummies. Is there such a guide?

I've looked a bit, but haven't found one. That said, there is at least one example of it in the FASTBuild source tree since FASTBuild can be used to build itself. The basic idea is that the fastbuild.bff file that contains the build configurations and executable targets also has targets of type "VCXProject" and "VSSolution". Those targets generate the appropriate visual studio projects/solutions that are configured to match what has been set in the .bff file (kinda like generating a VS solution with CMake if you're familiar with that). Included in the generated solution is everything that makes VS use FASTBuild instead of the default MSBuild process for this particular solution.

So really, there's no setup to do within VS itself. Once the solution/project is generated, you just open it normally, and the build action has been set to call FASTBuild properly for you.

The reference stuff for project/solution generation is kinda hard to find on the FASTBuild website, but it's available at the bottom on this page: http://fastbuild.org/docs/functions.html. If you can use FASTBuild to compile, you should be able to use it to generate a project relatively easily.

Again, I'd like to note that all of this is based on my experience using FASTBuild in an environment where other people were in charge of it, though. It might turn out to be harder to setup than I think it is.

[aboeren]

I was wondering if this tool could also be something that could help with the recompile times, I'm not a hundred percent sure if it suits your needs, it was talked about being used in the 'Bugsmashers' episodes of Star Citizen's community video's;

Recode - http://www.indefiant.com/

Love the game by the way =)

Boz

[obstinate]

Hi, I work at an organization that builds a very large software product that most people use every day. So I can speak somewhat to the compilation speed question.

There are a few things to look at. To begin with, you should always understand the source of a problem before trying to fix it. So it would be good to understand the bottleneck to your build: IO? CPU? Mem (unlikely)?

After that, you can start looking at some fixes. Y'all probably have SSDs in all your dev machines, so I assume IO isn't the issue. If CPU is the problem, there are still a few things you can check:

[*] Are your builds incremental?
[*] Are you parallelizing adequately?
[*] Where is time being consumed? Compilation or linking? (Or the pre-compilation eval step -- less likely, I think.)
[*] If compilation: are all files slow, or just one?
[*] Are you persisting intermediates across branches where possible?
[*] Are you including only what you use?

I ask the last question because you said in the blog post that you're considering extracting a boost subset that only implements what you need. I don't know what Boost's policy on unnecessary includes is. But I would hope that Boost files only include what they use, which suggests that making your own Boost subset would not be terribly productive.

If you still think it would be productive, it is a good idea to compile with -E on Clang to get a sense of what is dominating the transitive include tree. Maybe you can get away with just blanking one or two files from your Boost distribution. Many times, I've found that there was an unnecessary dependency in a commonly used h file that could be factored into a cc file. There have been occasions where I reduced the compile times of all source files in a project an order of magnitude by breaking a single dependency.

I'm also curious what build system y'all are using. Most of my experience is with Bazel (or, the internal version of the same).

[kovarex]

Hi, I work at an organization that builds a very large software product that most people use every day. So I can speak somewhat to the compilation speed question.

There are a few things to look at. To begin with, you should always understand the source of a problem before trying to fix it. So it would be good to understand the bottleneck to your build: IO? CPU? Mem (unlikely)?

After that, you can start looking at some fixes. Y'all probably have SSDs in all your dev machines, so I assume IO isn't the issue. If CPU is the problem, there are still a few things you can check:

[*] Are your builds incremental?

Yes, and when when I'm editing just cpp files, the compilation is practically instant.

[*] Are you parallelizing adequately?

Yes, it would be 10 times slower if not.

[*] Where is time being consumed? Compilation or linking? (Or the pre-compilation eval step -- less likely, I think.)

Compilation (when developing), on final release build, it is link, as we use link time code generation there, but it doesn't really matter.

[*] If compilation: are all files slow, or just one?

All of them, we have 1000 modules currently, some of them takes longer time, but it is a general problem.

[*] Are you persisting intermediates across branches where possible?

Yes, but when I switch branches, the probability of changing something in common.hpp is big. Also, the visual studio tentds to be buggy when checking for changed depencencies, so often I just recompile to be sure it won't be broken.

[*] Are you including only what you use?

Mostly yes, I created a small tool to prune inclues and I use it from time to time.

I ask the last question because you said in the blog post that you're considering extracting a boost subset that only implements what you need. I don't know what Boost's policy on unnecessary includes is. But I would hope that Boost files only include what they use, which suggests that making your own Boost subset would not be terribly productive.

After some inspection, I don't really believe. Take the boost::property_tree::ptree as an example. the ptree is just special case of boost::property_tree_tree which is a template and it uses huge amount of templates and includes. I tried the /P option (thanks for that), to see how many lines it includes, and it is 316 978 (more than all the lines including in cpp files in our whole project). I understand that the guys from boost wants to share all the code and make everything perfectly expandable and general. But if I make my own custom ptree that is not a template, so lot of the functionality can be hidden in .cpp and when it is simplified, it suddenly can use up thousandth of the lines in the include. And I strongly believe that the compilation is then also faster, as it is very often using just simple class methods, instead of making build (repeatedly) template code in the header in every module when it is used. And it will be also faster in runtime, as I can optimise the ptree for our use with state memory pool.

If you still think it would be productive, it is a good idea to compile with -E on Clang to get a sense of what is dominating the transitive include tree. Maybe you can get away with just blanking one or two files from your Boost distribution. Many times, I've found that there was an unnecessary dependency in a commonly used h file that could be factored into a cc file. There have been occasions where I reduced the compile times of all source files in a project an order of magnitude by breaking a single dependency.

I'm also curious what build system y'all are using. Most of my experience is with Bazel (or, the internal version of the same).

Visual studio on windows and makefile on linux/mac.

[woytaz]

After some inspection, I don't really believe. Take the boost::property_tree::ptree as an example. the ptree is just special case of boost::property_tree_tree which is a template and it uses huge amount of templates and includes. I tried the /P option (thanks for that), to see how many lines it includes, and it is 316 978 (more than all the lines including in cpp files in our whole project).

If this number includes templates that aren't instantiated, then no code will be generated for them. It'll still take time for the compiler to go through those lines, though I'd imagine it's not as bad.

Are you using "extern" (i.e. explicit instantiation declaration) with templates so the same instantiations aren't compiled multiple times?

[bear21]

At: Compilation times (C++ and technical)

You can try compiling the project as 1 .cpp file with all others included.

We started doing this at my work. Depending on your compiler, we saw a 15 times speed improvement, (3minutes > 12seconds) this is due to as you said, all of boost would only get compiled once per module, and there would be only 1 module. With GCC we saw no benefit, but Visual studio compiler loved this, obviously you have to solve any global naming conflicts and issues with multiple defines, but you shouldn't have these anyway. and it's good to clean them up.

[obstinate]

[*] Are you persisting intermediates across branches where possible?

Yes, but when I switch branches, the probability of changing something in common.hpp is big. Also, the visual studio tentds to be buggy when checking for changed depencencies, so often I just recompile to be sure it won't be broken.

Well, here we have two problems.

One: it's so surprising that Visual Studio in this day and age would require full rebuilds for correctness. It's been years since I issued a clean to get a correct binary. It would be interesting to see if you'd get better results with Bazel.

Two: you can output intermediates to a directory that varies with the git branch you're inspecting.

Code: Select all

.artifacts/$(git branch | sed -n '/\* /s///p')/...

or something like that (credit to SO for the one-liner). This would prevent the need for rebuilding. The only downside is that you might have many copies of similar intermediates in each branch, but you could use a cron job to clean up the older intermediates nightly.

[kovarex]

At: Compilation times (C++ and technical)

You can try compiling the project as 1 .cpp file with all others included.

We started doing this at my work. Depending on your compiler, we saw a 15 times speed improvement, (3minutes > 12seconds) this is due to as you said, all of boost would only get compiled once per module, and there would be only 1 module. With GCC we saw no benefit, but Visual studio compiler loved this, obviously you have to solve any global naming conflicts and issues with multiple defines, but you shouldn't have these anyway. and it's good to clean them up.

This doesn't speed it up for us, the paralel build wins here.
The only thing that might help is partial unity build. Grouping files to groups of 10 or more, so it is still paralell.

[*] Are you persisting intermediates across branches where possible?

Yes, but when I switch branches, the probability of changing something in common.hpp is big. Also, the visual studio tentds to be buggy when checking for changed depencencies, so often I just recompile to be sure it won't be broken.

Well, here we have two problems.

One: it's so surprising that Visual Studio in this day and age would require full rebuilds for correctness. It's been years since I issued a clean to get a correct binary. It would be interesting to see if you'd get better results with Bazel.

Two: you can output intermediates to a directory that varies with the git branch you're inspecting.

Code: Select all

.artifacts/$(git branch | sed -n '/\* /s///p')/...

or something like that (credit to SO for the one-liner). This would prevent the need for rebuilding. The only downside is that you might have many copies of similar intermediates in each branch, but you could use a cron job to clean up the older intermediates nightly.

I actually tried to solve this, but without a good result:
http://stackoverflow.com/questions/2583 ... -be-depend

[bear21]

At: Compilation times (C++ and technical)

You can try compiling the project as 1 .cpp file with all others included.

This doesn't speed it up for us, the paralel build wins here.
The only thing that might help is partial unity build. Grouping files to groups of 10 or more, so it is still paralell.

That's interesting. Do you use visual studio compiler?

At my work we don't use boost though or any 3rd party libraries in fact. So I couldn't say what my experience with this technique and boost is.

[ratchetfreak]

At: Compilation times (C++ and technical)

You can try compiling the project as 1 .cpp file with all others included.

This doesn't speed it up for us, the paralel build wins here.
The only thing that might help is partial unity build. Grouping files to groups of 10 or more, so it is still paralell.

That's interesting. Do you use visual studio compiler?

At my work we don't use boost though or any 3rd party libraries in fact. So I couldn't say what my experience with this technique and boost is.

unity build forces a non-parallel build

if you have n modules (where n is the amount of cores) and a fast enough disk/enough RAM to cache the files in memory then I can see a gain simply because of the parallelization.

[krux02]

Three minutes, I am jealous. I wrote my master thesis in a project, where just the include of the dependency that I had to use increased by one minute. This compilation time was per object file. So when you liked to work with several cc files, don't do it anymore, each of them increases the compilation time by a minute. And regarding parallel builds, don't do it, because the dependency consumes so much memory, that the compilation would end up swapping and take even more time. You should feel lucky that you do not work with people who want to write as generic as possible, and keep every door open. c++ templates suck. C++ compilations sucks.

[cncr04s]

If I make a change, only the changed c++ file gets re-built, and it just relinks. Takes all of a few seconds. I have 1.1MB of code, not as much as factorio but..
header changes.. well depends on which header and how many other objects it effects.

[radu]

One thing that makes a huge difference with the monolithic builds (all cpp files in one) is the optimization level. As the compiler needs to regenerate all the machine code, you don't want to give it too much work to do. On one of my projects I had gcc -O0 at 4.5 seconds, gcc -O1 at 10 seconds and clang (I don't remember the flag but I assume minimal optimization) at 2.5 seconds.

The project I had was ~9500 lines of code as measured by sloccount - if we scale that up to the size of factorio the technique doesn't look too promising (although it's hard to tell how much is one-time overhead and whether the rest would scale linearly or worse). Just to get a quick idea if this shows any promise you could benchmark a monolithic clang -O0 on Linux. I imagine it would be easy since you already have a Linux build. It would also tell you whether it's worth looking into clang for Windows. clang is definitely faster than gcc but the ms compiler is probably not bad itself.

On the other hand, compiling as modules of 10k lines could work better. (you'd still pay the price of linking though but even that should be cheaper than linking 1000 separate objects)

It has been a while since I used visual studio so I'm not sure what is the best setup for the ms compiler optimization-wise.
That is of course if you are willing to use a completely unoptimized build for development.

[SHiRKiT]

For some reason the FFF-115 on the website says it was published on December 8th instead of the regular date.

[kinnom]

I've got two FFF15s

[Rein]

The sound box in this post got me thinking: I do quite a bit of voice work on the side and really love Factorio. If you're ever in need of some voice assets I'd be totally willing to contribute.

[Nugetto]

The switch looks like it will come in very useful!

Can't wait to see the second trailer also!

[ralgrado]

I'm probably a bit late to the party but I just wonder if having cycles with your power switches are an issue. Probably the first thing you tested but I was wodnering anyway.