Debug build low performance.

Forum for technical discussions regarding development. If you have a general suggestion, problem or comment, please use one of the other forums.

Moderator: OpenTTD Developers

lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Debug build low performance.

Post by lukasz1985 »

I'm trying to run debug build in VC++ Express 2010. But it seems to be very slow and eats whole core of my CPU. While this doesn't happen in release build - it takes years there to compile (actually - to link).

So is there anyway I can improve the performance of this?
Eddi
Tycoon
Tycoon
Posts: 8257
Joined: 17 Jan 2007 00:14

Re: Debug build low performance.

Post by Eddi »

i don't know about Visual Studio, but the makefile (for GCC) supports 3 different debug levels, which enable or disable certain optimizations.

(optimizations generally increase compile time, while decreasing runtime. but optimized code may be more difficult to debug, as the reference to the source code may get lost)
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

How long does it take to link on each of this levels? In VS it's 5minutes on my machine (Core 2 Duo E8200 2x2.66GHZ) using release config. I know that things may differ between PC's but taking some benchmarks into account, maybe I could predict how it would look like here.

(well I guess, that I'll be using GCC for this anyway)
User avatar
doktorhonig
Tycoon
Tycoon
Posts: 1104
Joined: 22 Aug 2006 11:03
Location: Austria
Contact:

Re: Debug build low performance.

Post by doktorhonig »

If you want to debug, do a debug build, otherwise don't. If you actually want to play the game, do a release build. If you watch cat videos on youtube, 5 minutes feel like an instant.
Eddi
Tycoon
Tycoon
Posts: 8257
Joined: 17 Jan 2007 00:14

Re: Debug build low performance.

Post by Eddi »

5 minutes sounds like a quite short compilation time, tbh.
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Debug build low performance.

Post by JGR »

5 minutes sounds on the long side to me, but then again I usually compile using `make -j4`.
I don't think I've ever built a debug build of OpenTTD. Even if you are trying to debug an issue the release builds are almost always adequate.

If you're using MSVC you won't be using GCC.
Ex TTDPatch Coder
Patch Pack, Github
User avatar
adf88
Chief Executive
Chief Executive
Posts: 644
Joined: 14 Jan 2008 15:51
Location: PL

Re: Debug build low performance.

Post by adf88 »

If you disable "Whole Program Optimization" (in project options or by /GL- switch), linking will be almost instant. Is it worth of doing? Not really. Your OpenTTD, perhaps fast enough, will be eating more electricity. Save the planet and make a full release! :lol:
:] don't worry, be happy and checkout my patches
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

I'll move to gcc but still my curiosity about how to get to the middle point between link time and runtime performance in VC++ remains.
I need a way to be able to change something in code, compile and link in a reasonable time (1minute max) with a performance that at least wont cause such a slowdown as currently debug build causes.
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Debug build low performance.

Post by JGR »

Turning off link time optimisation ought to give an acceptable midpoint. MSVC also has some options for minimal and incremental builds which may be worth checking.
The configure script for GCC and similar does not enable link time optimisation by default.
Ex TTDPatch Coder
Patch Pack, Github
User avatar
adf88
Chief Executive
Chief Executive
Posts: 644
Joined: 14 Jan 2008 15:51
Location: PL

Re: Debug build low performance.

Post by adf88 »

Yes, it does, "whole program optimization" that I mentioned is something like an equivalent of GCC link time optimization.

Also from my experience I can say that compiling after small changes goes faster in MSVC then in GCC+ccache.

Worth of mentioning is also gold linker. It can be used with GCC. It's fast and suitable for (semi)debug builds. While being generally faster, in addition it can utilize multiple CPU cores (only x86-64 or x64, not availabale on x86). GCC linker is single-threaded, MSVC linker is multi-threaded.
:] don't worry, be happy and checkout my patches
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Debug build low performance.

Post by JGR »

adf88 wrote:Yes, it does, "whole program optimization" that I mentioned is something like an equivalent of GCC link time optimization.
The two terms are interchangeable in this instance as there is only one optimised link step which outputs the "whole program". To clarify my earlier post, in the case of MSVC, LTO/WPO is explicitly enabled by default for the release builds as per the generated project files and therefore explicitly switching it off is likely to improve incremental build times at the expense of runtime performance, and in the case of GCC/clang/etc. using the makefile LTO/WPO is not enabled by default as it requires the `--enable-lto` switch to be passed to `configure`.
adf88 wrote:Also from my experience I can say that compiling after small changes goes faster in MSVC then in GCC+ccache.
What do you gain by using ccache? make already does dependency tracking.
MSVC supports incremental compilation and linking (when LTO/WPO is not enabled) which GCC does not.
Ex TTDPatch Coder
Patch Pack, Github
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

>MSVC linker is multi-threaded.

Hmmm.. I didn't notice that. The task manager displays only half of my 2 core CPU usage.
User avatar
adf88
Chief Executive
Chief Executive
Posts: 644
Joined: 14 Jan 2008 15:51
Location: PL

Re: Debug build low performance.

Post by adf88 »

JGR wrote:What do you gain by using ccache? make already does dependency tracking.
make tracks file modification dates. ccache tracks preprocessed content of files. This is helpful sometimes (but it costs something too):
- when file modification date changes, but the file itself not, for whatsoever reason
- when adding changes that will be preprocessed out (e.g. comments)
I mentioned ccache because it's related to the issue.
lukasz1985 wrote:>MSVC linker is multi-threaded.

Hmmm.. I didn't notice that. The task manager displays only half of my 2 core CPU usage.
That depends on VS version. Also not whole process is paralleled. I found this discussion:
http://visualstudio.uservoice.com/forum ... c-c-linker
Marián Špánik wrote:When the Link time code generation (LTCG) is on the CPU gets to 100% when the linking is nearing its end - it seems that the code generation itself is multithreaded in Visual C++ 2013. I checked a Task Manager in Windows and new threads were spawned during this stage. However the linking itself (assigning the addresses to the global variables, functions, ets.) is still single-threaded.
:] don't worry, be happy and checkout my patches
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

It would be nice to know how much of the process is actually multithreaded for some real info, not just marketing "MSVC linker is multithreaded".
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Debug build low performance.

Post by JGR »

The actual link stage (after code generation is done) doesn't generally take very long and there isn't a huge amount to be gained by parallelisation.
For what it's worth GCC also supports parallelisation of link-time code generation, though it is multi-process rather than multi-threaded, and requires more hoop-jumping than MSVC (you need to pass -j to make and setup jobserver stuff in the makefile).

(I just tried this out and got some slightly alarming warnings from GCC, might have to have a look at this later...)

Edit: GCC has a known bug for spurious fortify source warnings when using -flto.
Ex TTDPatch Coder
Patch Pack, Github
User avatar
adf88
Chief Executive
Chief Executive
Posts: 644
Joined: 14 Jan 2008 15:51
Location: PL

Re: Debug build low performance.

Post by adf88 »

lukasz1985 wrote:It would be nice to know how much of the process is actually multithreaded for some real info, not just marketing "MSVC linker is multithreaded".
Smells like a M$ hater :) Come on, I wasn't market anything. I'm not on MSVC for years. This is how I remembered the linking process - it was utilizing multiple cores. Probably because most of the link time is taken by LTO, otherwise lining is almost instant so you don't even bother whether it is paralleled or not. Yes, I did generalize things, but don't accuse me of being biased. I already stated that not everything is paralleled.
JGR wrote:For what it's worth GCC also supports parallelisation of link-time code generation, though it is multi-process rather than multi-threaded, and requires more hoop-jumping than MSVC (you need to pass -j to make and setup jobserver stuff in the makefile).
Is it distcc or something else?
:] don't worry, be happy and checkout my patches
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

@adf88 Haha, I'm not a M$ hater, I'm universal hater! I'm cautious about things I hear, because lately I bought a SSD, and the marketing said it would 250Mb read and 250Mb write on my MotherBoard. While actually that this doesn't work for uncompressible data (which is the most part of the storage data) for which the write speed I'm getting is 60Mb (slower than 85Mb on my HDD)...


So yeah, it's better to be strict than disappointed. :)


@JGR I thought that anything multi-process can be split into multiple threads, or else what's the point of creating multiple processes? Or am I wrong somewhere here?
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Debug build low performance.

Post by JGR »

adf88 wrote:
JGR wrote:For what it's worth GCC also supports parallelisation of link-time code generation, though it is multi-process rather than multi-threaded, and requires more hoop-jumping than MSVC (you need to pass -j to make and setup jobserver stuff in the makefile).
Is it distcc or something else?
No, it's using -flto=jobserver and using the "+" prefix in the makefile. The effect is that gcc adds its own jobs to make's internal jobserver which handles all the parallelisation as per the -j switch.
The OpenTTD makefile includes it.
lukasz1985 wrote:@JGR I thought that anything multi-process can be split into multiple threads, or else what's the point of creating multiple processes? Or am I wrong somewhere here?
Getting things working correctly and performantly in a multi-threaded environment is more difficult than using multiple processes, because of all the potential for concurrency bugs, especially if the code was not originally written to be multi-threaded, which I suspect GCC was not.
On Windows, creating new processes is really expensive, which is why multi-threading seems to be preferred. On most Unixy platforms creating a new process is relatively cheap, so it's simpler to just create more processes unless you actually need to share mutable data between threads.
Ex TTDPatch Coder
Patch Pack, Github
lukasz1985
Route Supervisor
Route Supervisor
Posts: 429
Joined: 27 Mar 2013 08:58
Location: Strumien
Contact:

Re: Debug build low performance.

Post by lukasz1985 »

But AFAIK Windows automatically handles multiple processess by assigning them to different cores if it can. Isn't this a case for GCC? And if so doesn't this mean parallelism? Or are these processess somehow related to each other in time, by forcing some locks on each other, which they release after they complete (actually the first thing that comes to my mind are the file locks)?
Last edited by lukasz1985 on 23 Sep 2015 18:07, edited 1 time in total.
Eddi
Tycoon
Tycoon
Posts: 8257
Joined: 17 Jan 2007 00:14

Re: Debug build low performance.

Post by Eddi »

lukasz1985 wrote:Isn't this a case for GCC?
distributing processes/threads onto multiple cores happens in a lower layer of the OS, which neither MSVC nor GCC can and will care about.
Post Reply

Return to “OpenTTD Development”

Who is online

Users browsing this forum: No registered users and 18 guests