Compiling NewGRF very slow

[Transportman]

As the 2cc TrainsInNML is growing in size, the compile time is following and also increasing. Currently it takes 8-10 minutes on my system but the compile farm only takes 1.5 minute, so I would like to increase the speed of my system, as bug fixing becomes a bit tedious if I have to wait for that long to see if a change I made really works.

I now do the compiling in a VM with Fedora (following this guide), using

Code: Select all

nmlc --version
0.3.0.r2095:e72d25e21123
Library versions encountered:
PIL: 1.1.7
PLY: 3.3

I also have only 1 CPU core available for the VM (no hardware virtualization support) that is already quite busy when I'm doing nothing in the VM, so I suspect this slows the compilation down quite a bit.

I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows (without a VM)?

[planetmaker]

You basically need to install a MinGW environment which includes at least gcc, bash, make, sed, grep and the things I now likely forgot

[TadeuszD]

I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows?

I compile my set (Polish PKP Set 2.0) on Windows 7 every time. I hope that my post will be useful on Windows 8 too.
As planetmaker wrote, firstly you need to install MinGW environment in your Windows system. Next you should download NMLC executable for Windows ( http://bundles.openttdcoop.org/nml/nightlies/LATEST/ ). To optimize your work you should add GCC and NMLC directories to 'PATH' environment variable. After that you can compile your set using batch file, similar to my(http://dev.openttdcoop.org/projects/pkp ... try/cc.bat), or using make directive, of course...

[michael blunck]

Compiling NewGRF very slow

Python is slow.

make is not supported by Windows

Of course it is.

regards
Michael

[lukasz1985]

Haha, I liked the "Python is slow" comment. Correct accent in correct place, because python is extremaly slow.

As for an advice I would recomend you to separate your set into different GRF's. That's what I've done because of - you guess it - long compile times.
When you're finish you could create a whole new grf out of all the elements you are to include.

[Eddi]

of course, python is slow, but if you can still get orders of magnitude of speedups, you should invest in it anyway.

if you want to get really crazy, i have a way of splitting up my grf into one grf per vehicle, and then re-attaching them after the nml pass is finished. this actually makes compiling slower (as common header files must be processed multiple times), but it does allow using multiple cores. it may also be easy to break and might not fit your grf setup at all, and the nml devs do not endorse it.

[michael blunck]

if you want to get really crazy, i have a way of splitting up my grf into one grf per vehicle

Per vehicle? I remember you have hundreds of them?

m4nfo supports linking on newGRF level, that´s the real improvement.

I.e., for the DB Set, I have - one main file, containing switch and parameter handling, text strings, CTT, recolour maps, - one file for vehicle definitions, - one file for coaches, - one file for locomotives, - one file for freight car definitions, - one file for freight car implementation, - one file for AI handling (TTDPatch only), - one file for misc stuff like signals, gates, tunnels, etc. That´s a sufficient breakup.

regards
Michael

[Transportman]

of course, python is slow, but if you can still get orders of magnitude of speedups, you should invest in it anyway.

Indeed, the compile farm compiles 6 times faster than my current setup, so there is a huge room for improvement.

I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows?

I compile my set (Polish PKP Set 2.0) on Windows 7 every time. I hope that my post will be useful on Windows 8 too.
As planetmaker wrote, firstly you need to install MinGW environment in your Windows system. Next you should download NMLC executable for Windows ( http://bundles.openttdcoop.org/nml/nightlies/LATEST/ ). To optimize your work you should add GCC and NMLC directories to 'PATH' environment variable. After that you can compile your set using batch file, similar to my(http://dev.openttdcoop.org/projects/pkp ... try/cc.bat), or using make directive, of course...

That works, it just compiled the whole thing in less than a minute using your method, but I'm not completely happy. Some automatic replacements I use with the makefile are not replaced ({TITLE}, REPO_REVISION and others), but at least it shows that the improvement is possible.

Trying to compile using make gives me some errors. I first had to update my makefile, there was a bug that blocked compiling, grabbed findversion.sh that is needed, and I got a whole bunch of these errors:

Code: Select all

File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'

And then it fails with a

Code: Select all

Makefile:388: *** multiple target patterns.  Stop.

Compiling on the VM is a bit faster now, but still quite slow (from 8-10 to 5-6 minutes).

[planetmaker]

That works, it just compiled the whole thing in less than a minute using your method, but I'm not completely happy. Some automatic replacements I use with the makefile are not replaced ({TITLE}, REPO_REVISION and others), but at least it shows that the improvement is possible.

That's a bit more tricky to replicate. The repository revision is queried - and based on that result the custom_tags.txt is written. If it's just for you testing the build and not worry so much about the correct version, fill in fantasy values for those custom tags in your custom_tags.txt

Trying to compile using make gives me some errors. I first had to update my makefile, there was a bug that blocked compiling, grabbed findversion.sh that is needed, and I got a whole bunch of these errors:

Code: Select all

File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'

And then it fails with a

Code: Select all

Makefile:388: *** multiple target patterns.  Stop.

Compiling on the VM is a bit faster now, but still quite slow (from 8-10 to 5-6 minutes).

The version determination needs python and mercurial in your %PATH% variable in order to successfully run the ./findversion bash script. Probably the call of

Code: Select all

hg log -rtip --template='{time|shortdate}'

failed / fails on your command line as well?

Not sure about the successive error in Makefile:388 - but might be result of the previous one.

What made compilation in your VM suddenly faster by a factor of two?

[Transportman]

The version determination needs python and mercurial in your %PATH% variable in order to successfully run the ./findversion bash script.

Python, mercurial, NML, MinGW and MSYS are all in the %PATH% variable. I tried just running the script from the command line using

Code: Select all

sh findversion.sh

it gives the same error, with some additional lines:

Code: Select all

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'
15255092bc88    5267 days, 0:00:00      M               v5267 days, 0:00:00M (15
255092bc88)

Probably the call of

Code: Select all

hg log -rtip --template='{time|shortdate}'

failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.

[planetmaker]

Probably the call of

Code: Select all

hg log -rtip --template='{time|shortdate}'

failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.

Hm, so does it help, if you change in your Makefile:143 (as in https://rhodecode.openttdcoop.org/2ccts ... efile#L143) the ' to " then?

[Transportman]

Probably the call of

Code: Select all

hg log -rtip --template='{time|shortdate}'

failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.

Hm, so does it help, if you change in your Makefile:143 (as in https://rhodecode.openttdcoop.org/2ccts ... efile#L143) the ' to " then?

Locally I have now version r109 of the Makefile, which does not contain that line, but findversion.sh does have that line. Changing it does not show any improvement, I still can't compile it on Windows, with the same errors as before.

The line that throws the error from findversion.sh is line 97:

Code: Select all

VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1)).days"`

[EDIT]If I remove the .days part from that line, which is not really a solution, there are no warnings, but the set does not compile, with the Makefile:388 error I gave earlier[/EDIT]

[planetmaker]

Maybe this patch helps. It seems that the REPO_DATE variable is not properly set. Otherwise I don't see how (date(2014,6,5)-date(2000,1,1)).days is of type 'None'.

Code: Select all

diff --git a/findversion.sh b/findversion.sh
--- a/findversion.sh
+++ b/findversion.sh
@@ -93,7 +93,7 @@
 	REV="h`echo $HASH | cut -c1-8`"
 	BRANCH="`hg branch | sed 's@^default$@@'`"
 	TAG="`hg id -t | grep -v 'tip$'`"
-	REPO_DATE="`hg log -r$HASH --template='{date|shortdate}' | sed s/-/,/g | sed s/,0/,/g`"
+	REPO_DATE="`hg log -r$HASH --template=\"{date|shortdate}\" | sed s/-/,/g | sed s/,0/,/g`"
 	VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1)).days"`
 	DISPLAY_VERSION="v${VERSION}"
 	if [ -n "$TAG" ]; then

[Transportman]

Unfortunately it doesn't work. Is there a command which I can use to show the type of the variables?

[Eddi]

you can use the "-i" switch on python to enter an interactive mode where you can view the results after the initial command/script has finished.

what's your python version, anyway?

[Transportman]

you can use the "-i" switch on python to enter an interactive mode where you can view the results after the initial command/script has finished.

Does that also work for .sh scripts and/or python commands in those scripts?

what's your python version, anyway?

Python 3.3.5.

[EDIT]I think I got one step closer, I don't get any AttributeErrors any more, but compiling still fails:

Code: Select all

[CPP] 2ccts.nml
[LNG] custom_tags.txt
[NML] 2ccts.grf
/bin/bash: : command not found
make: *** [grf] Error 127

I know it is ugly code, but I changed the VERSION= line from findversion.sh to

Code: Select all

	VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1))"`
	VERSION="`echo $VERSION | sed s/' days, 0:00:00'//g`"

[/EDIT]

[Transportman]

Apparently the shell doesn't recognize the nmlc command. I do have the Windows version installed on my computer, but I guess bash wants the Linux-version. Do I need to do anything special for that, or can I just do a repository checkout and add that to my Windows PATH variable?

[planetmaker]

I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

[Transportman]

I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

That is indeed it, if I add the complete path to my nmlc in the Makefile it does compile, in something like 3 minutes. Now to figure out how to add the path to the bash shell, so that I can just leave the current hack out.

[planetmaker]

I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

That is indeed it, if I add the complete path to my nmlc in the Makefile it does compile, in something like 3 minutes. Now to figure out how to add the path to the bash shell, so that I can just leave the current hack out.

there probably is somewhere a .bashrc or a bash.profile file.

Maybe you want to write down your experiences, the way to get the compile environment on windows going? As a note for yourself and others to profit from it?
http://dev.openttdcoop.org/projects/home/wiki could well use that addition