Transport Tycoon Forums

The place to talk about Transport Tycoon
It is currently Wed Apr 25, 2018 12:49 pm

All times are UTC




Post new topic  Reply to topic  [ 33 posts ]  Go to page 1 2 Next
Author Message
PostPosted: Wed Jun 04, 2014 1:02 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
As the 2cc TrainsInNML is growing in size, the compile time is following and also increasing. Currently it takes 8-10 minutes on my system but the compile farm only takes 1.5 minute, so I would like to increase the speed of my system, as bug fixing becomes a bit tedious if I have to wait for that long to see if a change I made really works.

I now do the compiling in a VM with Fedora (following this guide), using
Code:
nmlc --version
0.3.0.r2095:e72d25e21123
Library versions encountered:
PIL: 1.1.7
PLY: 3.3

I also have only 1 CPU core available for the VM (no hardware virtualization support) that is already quite busy when I'm doing nothing in the VM, so I suspect this slows the compilation down quite a bit.

I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows (without a VM)?

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Wed Jun 04, 2014 1:14 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
You basically need to install a MinGW environment which includes at least gcc, bash, make, sed, grep and the things I now likely forgot :)

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
PostPosted: Wed Jun 04, 2014 2:13 pm 
Offline
Transport Coordinator
Transport Coordinator

Joined: Mon Nov 07, 2011 7:32 pm
Posts: 319
Location: PL
Transportman wrote:
I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows?

I compile my set (Polish PKP Set 2.0) on Windows 7 every time. I hope that my post will be useful on Windows 8 too. :wink:
As planetmaker wrote, firstly you need to install MinGW environment in your Windows system. Next you should download NMLC executable for Windows ( http://bundles.openttdcoop.org/nml/nightlies/LATEST/ ). To optimize your work you should add GCC and NMLC directories to 'PATH' environment variable. After that you can compile your set using batch file, similar to my(http://dev.openttdcoop.org/projects/pkp ... try/cc.bat), or using make directive, of course... ;)

_________________
Image


Last edited by TadeuszD on Wed Jun 04, 2014 2:19 pm, edited 2 times in total.

Top
   
PostPosted: Wed Jun 04, 2014 2:14 pm 
Offline
Tycoon
Tycoon

Joined: Wed Apr 27, 2005 7:09 am
Posts: 5155
Transportman wrote:
Compiling NewGRF very slow

Python is slow.

Transportman wrote:
make is not supported by Windows

Of course it is.

regards
Michael

_________________
Image


Top
   
PostPosted: Wed Jun 04, 2014 3:37 pm 
Offline
Transport Coordinator
Transport Coordinator

Joined: Wed Mar 27, 2013 8:58 am
Posts: 359
Haha, I liked the "Python is slow" comment. Correct accent in correct place, because python is extremaly slow.

As for an advice I would recomend you to separate your set into different GRF's. That's what I've done because of - you guess it - long compile times.
When you're finish you could create a whole new grf out of all the elements you are to include.


Top
   
PostPosted: Wed Jun 04, 2014 10:06 pm 
Offline
Tycoon
Tycoon

Joined: Wed Jan 17, 2007 12:14 am
Posts: 6985
of course, python is slow, but if you can still get orders of magnitude of speedups, you should invest in it anyway.

if you want to get really crazy, i have a way of splitting up my grf into one grf per vehicle, and then re-attaching them after the nml pass is finished. this actually makes compiling slower (as common header files must be processed multiple times), but it does allow using multiple cores. it may also be easy to break and might not fit your grf setup at all, and the nml devs do not endorse it.

_________________
You might not exactly be interested in Ferion, but if you are, have fun :)


Top
   
PostPosted: Thu Jun 05, 2014 6:04 am 
Offline
Tycoon
Tycoon

Joined: Wed Apr 27, 2005 7:09 am
Posts: 5155
Eddi wrote:
if you want to get really crazy, i have a way of splitting up my grf into one grf per vehicle

Per vehicle? I remember you have hundreds of them? :cool:

m4nfo supports linking on newGRF level, that´s the real improvement.

I.e., for the DB Set, I have - one main file, containing switch and parameter handling, text strings, CTT, recolour maps, - one file for vehicle definitions, - one file for coaches, - one file for locomotives, - one file for freight car definitions, - one file for freight car implementation, - one file for AI handling (TTDPatch only), - one file for misc stuff like signals, gates, tunnels, etc. That´s a sufficient breakup.

regards
Michael

_________________
Image


Top
   
PostPosted: Thu Jun 05, 2014 7:55 am 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
Eddi wrote:
of course, python is slow, but if you can still get orders of magnitude of speedups, you should invest in it anyway.

Indeed, the compile farm compiles 6 times faster than my current setup, so there is a huge room for improvement.

TadeuszD wrote:
Transportman wrote:
I want to try compiling on Windows 8 (the OS my laptop actually runs) to see if that is a bit faster, but make is not supported by Windows. So I was wondering if it is possible to do and if there is a step by step guide on how to do it and which programs to install to compile NewGRFs using the make system on Windows?

I compile my set (Polish PKP Set 2.0) on Windows 7 every time. I hope that my post will be useful on Windows 8 too. :wink:
As planetmaker wrote, firstly you need to install MinGW environment in your Windows system. Next you should download NMLC executable for Windows ( http://bundles.openttdcoop.org/nml/nightlies/LATEST/ ). To optimize your work you should add GCC and NMLC directories to 'PATH' environment variable. After that you can compile your set using batch file, similar to my(http://dev.openttdcoop.org/projects/pkp ... try/cc.bat), or using make directive, of course... ;)

That works, it just compiled the whole thing in less than a minute using your method, but I'm not completely happy. Some automatic replacements I use with the makefile are not replaced ({TITLE}, REPO_REVISION and others), but at least it shows that the improvement is possible.

Trying to compile using make gives me some errors. I first had to update my makefile, there was a bug that blocked compiling, grabbed findversion.sh that is needed, and I got a whole bunch of these errors:
Code:
File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'


And then it fails with a
Code:
Makefile:388: *** multiple target patterns.  Stop.


Compiling on the VM is a bit faster now, but still quite slow (from 8-10 to 5-6 minutes).

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Thu Jun 05, 2014 8:23 am 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
Transportman wrote:
That works, it just compiled the whole thing in less than a minute using your method, but I'm not completely happy. Some automatic replacements I use with the makefile are not replaced ({TITLE}, REPO_REVISION and others), but at least it shows that the improvement is possible.

That's a bit more tricky to replicate. The repository revision is queried - and based on that result the custom_tags.txt is written. If it's just for you testing the build and not worry so much about the correct version, fill in fantasy values for those custom tags in your custom_tags.txt

Quote:
Trying to compile using make gives me some errors. I first had to update my makefile, there was a bug that blocked compiling, grabbed findversion.sh that is needed, and I got a whole bunch of these errors:
Code:
File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'


And then it fails with a
Code:
Makefile:388: *** multiple target patterns.  Stop.


Compiling on the VM is a bit faster now, but still quite slow (from 8-10 to 5-6 minutes).

The version determination needs python and mercurial in your %PATH% variable in order to successfully run the ./findversion bash script. Probably the call of
Code:
hg log -rtip --template='{time|shortdate}'
failed / fails on your command line as well?

Not sure about the successive error in Makefile:388 - but might be result of the previous one.

What made compilation in your VM suddenly faster by a factor of two?

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
PostPosted: Thu Jun 05, 2014 11:17 am 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
planetmaker wrote:
The version determination needs python and mercurial in your %PATH% variable in order to successfully run the ./findversion bash script.

Python, mercurial, NML, MinGW and MSYS are all in the %PATH% variable. I tried just running the script from the command line using
Code:
sh findversion.sh
it gives the same error, with some additional lines:
Code:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'days'
15255092bc88    5267 days, 0:00:00      M               v5267 days, 0:00:00M (15
255092bc88)


Quote:
Probably the call of
Code:
hg log -rtip --template='{time|shortdate}'
failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

Quote:
What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Thu Jun 05, 2014 12:04 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
Transportman wrote:
planetmaker wrote:
Probably the call of
Code:
hg log -rtip --template='{time|shortdate}'
failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

Quote:
What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.


Hm, so does it help, if you change in your Makefile:143 (as in https://rhodecode.openttdcoop.org/2ccts ... efile#L143) the ' to " then?

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
PostPosted: Thu Jun 05, 2014 1:05 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
planetmaker wrote:
Transportman wrote:
planetmaker wrote:
Probably the call of
Code:
hg log -rtip --template='{time|shortdate}'
failed / fails on your command line as well?

When I change the ' to " it works, the result is 2014-06-03.

Quote:
What made compilation in your VM suddenly faster by a factor of two?

The only change I made was the makefile I used, I had a very outdated one.


Hm, so does it help, if you change in your Makefile:143 (as in https://rhodecode.openttdcoop.org/2ccts ... efile#L143) the ' to " then?

Locally I have now version r109 of the Makefile, which does not contain that line, but findversion.sh does have that line. Changing it does not show any improvement, I still can't compile it on Windows, with the same errors as before.

The line that throws the error from findversion.sh is line 97:
Code:
VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1)).days"`


[EDIT]If I remove the .days part from that line, which is not really a solution, there are no warnings, but the set does not compile, with the Makefile:388 error I gave earlier[/EDIT]

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Thu Jun 05, 2014 1:25 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
Maybe this patch helps. It seems that the REPO_DATE variable is not properly set. Otherwise I don't see how (date(2014,6,5)-date(2000,1,1)).days is of type 'None'.
Code:
diff --git a/findversion.sh b/findversion.sh
--- a/findversion.sh
+++ b/findversion.sh
@@ -93,7 +93,7 @@
    REV="h`echo $HASH | cut -c1-8`"
    BRANCH="`hg branch | sed 's@^default$@@'`"
    TAG="`hg id -t | grep -v 'tip$'`"
-   REPO_DATE="`hg log -r$HASH --template='{date|shortdate}' | sed s/-/,/g | sed s/,0/,/g`"
+   REPO_DATE="`hg log -r$HASH --template=\"{date|shortdate}\" | sed s/-/,/g | sed s/,0/,/g`"
    VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1)).days"`
    DISPLAY_VERSION="v${VERSION}"
    if [ -n "$TAG" ]; then

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
PostPosted: Thu Jun 05, 2014 1:41 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
Unfortunately it doesn't work. Is there a command which I can use to show the type of the variables?

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Thu Jun 05, 2014 5:37 pm 
Offline
Tycoon
Tycoon

Joined: Wed Jan 17, 2007 12:14 am
Posts: 6985
you can use the "-i" switch on python to enter an interactive mode where you can view the results after the initial command/script has finished.

what's your python version, anyway?

_________________
You might not exactly be interested in Ferion, but if you are, have fun :)


Top
   
PostPosted: Thu Jun 05, 2014 6:22 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
Eddi wrote:
you can use the "-i" switch on python to enter an interactive mode where you can view the results after the initial command/script has finished.

Does that also work for .sh scripts and/or python commands in those scripts?

Quote:
what's your python version, anyway?

Python 3.3.5.

[EDIT]I think I got one step closer, I don't get any AttributeErrors any more, but compiling still fails:
Code:
[CPP] 2ccts.nml
[LNG] custom_tags.txt
[NML] 2ccts.grf
/bin/bash: : command not found
make: *** [grf] Error 127


I know it is ugly code, but I changed the VERSION= line from findversion.sh to
Code:
   VERSION=`python -c "from datetime import date; print (date($REPO_DATE)-date(2000,1,1))"`
   VERSION="`echo $VERSION | sed s/' days, 0:00:00'//g`"
[/EDIT]

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Sat Jun 07, 2014 3:48 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
Apparently the shell doesn't recognize the nmlc command. I do have the Windows version installed on my computer, but I guess bash wants the Linux-version. Do I need to do anything special for that, or can I just do a repository checkout and add that to my Windows PATH variable?

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Sat Jun 07, 2014 4:20 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
PostPosted: Sat Jun 07, 2014 6:55 pm 
Offline
Tycoon
Tycoon

Joined: Tue Feb 22, 2011 6:34 pm
Posts: 2627
planetmaker wrote:
I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

That is indeed it, if I add the complete path to my nmlc in the Makefile it does compile, in something like 3 minutes. Now to figure out how to add the path to the bash shell, so that I can just leave the current hack out.

_________________
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML


Top
   
PostPosted: Sat Jun 07, 2014 6:57 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Wed Nov 07, 2007 10:44 pm
Posts: 9025
Location: Sol d
Transportman wrote:
planetmaker wrote:
I would not expect a difference in the linux of windows version there. But it simply might be that the path is not defined within the shell.. And as a result your windows nmlc is not found.
Maybe you can add the nmlc search path to the default path definiton of that shell, too.

That is indeed it, if I add the complete path to my nmlc in the Makefile it does compile, in something like 3 minutes. Now to figure out how to add the path to the bash shell, so that I can just leave the current hack out.


there probably is somewhere a .bashrc or a bash.profile file.

Maybe you want to write down your experiences, the way to get the compile environment on windows going? As a note for yourself and others to profit from it?
http://dev.openttdcoop.org/projects/home/wiki could well use that addition :)

_________________
Image
OpenTTD: manual | online content | translations | Wanted contributions and patches
#openttdcoop: blog | wiki | public server | DevZone | NewGRF web translator
DevZone - home of the free NewGRFs: OpenSFX | OpenMSX | OpenGFX | Swedish Rails | OpenGFX+ Trains|RV|Industries|Airports|Landscape | NML


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 33 posts ]  Go to page 1 2 Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000-2018 phpBB Limited

Copyright © Owen Rudge/The Transport Tycoon Forums 2001-2018.
Hosted by Zernebok Hosting.