NML + pypy

Discussions about the technical aspects of graphics development, including NewGRF tools and utilities.

Moderator: Graphics Moderators

Post Reply
User avatar
andythenorth
Tycoon
Tycoon
Posts: 5656
Joined: 31 Mar 2007 14:23
Location: Lost in Music

NML + pypy

Post by andythenorth »

I have been testing the performance of various python versions with nmlc.

Python 3.8 is a recent release, and has performance improvements over previous Python 3.x versions.
Meanwhile PyPy is an alternative Python interpreter built for speed.

Testing with Iron Horse and FIRS:
  • Python 3.8 is often 10%-20% faster than Python 3.5
  • PyPY is up to 50% faster than Python 3.5 (i.e. it completes an nml run in half the time)
The benefits will vary by project, but it's worth exploring.

Not everything is faster with PyPy. Compiling docs and generating images are slower for me with PyPy than Python 3.8, but that's my code not nmlc.

For more on PyPy: http://pypy.org/
Alberth
OpenTTD Developer
OpenTTD Developer
Posts: 4763
Joined: 09 Sep 2007 05:03
Location: home

Re: NML + pypy

Post by Alberth »

Somewhat off-topic, but I ran recently into https://nuitka.net/pages/overview.html It compiles python to C/exe. May be worth a try.
Being a retired OpenTTD developer does not mean I know what I am doing.
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: NML + pypy

Post by planetmaker »

Alberth wrote: 01 Dec 2019 20:34 Somewhat off-topic, but I ran recently into https://nuitka.net/pages/overview.html It compiles python to C/exe. May be worth a try.
Interesting find, thanks! I shall have a look as time allows, especially when DevZone is overhauled.

EDIT: a quick tests seems to indicate that this might fly:

Code: Select all

[planetmaker:~/ottd/grfdev/nml] a6ce9f8+* ± nuitka --python-version=3.5 nml/main.py
Nuitka:WARNING:Not recursing to 'nml' (/home/planetmaker/ottd/grfdev/nml/nml), please specify --recurse-none (do not warn), --recurse-all (recurse to all), --recurse-not-to=nml (ignore it), --recurse-to=nml (recurse to it) to change.
Nuitka:WARNING:Not recursing to 'nml.actions' (/home/planetmaker/ottd/grfdev/nml/nml/actions), please specify --recurse-none (do not warn), --recurse-all (recurse to all), --recurse-not-to=nml.actions (ignore it), --recurse-to=nml.actions (recurse to it) to change.
Nuitka:WARNING:Not recursing to 'nml.ast' (/home/planetmaker/ottd/grfdev/nml/nml/ast), please specify --recurse-none (do not warn), --recurse-all (recurse to all), --recurse-not-to=nml.ast (ignore it), --recurse-to=nml.ast (recurse to it) to change.
Nuitka:WARNING:nml/main.py:22: Cannot find 'PIL' as absolute import.
Nuitka:WARNING:nml/main.py:25: Cannot find 'Image' as absolute import.

⌂67% [planetmaker:~/ottd/grfdev/airportsplus] default ☿ ../nml/main.exe ogfx-airports.nml
 nmlc info: 478 sprites, 0 cached, 384 orphaned, 94 duplicates, 384 newly encoded (native)
 nmlc info: Airport items: 14/128
 nmlc info: Airport Tile items: 147/256
 nmlc info: Concurrent Action10 labels: 1/240 ("sprites/header.pnml", line 8)
 nmlc info: Concurrent spritesets: 9/255 ("sprites/airport_tiles.pnml", line 218)
 nmlc info: Concurrent spritegroups: 81/256 ("sprites/airport_tiles.pnml", line 421)
 nmlc info: Concurrent Action2 registers: 29/127 ("sprites/airport_modern_small.pnml", line 42)
 nmlc info: Concurrent ActionD registers: 27/64 ("sprites/airport_tiles.pnml", line 11)
 nmlc info: GRF parameter registers: 3/64
 nmlc info: D0xx strings: 4/1024
 nmlc info: DCxx strings: 5/2048
2& [planetmaker:~/ottd/grfdev/nml] a6ce9f8+* 1 ± nuitka --version
0.5.25
Transportman
Tycoon
Tycoon
Posts: 2781
Joined: 22 Feb 2011 18:34

Re: NML + pypy

Post by Transportman »

planetmaker wrote: 02 Dec 2019 09:22
Alberth wrote: 01 Dec 2019 20:34 Somewhat off-topic, but I ran recently into https://nuitka.net/pages/overview.html It compiles python to C/exe. May be worth a try.
Interesting find, thanks! I shall have a look as time allows, especially when DevZone is overhauled.
Bit off-topic, but are there plans for that?
Coder of the Dutch Trackset | Development support for the Dutch Trainset | Coder of the 2cc TrainsInNML
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: NML + pypy

Post by planetmaker »

Transportman wrote: 05 Dec 2019 18:21
planetmaker wrote: 02 Dec 2019 09:22 Interesting find, thanks! I shall have a look as time allows, especially when DevZone is overhauled.
Bit off-topic, but are there plans for that?
Yes, there are, yet I won't give an exact timeframe.

Likely we shall move the repositories to github but keep a way to build NewGRFs and providing resulting builds.
User avatar
andythenorth
Tycoon
Tycoon
Posts: 5656
Joined: 31 Mar 2007 14:23
Location: Lost in Music

Re: NML + pypy

Post by andythenorth »

I tested PyPy some more with FIRS (all times are total run time for the makefile, with rounding):

Python3.8, empty caches: 55s
Python3.8, primed caches: 40s

PyPy, empty caches: 25s
PyPy, primed caches: 15s

FIRS uses PyPy specifically for the nmlc stage in the compile. The FIRS compile has other Python steps which are faster with Python 3.8, so the Makefile switches interpreter for different stages.

Conclusion: any nml project should see a substantial speed increase from switching to PyPy.

https://pypy.org/download.html

Thanks to FLHerne for pointing me to nmlc performance with PyPy :)
Post Reply

Return to “NewGRF Technical Discussions”

Who is online

Users browsing this forum: No registered users and 3 guests