Templating NML with Python - the non-comprehensive guide...

[andythenorth]

Introduction

Want to create newgrfs? NML is very elegant and very easy to pickup. This will not be a guide to NML.
Find a guide here: http://newgrf-specs.tt-wiki.net/wiki/NML:Main


Do I need this?
For those creating a small set you'll do fine with plain NML. You should consider a code repository (git), and you might want to try using a makefile to build your grf.

For those who create larger sets, templating repetitive code soon starts to become a concern. Templating reduces hassles like maintenance, formatting, debugging, keeping up with changes in NML spec etc.

A 'gotcha' for NFO authors
[Skip this if you're not used to working with complex NFO]

For those who are used to working with large NFO sets, there's one of aspect of NML which can be filed under 'gotcha': you can't repeat arbitrary blocks of varaction 2. Switch identifiers in NML must be unique. Try it and you'll see what I mean. It's not a showstopper, but means some common design and templating patterns can't be used.

NML, templates and YOU
NML lends itself well to templating. You can probably bend [template language of choice] to fit.

One proven option is the C Pre-Processor, providing constants, file includes, and using variadic macros to construct switch identifiers and other useful things. http://en.wikipedia.org/wiki/C_preprocessor

Another option is the macro language m4. This is proven for use with NFO, and will probably work well with NML. http://en.wikipedia.org/wiki/M4_(computer_language)
http://www.ttdpatch.de/grfspecs/m4nfoMa ... Intro.html

For various reasons I opted to try templating with Python. I think the results are interesting.

Not a comprehensive 'how-to'
I'm not a good teacher. I can't write a comprehensive 'how-to' guide. But I can share some tips, tricks and ideas for approaching NML templating with Python; posts in this thread will do just that.

[andythenorth]

Basics

Know python? Skip this bit.

1. You'll need python. You may have it (see step 4). If you can't figure out googling python and installing it, you're probably not going to have much fun with the rest of this thread. I'd try something else

2. My stuff has been done with python 2.6.1 You probably need a version not too far ahead or behind that.

3. Using Windows? I have no idea how you do stuff on Windows. This can be done on Windows, but details of how you setup your development environment differ. Examples should work on *nix systems. Mine's OS X.

4. Type 'python' into a shell/terminal/prompt/whatever. You should get the python interpreter. Type 'import this', you'll get poetry. Type 'ctrl-D'...you're done with the python interpreter for now.

5. Open a file in a (plain) text editor. Type 'import this' into the file. Save it as 'test.py'. In your shell/terminal/prompt/whatever, navigate to the place you saved test.py. Type 'python test.py'. If you see poetry, you win. If you don't, I screwed up or you did.

6. Delete 'test.py' if you care about disk space. You're done with it. Otherwise leave it to rot naturally.

7. Shell = shell/terminal/prompt/whatever from now on.

8. You are now a python programmer. Congratulations. If you'd like to actually learn the language, there are books and stuff.

What next?

Might as well install NML and dependencies and such if you don't have them
http://newgrf-specs.tt-wiki.net/wiki/NM ... ng_started

It's nice to idea of how NML works, and useful to have a project to try templating. So maybe do the NML tutorial, and get a few pngs with sprites in and things (use coloured blocks if you can't draw).

Tutorial is here: http://www.tt-wiki.net/wiki/NMLTutorial

So you've coded a working NML-based newgrf and have it in the game?
Congatulations Put a copy of the project files somewhere safe and...make some tea. (This assumes you like tea, most wise people do).

[oberhümer]

<x>

[andythenorth]

And now we will run off a cliff
Everything from here will go better if you understand enough python to know how to use lists, dicts, tuples, iterators, vars, and have at least a rough understanding of objects, functions, imports and namespaces. If you don't know that, you won't learn it well from me. I just point and wave my arms a lot.

Also - about a week before I wrote this I didn't know much about objects, file I/O, enumerate iterators, os.path, imports, or python globals either - but I do know how to use google, which finds this: http://docs.python.org/ (I have no training in compsci, except I once failed a course in Pascal. I also failed a course in Excel).

Rocks at the bottom

The basic idea of templating is to create reusable blocks of code that can then be customised per vehicle, house, industry, object etc. These templates are then compiled to one .nml file that can be turned into a newgrf by nmlc.

By reusing code, projects are easier to develop and easier to maintain. The first newgrf I wrote (HEQS) didn't use templating, and was just one very long NFO file. This was hard to work with. With help from others, I started using templating for NFO. Good templating wins.

A few things are of interest when templating:

1. Substitution of values. If you've used NML built-ins like 'ALL_CLIMATES' or 'VEHICLE_NEVER_EXPIRES', then you've used value substitution (maybe without knowing it). Value substitution is useful for many things, like setting identifiers (for items and switches) and item properties (e.g. speed, cargo refits, vehicle life, etc).

2. Macros / code reuse from pre-defined blocks of code, inserted into a file as needed. Parameters can be passed to the macro for use inside the macro. Macros are flexible way to reuse code. NML already permits this with sprite templates.

3. Dynamic code generation. An example is switch blocks that may have different checks depending on the vehicle type. These can be generated in the code with a 'for' loop. Generators can use several approaches, including calling macros, or composing code from raw strings.

Caveats: complex templating can be worse than no templating at all. It would be easy to spend more time maintaining a template system than just writing out plain NML in one large file.

Particularly, excessive code generation mixed with templates can be un-fun. When working with code generation, you need to be able to visualise what it will do for many different cases, and/or check output constantly. You're also working with two (or more) languages combined in the same file, which can lead to a lot of easy syntax mistakes, hassle dealing with string escapes and general head scratching. Code generation is great for those who know how to write a proper compiler (hi Eddi), but might not be fun for all.

In my other life I've written hundreds of templates for web pages and simple web apps. These combine xhtml, css and javascript with dynamic content. Learning the hard way, I now prefer to keep templates as dumb as possible.

It's no different for an NML project: quite-dumb templating means that the NML is easy to read, check, and debug. Debugging templating systems is dull.

General approach (no code yet)

To get 'stuff' into a newgrf, it's useful to have some idea of a pipeline.

For plain NML, the pipeline is .nml file -> nmlc -> newgrf. (This ignores the pngs etc, code is the significant factor here).

For templating with python we add to the start of our pipeline. The goal is to combine some data with one or more templates, and arrive at the .nml file to feed to nmlc. There are many ways to build a pipeline. CETS (mentioned above) does something like this:
Google Docs -> .tsv file -> python -> .pnml -> C Pre-processor -> .nml file

Another way would be to do:
One giant python file defining data and templates -> .nml file

I've been testing python templating for the BANDIT truck set, for which the pipeline is:
web forms -> database -> python -> output giant dict as text in browser -> copy and paste to plain text file -> more python -> template files -> .nml
Which is kind of overkill, and not at all to be advocated.

Generally an ideal pipeline looks like:
simple data file -> simple python -> simple template files -> .nml file

What's a good data file format?
Dunno. Not sure yet. Options include:
csv or tsv
- probably the most logical
- can be edited with spreadsheet app (including online apps) or text editor
- easy to parse (python has libraries for it)
- easy to share around between people

plain text - name/value pairs
- simple & robust
- edit with just about anything
- slightly a pain in the arse to maintain when adding new properties

row-based database - sqllite or similar
- can run sql queries or similar on these to add/remove/change properties
- can validate data types etc
- adds complexity and overhead

python dicts in text files - import as modules
- almost identical to NML 'item' format
- easy to understand
- pain in the arse to format and maintain when adding / removing properties (unless you know regexp etc)

JSON
- never used it, can't comment. Widely used and afaik has good python module support.

XML
- 'lol'
- way too much scaffolding for this problem. Any time I use XML, I end up templating it. Templating the data input format seems rather circular at best.

I concluded 'none of the above' and wrote a python object database in the Zope web framework, but the number of people likely to do that can be counted on the thumb of one hand. It also requires an odd copy-paste step (data could be fetched from the web, but then people without a web connection can't easily build the grf).

(Templating) weapons of choice

Python has long had a built in string templater. It uses a %s notation. I hate it.
But wait! Python has a better template strings library: http://docs.python.org/release/2.5.2/lib/node40.html

The built-in Python template strings library has a few neat features:
- it's easy to use
- it can't do much, so it's hard to make things too complicated when using it
- no dependencies need to be installed. If you have python > 2.5.2, it's just available

I tried it out, and it offers some useful features for NML authors who want simple templating. I'll post some examples later. However, I wanted a template library that could do a bit more. This means picking a library.

Python has lots of template libraries: http://wiki.python.org/moin/Templating

One basic way to make the choice is to ask 'does XML make me sick into my breakfast cup?'. If the answer is yes, choose a non-xml based library. The xml-based libraries are really orientated towards templating for the web, and NML gains nothing from them.

I have no allergy to xml though, so I chose Chameleon, because I know it, and because it's quick, easy to understand, flexible, but not overloaded with features that inspire complexity. It also doesn't enforce the use of xml, and has a 'text' option for templates. Handy.
http://chameleon.repoze.org/docs/latest/

There are many other choices: I also considered Mako (non-xml), based on recommendation, but it looked more powerful than I wanted. Plus, I know Chameleon, so go figure

To be continued....

[andythenorth]

"Hello world..."

I could write more background, but it's better to learn by doing.

Let's put together a simple render pipeline, in the form.
simple data file -> simple python -> simple template files -> .nml file

To do this we need some initial steps. If you're comfortable with iterators and basic python file i/o, skip this bit.

We'll start at the end, by figuring out how to write files to disk. Without this, everything else we do is pointless, as we won't be able to save anything to feed into nmlc.

First, a python script that writes a simple text file to disk, then we'll learn how iterators work.

1. Create a clean folder to work in. Doesn't matter what you call it; if you're short of ideas call it 'bob'.

2. Open a file in a text editor, and save it to 'bob' as 'learn_stuff.py'

3. Paste the following into 'learn_stuff.py', then save.

Code: Select all

# print statements aren't necessary, but they help see what the script is doing when it runs
# print is helpful for 'wtf?' moments
print "Running script..."

 # this module is a good way to handle working with files that might contain unicode (e.g. translations) 
import codecs

# create a new file on disk, which will have a name and be writable
test_file = codecs.open('test_file.txt','w','utf8')

# write a string into the file
test_file.write("it works")
test_file.close() # we're done with this file now, finish with it

4. in your shell, navigate to 'bob' and enter 'python learn_stuff.py'

5. if you now have a file in 'bob' called 'test_file.txt', you win. 'test_file.txt' should contain the string 'it works'.

The King and i
The next building block we need is the ability to iterate over sets of things. Iterators are usually integral to a templating project. for example, we might want to use one template for multiple vehicles, or new objects or so on.

1. Delete the contents of 'learn_stuff.py' and replace it with this, then save.

Code: Select all

# print statements aren't necessary, but they help see what the script is doing when it runs
# print is helpful for 'wtf?' moments
print "Running script..."

# this is a tuple (kind of list) containing some strings
data = ('Cat','Dog','Ham','Eggs','Foo','Bar') 
stuff = '' # create an empty string

print "Iterate..."
for i in data:
  print i
  stuff = stuff + i + '\n' # do world's dumbest string concatenation, plus a newline char for better formatting


print "Writing the file..."
 # this module is a good way to handle working with files that might contain unicode (e.g. translations) 
import codecs

# create a new file on disk, which will have a name and be writable
stuff_file = codecs.open('stuff_file.txt','w','utf8')

# write stuff into the file
stuff_file.write(stuff)
stuff_file.close() # we're done with this file now, finish with it

2. at the shell enter 'python learn_stuff.py'

3. you should now have a file called 'stuff.txt' with contents that look like this:

Code: Select all

Cat
Dog
Ham
Eggs
Foo
Bar

[Alberth]

/me thinks a few wiki pages may be easier to manage/find.

[andythenorth]

/me thinks a few wiki pages may be easier to manage/find.

Agreed. But initially I'm looking for feedback / questions / advice etc

Otherwise, yes, wiki is more logical for this. Where's best though? General newgrf spec section? NML manual? OpenTTD wiki?

[Eddi]

Code: Select all

test_file = open('test_file.txt','w')

may i suggest here:

Code: Select all

import codecs
test_file = codecs.open('test_file.txt','w','utf8')

... way less trouble in the long run, when you want to handle translations and stuff

[FooBar]

Where's best though? General newgrf spec section? NML manual? OpenTTD wiki?

I'd say the TT-wiki. That is also where the NML tutorial is.
As this is not a specification, and not specifically (:P) OpenTTD related, I'd say next to the other tutorials is the best place. I know you didn't really want to call it tutorial, but still...

Maybe link it from this page: http://www.tt-wiki.net/wiki/Tutorials "I think it fits the Coding NewGRFs for OpenTTD and TTDPatch" section.

Other than that I enjoy reading this

[andythenorth]

may i suggest here: (use codecs)

Changed in the examples above, thanks.

[andythenorth]

Baby steps with Python Template()

Template() is a class in python's default string module. http://docs.python.org/release/2.5.2/lib/node40.html
It's limited: it can only do value replacement. But it has a few things to commend it:
- simplicity can be good. As a starting point for people who want to do basic NML templating, Template() offers enough, without the complexity of expressions, macros, repeats etc.
- no dependencies; it ships with python versions > 2.5.2. That means no need to install any additional python modules.

If you're familiar with templating, and plan to use a more sophisticated templating library, skip this. I'll cover examples using the Chameleon library later.

We'll start by templating some plain text. Constructing an NML file requires a few extra steps, so let's get templating basics sorted out first.

1. In a text editor, create a new plain text file, and save it as 'template.ttxt'. (.ttxt is a made-up extension representing template text; it's not significant what the extension actually is, you could change this).

2. paste the following code into 'template.ttxt' and save.

Code: Select all

The name is $name

3. clear the contents of 'learn_stuff.py' and replace it with this, then save.

Code: Select all

# print statements aren't necessary, but they help see what the script is doing when it runs
# print is helpful for 'wtf?' moments
print "Running script..."

 # this module is a good way to handle working with files that might contain unicode (e.g. translations) 
import codecs

# import the template module
from string import Template

# this is a tuple (kind of list) containing some strings
data = ('Cat','Dog','Ham','Eggs','Foo','Bar') 
templated_stuff = [] # create an empty list, we'll put strings in here then join them later

my_template = Template(codecs.open('template.ttxt','r','utf8').read()) # can you guess what this does?

print "Iterate..."
for i in data:
  print "Templating: " + i
  result = my_template.substitute(name=i) 
  templated_stuff.append(result) # put the templated string into a list
  
print "Writing the file..."

# create a new file on disk, which will have a name and be writable
templated_stuff_file = codecs.open('templated_stuff_file.txt','w','utf8')

# write stuff into the file
templated_stuff_file.write('\n'.join(templated_stuff)) # join the list of templated stuff with newlines, and write to a file
templated_stuff_file.close() # we're done with this file now, finish with it

4. enter 'python learn_stuff.py' in your shell

5. you win if you now have a file called 'templated_stuff.txt', with contents like this:

Code: Select all

The name is Cat
The name is Dog
The name is Ham
The name is Eggs
The name is Foo
The name is Bar

[andythenorth]

build_mygrf.py - start templating NML

We're ready to use python templating with NML. We only know how to template one value ('name'), but we can learn how to add more later. However we need to deal first with outputting a valid NML file. This takes a little more work, and more files, so here's a zip of an example project to work with.

example_1.zip

(8.85 KiB) Downloaded 353 times

Follow the instructions in the readme.txt to build the grf. The result should be three trucks (introduced 1900), with identical specs and different names. Let's look at how this is done.

What files do we have?

Code: Select all

build_mygrf.py - this is the build script
header.nml - this is information that should only be included once per grf, e.g. header strings, cargo table etc.
truck.tnml - this is the template nml code for a truck
lang/english.lng - the lang file for this project
cabover_truck_test.png - the png file for truck graphics  

Let's look in a little more detail at the build script:

Code: Select all

# print statements aren't necessary, but they help see what the script is doing when it runs
# print is helpful for 'wtf?' moments
print "Running script..."

 # this module is a good way to handle working with files that might contain unicode (e.g. translations) 
import codecs

# import the template module
from string import Template

sections = [] # create an empty list, we'll put strings in here then join them later

# get the header file and append to a list
header = codecs.open("header.nml",'r','utf8')
sections.append(header.read())
header.close()

trucks = ('super_speedy','extra_slow','middling')

# get a template to use for trucks
truck_template = Template(codecs.open('truck.tnml','r','utf8').read())


for i in trucks:
  print "Templating: " + i
  sections.append(truck_template.substitute(id=i))

  
print "Writing the file..."

# create a new file on disk, which will have a name and be writable
processed_nml_file = codecs.open('example.nml','w','utf8')

# write stuff into the file
processed_nml_file.write('\n'.join(sections)) # join the list of templated stuff with newlines, and write to a file
processed_nml_file.close() # we're done with this file now, finish with it

This script reads the header file and stores it to a list 'sections'. Then a list of truck ids - 'trucks' - is iterated over. For each truck, the truck template is called, passing the truck id. The result is stored to the 'sections' list. Finally we concatenate everything in 'sections' list and write it to a file.

Let's look at the template:

Code: Select all

// -- begin truck --/

spriteset(ss_truck_${id}, "cabover_truck_test.png") {
  spriteset_template_truck_16(0, 20) // (search 'sprites/nml' to find where spriteset templates are defined)  
}

spritegroup sg_truck_${id} {
    loaded: ss_truck_${id};
    loading: ss_truck_${id};
}

item(FEAT_ROADVEHS, truck_${id}) {
    property {
        name: string(str_name_truck_${id});
        climates_available: ALL_CLIMATES; // nml constant
        sound_effect: SOUND_TRUCK_START_2; //nml constant  
        visual_effect: visual_effect(VISUAL_EFFECT_DIESEL, 0); // nml constant for DIESEL
        introduction_date: date(1900,01,01); //
        cargo_capacity: 0;
        sprite_id: SPRITE_ID_NEW_ROADVEH; //enable new graphics - nml constant
        speed: 30mph;
        power: 100hp;
        weight: 10;
        tractive_effort_coefficient: 0.3; // just use default value until evidence arises for need to vary this per truck
        cost_factor: 100;
        running_cost_factor: 10;
        refittable_cargo_classes: bitmask(CC_MAIL, CC_EXPRESS, CC_ARMOURED, CC_BULK, CC_PIECE_GOODS,CC_LIQUID, CC_REFRIGERATED, CC_COVERED,);
        non_refittable_cargo_classes: bitmask(CC_PASSENGERS);
        cargo_allow_refit: [];
        cargo_disallow_refit: [];
        misc_flags: bitmask(ROADVEH_FLAG_2CC, ROADVEH_FLAG_AUTOREFIT); // nml constants
        model_life: VEHICLE_NEVER_EXPIRES;
        vehicle_life: 255;
    }
    graphics {
        default: sg_truck_${id};
    }
}

// -- end truck --/

This is very simple. For each occurrence of ${id}, the truck id is substituted, resulting in unique identifiers for items and switches, e.g. 'ss_truck_super_speedy' and 'str_name_truck_middling'.

To call the build script and then nmlc we enter the following in the shell:

Code: Select all

python build_mygrf.py
nmlc -c --grf example.grf example.nml

Then move the grf to OpenTTD's data directory to use it.
Incidentally, this build step could be simplified in a number of ways - calling nmlc from the build script, using a shell script, or a makefile. Topic for another day, or another person (hi planetmaker?).

We're now templating NML. Aren't we good? Next we'll explore how this can be extended to do a little more. For this you may want more tea.

[MinchinWeb]

Hey Andy,

I've been watching this thread with much interest. I like your instruction style too - it covers enough to allow the steps to be followed easily, it doesn't get caught up in the deep inner workings, and we don't get lost by having tons of options.

(..) calling nmlc from the build script (..)

I hope you'll cover this. I guess I could write a Windows BATCH script to run the two commands to build the NML file and then the GRF file, I think it would be neat to be able to automatically have the GRF built in one 'step'.

I look forward to reading the rest!

[andythenorth]

I hope you'll cover this.

I use the build framework from OpenTTDCoop, which is very comprehensive: http://dev.openttdcoop.org/projects/newgrf-makefile

The ideal is to build the grf, running build scripts and also building any readme files needed, and optionally package it for bananas, or install it directly for local testing. Python could do this, but it's not designed as a build tool.

Maybe someone else might start a thread on ways to do it?

[andythenorth]

Template like a (trainee) ninja
So far I've written all the code. Now it's your turn. Here are some exercises using the project in example_1.zip (posted above). Fail these and you will never be a ninja.

1. Add another vehicle called 'very_good'. This requires precisely two changes. One of the changes will show you why code generation in templates might be useful. Prove your ninja skills: what are the necessary changes?

2. Convert the list to a dict, and template both the identifiers and the capacity for each truck. Potential ninjas will read the python docs on Template substitute() carefully, to avoid creating more iterators than they need.

Who will win first? Post solutions here...

[andythenorth]

Here's an example for the challenges in the previous post.

example_2.zip

(24.02 KiB) Downloaded 421 times

This converts 'trucks' to a dict not a list. The dict contains further nested dicts, defining speed and capacity.
These are passed to the template.

For each truck added, the lang file also has to be updated adding a name for the truck.
The names could be added to the truck dicts, and the lang file could also be templated.
The truck names could be appended to the lang file one line at a time by the build script.

A better method might be to repeat a block for truck names in a templated lang file (using a template language that supports repeated blocks). This approach makes the build script simpler and provides a better separation of concerns between templates and build script. However the simple Template module provided by python has no ability to repeat. It's simple - and there are many cases where it might be enough

Another issue is that it's not a very good design to keep the config data for the trucks in the build script. There are many ways to split this config data into other files, which makes it easier to maintain and edit both the config data and the code.

The next posts will look at why it can be useful to have more powerful templating. I'll introduce the Chameleon templating library, explore why constructing first-class python objects for things like vehicles can be useful, and look at ways to get data into the build script from external sources

[MinchinWeb]

Success! After far to long of thinking of doing it, I managed to sit down at the computer and actually work through the tutorial (thus far). I'm working on windows, so with that in mind, a couple of thoughts:

• In Notepad, to get proper line breaks, I need /r/n
• Python, when installed in Windows, will automatically run .py when you click on them. However, it promptly exits without showing you any error codes that show up. I found add the following code at the end of the files was handy (it keeps the window up until you hit enter):

Code: Select all

raw_input('Press Enter...')

Overall, the tutorial was very easy to follow. I look forward to the next lesson, whenever you decide to take a break from templating truck graphics.

[FooBar]

suggestions: don't use notepad, but download notepad++ or something. And run your scripts from a command prompt, so that you don't need these ugly tricks. Makes it impossible to use your script in nightly builds if you ever get to that point.

[Eddi]

once upon a time you could tell windows to keep the console window open after the program finished...

[frodo25]

Hello, andythenorth,


Thanks for sharing code details I'm use this your code is very helpful.

Thanks