And now we will run off a cliff
Everything from here will go better if you understand enough python to know how to use lists, dicts, tuples, iterators, vars, and have at least a rough understanding of objects, functions, imports and namespaces. If you don't know that, you won't learn it well from me. I just point and wave my arms a lot.
Also - about a week before I wrote this I didn't know much about objects, file I/O, enumerate iterators, os.path, imports, or python globals either - but I do know how to use google, which finds this: http://docs.python.org/
(I have no training in compsci, except I once failed a course in Pascal. I also failed a course in Excel).Rocks at the bottom
The basic idea of templating is to create reusable blocks of code that can then be customised per vehicle, house, industry, object etc. These templates are then compiled to one .nml file that can be turned into a newgrf by nmlc.
By reusing code, projects are easier to develop and easier to maintain. The first newgrf I wrote (HEQS) didn't use templating, and was just one very long NFO file. This was hard to work with. With help from others, I started using templating for NFO. Good templating wins.
A few things are of interest when templating:
1. Substitution of values. If you've used NML built-ins like 'ALL_CLIMATES' or 'VEHICLE_NEVER_EXPIRES', then you've used value substitution (maybe without knowing it). Value substitution is useful for many things, like setting identifiers (for items and switches) and item properties (e.g. speed, cargo refits, vehicle life, etc).
2. Macros / code reuse from pre-defined blocks of code, inserted into a file as needed. Parameters can be passed to the macro for use inside the macro. Macros are flexible way to reuse code. NML already permits this with sprite templates.
3. Dynamic code generation. An example is switch blocks that may have different checks depending on the vehicle type. These can be generated in the code with a 'for' loop. Generators can use several approaches, including calling macros, or composing code from raw strings.
Caveats: complex templating can be worse than no templating at all. It would be easy to spend more time maintaining a template system than just writing out plain NML in one large file.
Particularly, excessive code generation mixed with templates can be un-fun. When working with code generation, you need to be able to visualise what it will do for many different cases, and/or check output constantly. You're also working with two (or more) languages combined in the same file, which can lead to a lot of easy syntax mistakes, hassle dealing with string escapes and general head scratching. Code generation is great for those who know how to write a proper compiler (hi Eddi), but might not be fun for all.
It's no different for an NML project: quite-dumb templating means that the NML is easy to read, check, and debug. Debugging templating systems is dull.General approach (no code yet)
To get 'stuff' into a newgrf, it's useful to have some idea of a pipeline.
For plain NML, the pipeline is .nml file -> nmlc -> newgrf. (This ignores the pngs etc, code is the significant factor here).
For templating with python we add to the start of our pipeline. The goal is to combine some data
with one or more templates
, and arrive at the .nml file to feed to nmlc. There are many ways to build a pipeline. CETS (mentioned above) does something like this:
Google Docs -> .tsv file -> python -> .pnml -> C Pre-processor -> .nml file
Another way would be to do:
One giant python file defining data and templates -> .nml file
I've been testing python templating for the BANDIT truck set, for which the pipeline is:
web forms -> database -> python -> output giant dict as text in browser -> copy and paste to plain text file -> more python -> template files -> .nml
Which is kind of overkill, and not at all to be advocated.
Generally an ideal pipeline looks like:
simple data file -> simple python -> simple template files -> .nml fileWhat's a good data file format?
Dunno. Not sure yet. Options include:csv or tsv
- probably the most logical
- can be edited with spreadsheet app (including online apps) or text editor
- easy to parse (python has libraries for it)
- easy to share around between peopleplain text - name/value pairs
- simple & robust
- edit with just about anything
- slightly a pain in the arse to maintain when adding new propertiesrow-based database - sqllite or similar
- can run sql queries or similar on these to add/remove/change properties
- can validate data types etc
- adds complexity and overheadpython dicts in text files - import as modules
- almost identical to NML 'item' format
- easy to understand
- pain in the arse to format and maintain when adding / removing properties (unless you know regexp etc)JSON
- never used it, can't comment. Widely used and afaik has good python module support.XML
- way too much scaffolding for this problem. Any time I use XML, I end up templating it. Templating the data input format seems rather circular at best.
I concluded 'none of the above' and wrote a python object database in the Zope web framework, but the number of people likely to do that can be counted on the thumb of one hand. It also requires an odd copy-paste step (data could be fetched from the web, but then people without a web connection can't easily build the grf).(Templating) weapons of choice
Python has long had a built in string templater. It uses a %s notation. I hate it.
But wait! Python has a better template strings library: http://docs.python.org/release/2.5.2/lib/node40.html
The built-in Python template strings library has a few neat features:
- it's easy to use
- it can't do much, so it's hard to make things too complicated when using it
- no dependencies need to be installed. If you have python > 2.5.2, it's just available
I tried it out, and it offers some useful features for NML authors who want simple templating. I'll post some examples later. However, I wanted a template library that could do a bit more. This means picking a library.
Python has lots
of template libraries: http://wiki.python.org/moin/Templating
One basic way to make the choice is to ask 'does XML make me sick into my breakfast cup?'. If the answer is yes, choose a non-xml based library. The xml-based libraries are really orientated towards templating for the web, and NML gains nothing from them.
I have no allergy to xml though, so I chose Chameleon, because I know it, and because it's quick, easy to understand, flexible, but not overloaded with features that inspire complexity. It also doesn't enforce the use of xml, and has a 'text' option for templates. Handy.http://chameleon.repoze.org/docs/latest/
There are many other choices: I also considered Mako (non-xml), based on recommendation, but it looked more powerful than I wanted. Plus, I know Chameleon, so go figure To be continued....