YAGL - Yet Another GRF Language (maybe)

Discussions about the technical aspects of graphics development, including NewGRF tools and utilities.

Moderator: Graphics Moderators

User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

I have recently been working on a tool which reads a GRF file into memory, and then writes it out using a human-readable text format (at least, hopefully more readable than NFO), along with sprite sheets. This is not yet complete, but is coming along quite well. It's largely a matter of filling in the blanks now. I'm aiming for complete coverage of the NewGRF specs, which might entail a bit of digging in the OTTD source and/or questions.

For an example, the Action08 pseudo-sprite is currently rendered as:

Code: Select all

grf // Action08
{
    grf_id: "\xFB\xFB\x06\x01";
    version: 8;
    name: "Dutch Trainset 2.1.0";
    description: "{lt-gray}Dutch Trains for OpenTTD {new-line}{black}First vehicle: 1839.{new-line}{new-line}(c)Dutch Trainset Team {new-line}License: GPLv2 or higher. {new-line}See readme for details.";
}
It looks vaguely reminiscent of NML but is most definitely *not* NML. That would be a reverse engineering task beyond my skills, and probably impossible.

For another example, a VarAction02:

Code: Select all

switch<Trains> // Action02 variable
{
    this_set_id: 0xEC;
    test: (year_of_construction & 0xFFFFFFFF);
    switch (test)
    {
        0x00000000..0x000007BE: 0x00F0; 
        0x000007BF..0x000007C5: 0x00F4; 
        0x000007C6: 0x00F5; 
        0x000007C7..0x000007CE: 0x00F4; 
        0x000007CF..0x000007D3: 0x00F8; 
        0x000007D4: 0x00F1; 
        0x000007D5..0x000007D6: 0x00F8; 
        0x000007D7: 0x00EC; 
        default: 0x00EB;
    }
}
I've pulled out the name of the variable being tested here, though this idea might not work out as well as I'd like. It would be nice to format the values in the ranges to match the variable (decimal in this case 0x7BFE = 1982), but that's a refinement for later. Nothing about the text format is set in stone: I'm currently just trying to get something reasonable done for every type of pseudo-sprite.

A secondary, but maybe more interesting, goal for this tool is that it should be able to read its own text files and sprite sheets back into memory, and then write out the data as a new GRF file. This is why the output has a lot of apparently superfluous braces and semicolons. This feature has become very important in my mind, but is much less complete than writing the text file. I pretty much know how to do it, but my approach to parsing the text is a bit tedious and verbose, and there is quite a lot to do. Again, it's largely a matter of filling in the blanks.

Anyway... the reason for posting is that I've been working on this for a little while now, and I kind of wanted to give a heads up. And maybe get some direction from feedback. Obviously NML (or M4NFO, I guess) is the way to go for most GRF development, but I wondered if there is still a place for something more low level. The main reason I started this was curiosity about how NML constructs map on to the NewGRF specs. And then I got a bit carried away...
User avatar
wallyweb
Tycoon
Tycoon
Posts: 6102
Joined: 27 Nov 2004 15:05
Location: Canada

Re: YAGL - Yet Another GRF Language (maybe)

Post by wallyweb »

This could be interesting. I'll be subscribing and following your progress.
Once you get it working it would be an interesting developing tool.
One of the benefits of nforenum (included with grfcodec) is that it is a lintner and it verifies NFO code with error messages where required. Unfortunately it is no longer maintained and is missing new features such as NRT.
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: YAGL - Yet Another GRF Language (maybe)

Post by planetmaker »

Even when you write the contrary: from the looks it's something like a de-compiler from grf to nml. Such thing can indeed be useful and could become a nice part of NML itself.

(EDIT to add my personal 2ct: Everyone does what s/he likes to do - we all work for our own fun, so do what you enjoy! Personally I'd like to rather see one decently maintained NewGRF dev tool than a 4th badly ;) Thus I'd like to invite you to work on nml and/or grfcodec. Plenty enough to add to gain fame :) )
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

Thank you both for your comments. I'm flattered by the invitation to work on nml and grfcodec. As it happens, this project has had me digging around inside OpenTTD and NML a little, though not grfcodec. I have to say that I have never loved Python: much prefer static type checking. But it could happen...

It really is not going to be an NML decompiler. Some of the constructs in NML have an almost one-to-one relationship with records in the NewGRF specs, and it is possible to make the output closely match NML. In general this is not the case. Expressions used in switches are a bit of a nightmare. Variables are a pain because some are actually two or more variables in NML terms: unpicking those might be tricky. No block names can be recovered, of course. Nor templates. Still, I had wondered whether some sort of approximation to NML could help with debugging.

I suspect planetmaker might be making a reference to an unmaintained GUI tool I wrote some years ago. wallyweb was kind enough to look at it. Fair comment. Life got in the way, I'm afraid, and I'm sorry for letting it lapse. I accept your criticism. In retrospect, I feel that a GUI was not the best approach for managing a something as convoluted as the low level minutiae of a GRF. It was a great exercise in learning a lot about Qt, though. A text format seems like a better fit. And I think I'm in a better place to commit to some level of maintenance. :)

Work in progress: town names/Action0F were added the other day. Here is a complete dump of dtnames.grf as a demo:
dtnames.txt
(129.3 KiB) Downloaded 181 times
. I haven't written the parser for Action0F yet, but it looks to be simple...

Currently trying to work out a reasonable format for sprite layouts. So complicated!
Alberth
OpenTTD Developer
OpenTTD Developer
Posts: 4763
Joined: 09 Sep 2007 05:03
Location: home

Re: YAGL - Yet Another GRF Language (maybe)

Post by Alberth »

Nice work!

I have been trying to find a way into the NewGRF spec several times, but failed so far. You seem to have found a better path :)

I noticed the Thorn character in

Code: Select all

text("Þ´t Eind", 1);
It's NewGRFs way of telling the text is Unicode (utf-8) rather than Ascii. You can detect that while reading the text again, so you may want to drop the Thorn character from the output.
I haven't written the parser for Action0F yet, but it looks to be simple...
It is in general, NML does a bit more. It normalizes the chance values (reducing the number of needed bits), and automatically splitting lists that are too long for GRF. That are however features to make life simpler for the NewGRF authors, which arguably don't belong in a de-compiler.

Do you have a repository with the code somewhere? I tried looking for YAGL, but there are a lot of YAGLs out there, so nothing useful turned up.
Being a retired OpenTTD developer does not mean I know what I am doing.
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: YAGL - Yet Another GRF Language (maybe)

Post by planetmaker »

UnicycleBloke wrote: 17 Dec 2019 22:30 I suspect planetmaker might be making a reference to an unmaintained GUI tool I wrote some years ago.
Actually not, I wasn't even aware of than when writing. I was referring to the fact that nml and also grfcodec and nforenum all need people who care about it and can dedicate time to it so that they remain useful.
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

planetmaker wrote: 18 Dec 2019 08:41 Actually not, I wasn't even aware of than when writing.
That was a silly comment (by me). That long lost program was on my mind, and obviously the entire universe revolves around me. ;)

I haven't created a public repo yet, but am planning to release it under GPLv3. I'm hoping that the software architecture and implementation is simple enough for others to follow. It's written in C++.

Good point about thorn. I decided to convert all strings to UTF8 for the output text. It was a bit of a devil to decode the not-quite-Latin1 and not-quite-UTF8 into a common UTF8 readable format (with control codes rendered with escaped names). Going back the other way may or may not result in an identical encoding because some control codes can be both short form and long form in UTF8 strings. I wondered whether it made sense to simply encode all strings as UTF8 for simplicity...

On the NewGRF specs generally, I haven't hit a wall yet, but I'm likely to have questions as I go along. I am certain to have misunderstood or overlooked some things, but these errors should be simple to fix when pointed out. It would be nice to respect earlier GRF versions when re-encoding, which is sure to lead to buglets. At some point, I'm going to need a pile of GRFs which together give complete coverage of the specs. On that subject, is there a good example around of using Action11, with both binary sounds and imported sounds?
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

So. Work continues, and I am approaching a milestone - decoding theoretically complete. I've hit an issue that has me a bit puzzled, and would appreciate advice.

What is the range of possible colour depths for sprites in Container2 files. I thought it was one of RGB, RGBA, or P (palette index). At least, that's what I thought after looking at the NML source. I have found a number of RGBAP sprites in zbase (zbase-v5588/zbase_extra.grf is what I have), with 5 bytes per pixel. If I decode them as such, everything is fine. What does this format mean?

Thanks.
User avatar
PikkaBird
Graphics Moderator
Graphics Moderator
Posts: 5601
Joined: 13 Sep 2004 13:21
Location: The Moon

Re: YAGL - Yet Another GRF Language (maybe)

Post by PikkaBird »

32bpp sprites can have a TTD-palette mask for company colours. I'm guessing that's the RGBAP format.
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: YAGL - Yet Another GRF Language (maybe)

Post by planetmaker »

What Pikka sais. See the NewGRF specs on real sprites: https://newgrf-specs.tt-wiki.net/wiki/RealSprites#Type
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

32bpp sprites can have a TTD-palette mask for company colours. I'm guessing that's the RGBAP format.
Of course. Thanks for the link, planetmaker. I'd seen that already but didn't connect the dots. I now handle these cases by creating a separate spritesheet for the masks.
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

For those who are interested, I have now placed the project source here: https://github.com/UnicycleBloke/yagl.git.

Decoding a GRF into YAGL script and spritesheets is theoretically complete, or at least not very far away. On the down side, I have not been able yet to test all of the different types of pseudo-sprites, and there are certain to be some errors, omissions and inconsistencies. There are sure to be parts of the specs which I have misunderstood.

If you wish to try the software, you will need to build it. I have developed it in C++17 under Linux using g++, and haven't given much thought to other environments or release images yet. It will likely build and run in pretty much any reasonable Linux or Linux-like system such as Cygwin or MSYS2, and I have done so in the Windows 10 Subsystem for Linux. I'm using g++9.2 most of the time, but have built it on another machine with g++7.4. I suppose clang would also be fine for compilation, but have not tried it. The repository has more instructions in README.md.

There is much still to do, but I have successfully decoded some quite large GRFs, including FIRS, zBase, and Dutch trains. I would be very grateful for any reports of GRFs which throw an exception during decoding. I've had a few of these, and they have mostly been my fault. I would also appreciate any other feedback or suggestions.

The next tasks are:

- To verify that all the pseudo-sprites are represented in the script in a way which is semantically identical to the equivalent NFO, i.e. can represent everything the NewGRF specs allow. This may be explicit or implicit (e.g. the number of items in a list). It would nice to start adding assertions for what the specs do not allow.

- To improve the script itself to make it easier to understand, less verbose, and more consistent. Making changes is simple. I have represented most properties in Action00 features with hexadecimal numbers for now (for expediency), but it is easy to split out enumerations and bitfields with named values. Dates should be also be simple, and probably some other types. I'm not very happy with some of the other pseudo-sprites, such as VarAction02. Some of the examples of this from FIRS are quite breathtaking. I hadn't expected such calculations.

Anyway, I hope it's useful. Enjoy.
michael blunck
Tycoon
Tycoon
Posts: 5948
Joined: 27 Apr 2005 07:09
Contact:

Re: YAGL - Yet Another GRF Language (maybe)

Post by michael blunck »

I'm not very happy with some of the other pseudo-sprites, such as VarAction02. Some of the examples of this from FIRS are quite breathtaking. I hadn't expected such calculations.
These might be because of the very special encoding of some of those grfs (python -> nml -> nfo -> grf).

Would be interesting to hear about results from some larger m4nfo-coded grfs like MariCo or NewStations, see

http://www.ttdpatch.de/download.html

regards
Michael
Image
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

michael blunck wrote: 31 Dec 2019 08:02 Would be interesting to hear about results from some larger m4nfo-coded grfs like MariCo or NewStations,
It falls over on this record in MariCo:

Code: Select all

7395 * 23	 02 0F 00 41 8D 0F 00 00 00 00 00 80 44 80 0C 00 00 00 00 10 10 10 0A
It's an advanced sprite layout for an Object. I must have misread the spec. I treated this record as follows, perhaps someone can point out the error:

Code: Select all

7395 * 23	 
02            Action02
0F            Objects
00            This set ID
41            Advanced (uses registers) with 1 building sprite
8D 0F 00 00   Ground sprite - assumed to be a parent (creates new bounding box)
00 00         Register flags - none set
00 80 44 80   Building sprite - value looks weird to me, but high bits have special meanings
0C 00         Register flags - bits 2 and 3 set
00 00 00      x, y, z offsets
10 10 10      x, y, z extents
0A            Register value for bit2
...           Oops - ran out of bytes
I totally expected this. Nothing's ever right first time. Investigating...
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

After looking at the OTTD source which reads sprite layouts, I still can't work out what I'm doing wrong. Hmm...

NewStations was fine, though. I've attached the output files, and the NFO created by grfcodec for comparison. You can see that the YAGL is quite verbose in its current form, but that there is a direct correspondence to records in the NFO.

I don't know much about M4NFO. Does that generate NFO from macros, or does it directly create a GRF like NML?
Attachments
newstats-8bpp-normal-0.png
(886.76 KiB) Not downloaded yet
newstats.nfo
(1.04 MiB) Downloaded 148 times
newstats.txt
(3.23 MiB) Downloaded 159 times
michael blunck
Tycoon
Tycoon
Posts: 5948
Joined: 27 Apr 2005 07:09
Contact:

Re: YAGL - Yet Another GRF Language (maybe)

Post by michael blunck »

00 80 44 80 - a recoloured building sprite
0A <oops ...> - a custom recolour sorite does not use a register

Here's the m4nfo:

Code: Select all

def(0) spriteset(
    ground(GRASS)
    recolour(0, xyz(0,0,0), dxdydz(16,16,16), _MINERAL, aslflags({OFFSET_RCSPRITE, CUSTOM_RCSPRITE}), registers(REGISTER))
)
regards
Michael
Image
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

Thanks, Michael. I need to pay more attention to the fine print. With that minor correction in place, MariCo now decodes just fine. To return to the subject of VarAction02 complexity, I'd be interested to know the M4NFO that produces this in MariCo (I know that var_1A is all ones, and can be used to create constants - a lot of the apparent complexity comes just from that):

Code: Select all

// Record #7821
switch<Objects> // Action02 variable
{
    this_set_id: 0x10;
    test: (variable[0x60](0xF0) & 0x000000FF);
    test: signed_cmp(test, (variable[0x1A] & 0x00000053));
    test: bitwise_and(test, (variable[0x1A] & 0x00000001));
    test: mul(test, (variable[0x1A] & 0x00000008));
    test: temp_store(test, (variable[0x1A] & 0x00000000));
    test: assign(test, (variable[0x60](0x0F) & 0x000000FF));
    test: signed_cmp(test, (variable[0x1A] & 0x00000053));
    test: bitwise_and(test, (variable[0x1A] & 0x00000001));
    test: mul(test, (variable[0x1A] & 0x00000004));
    test: bitwise_or(test, (variable[0x7D](0x00) & 0x0000000F));
    test: temp_store(test, (variable[0x1A] & 0x00000000));
    test: assign(test, (variable[0x60](0x10) & 0x000000FF));
    test: signed_cmp(test, (variable[0x1A] & 0x00000053));
    test: bitwise_and(test, (variable[0x1A] & 0x00000001));
    test: mul(test, (variable[0x1A] & 0x00000002));
    test: bitwise_or(test, (variable[0x7D](0x00) & 0x0000000F));
    test: temp_store(test, (variable[0x1A] & 0x00000000));
    test: assign(test, (variable[0x60](0x01) & 0x000000FF));
    test: signed_cmp(test, (variable[0x1A] & 0x00000053));
    test: bitwise_and(test, (variable[0x1A] & 0x00000001));
    test: bitwise_or(test, (variable[0x7D](0x00) & 0x0000000F));
    switch (test)
    {
        0x00000000: 0x0000; 
        0x00000001: 0x0001; 
        0x00000002: 0x0002; 
        0x00000003: 0x0003; 
        0x00000004: 0x0004; 
        0x00000005: 0x0005; 
        0x00000006: 0x0006; 
        0x00000007: 0x0007; 
        0x00000008: 0x0008; 
        0x00000009: 0x0009; 
        0x0000000A: 0x000A; 
        0x0000000B: 0x000B; 
        0x0000000C: 0x000C; 
        0x0000000D: 0x000D; 
        0x0000000E: 0x000E; 
        default: 0x000F;
    }
}
This is theoretically identical in content to the binary, but I don't think I have made it at all clear what is going on. I'd like at least to substitute "variable[...]" with the name of the variable (depends on feature type). I know that NML splits some variables into two or more named values using masks, but I'm not sure that could be decoded cleanly.
michael blunck
Tycoon
Tycoon
Posts: 5948
Joined: 27 Apr 2005 07:09
Contact:

Re: YAGL - Yet Another GRF Language (maybe)

Post by michael blunck »

NewStations was fine, though.
Wow, that version is already 4 years old. And misses lots of the new features.
I don't know much about M4NFO. Does that generate NFO from macros, or does it directly create a GRF like NML?
It is based on the M4 macro processor using grfcodec for graphics encoding.
To return to the subject of VarAction02 complexity, I'd be interested to know the M4NFO that produces this in MariCo [...]
That's the output of m4nfo function obj_infotype4():

http://www.ttdpatch.de/grfspecs/m4nfoMa ... info_type4

It encodes the 4-neighbourhood of a given object's tile. See the tutorial for description and example:

http://www.ttdpatch.de/grfspecs/m4nfoMa ... l#example1

In m4nfo, the more complex functions are "hardcoded" using varAction2s, rather than being implemented by using a bunch of actionDs according to user input on the fly.

BTW, for stations there's a similar function plt_edges():

http://www.ttdpatch.de/grfspecs/m4nfoMa ... #plt_edges

though it's not being used in that old NewStations grf that you've been decoding.

regards
Michael
Image
User avatar
wallyweb
Tycoon
Tycoon
Posts: 6102
Joined: 27 Nov 2004 15:05
Location: Canada

Re: YAGL - Yet Another GRF Language (maybe)

Post by wallyweb »

michael blunck wrote: 01 Jan 2020 07:42 Wow, that version is already 4 years old. And misses lots of the new features.

Code: Select all

3 * 109	 08 07 "mb" 06 01 "NewStations v0.6 14.08.2014"
lNewStations (v0.6)
last update 14-aug-2014
:roll:

So where is the latest one with all the new features? :wink:
8)
User avatar
UnicycleBloke
Engineer
Engineer
Posts: 71
Joined: 30 Aug 2011 14:39
Location: Cambridge, England

Re: YAGL - Yet Another GRF Language (maybe)

Post by UnicycleBloke »

Time for an update. It has taken longer than I expected, but the tool is now capable of the full cycle of read-print-parse-write. That is to say, it can read a GRF and print a matching YAGL file plus associated sprite sheets, and it can also do the inverse (parse the YAGL and write a GRF). I guess these functions are broadly equivalent to grfcodec -d and grfcodec -e, respectively. For example:

Code: Select all

// This creates a subfolder called sprites, and creates files dutchtrains.yagl and dutchtrains-8bpp-normal-0.png.
yagl -d dutchtrains.grf 

// This looks for a subfolder called sprites, moves dutchtrains.grf to a backup file, and creates a new dutchtrains.grf.
yagl -e dutchtrains.grf 

// This reads the GRF and dumps all the records, including realsprites, as hex. The main reason for this is to compare the recreated GRF with the original.
yagl -x dutchtrains.grf 
Due to the nature of the NewGRF specs, the binary encoding is not unique, which has made testing for errors a bit harder. The main source of difference seems to be the encoding of unicode strings, for which some of the control code have two possible encodings (I always use the long form), but there are one or two others, such as extended bytes, which can also be encoded in two ways (NML seems to always use the long form, so I copied that).

I need to try this out with a whole bunch of GRFs but hopefully I've ironed out most of the wrinkles. I've mostly worked in Linux, but have made some efforts to build it on Windows in conjunction with vkpkg for libpng and zlib. This should work but probably needs a little more effort.

I'd be grateful for any feedback, ideas, bug reports or whatever. Here's the slightly weird thing: i'm not personally interested in writing GRFs. But for some reason I don't really understand, I have been fascinated by the NewGRF specs, and I enjoy writing productivity tools. So... I was hoping that this tool might make it possible to make a few minor mods to a GRF, perhaps with features that are not yet supported in NML. If it has some legs, I'll be sure to maintain it. If not, well, no matter. I learned loads from doing it.

The code theoretically covers the entire current specs, but i am sure to have missed something. It is designed to be simple to extend with new properties. Most properties values are rendered as plain numbers (this was quick and convenient), but it would be simple to convert relevant items to enumerations or bit fields.

If you are interested, the project can be found here: https://github.com/UnicycleBloke/yagl.git.

Anyway, this has been a huge task, and I probably need a breather.
Post Reply

Return to “NewGRF Technical Discussions”

Who is online

Users browsing this forum: No registered users and 1 guest