YAGL - Yet Another GRF Language (maybe)

[UnicycleBloke]

This is because the NewGRF specs have been extended. The code was recently updated by a kind contributor but I need to publish a fresh build. It's on my list but I've been busy of late. Sorry. I'll try to make time this weekend.

[UnicycleBloke]

I have created a new release of yagl at https://github.com/UnicycleBloke/yagl/releases/tag/v1.2. GlobalSettings has a couple of properties which apparently did not exist when I wrote the code. I have updated the code to include these.

Please try this to see if it fixes the problem. If not, I would be very grateful if you could provide the offending NewGRF. I don't play OTTD at the moment and don't have any recent files. Bananas seems to list them all as available-in-game. I did try installing the game on Linux but was unable to work out where downloaded NewGRF files are placed in the file system.

It's been a while since I looked at this code. Hopefully all is well.

[kamnet]

C:\Users\Fierce PC\Desktop\Programs\NewGRF Creation\YAGL>yagl.exe -d dropjffork.grf

The source code for Drop JF Fork is here, it would be best to work from that.

viewtopic.php?p=1257036#p1257036

[UnicycleBloke]

Thanks for the link, kamnet. I tried out the new version and had indeed fixed the unknown property, but then there was another to fix... They were related to roadtype/tramtype translation tables. This is a feature introduced since I wrote the code. There are sure to be some others lurking here and there. It's easy to support new properties, but I haven't been proactive keeping up to date with the NewGRF specs. I hadn't really expected that people would use yagl for recent GRFs for which there is NML source.

I would like to bring the code up to date with the specs but I'm not sure when I'll find the time...

[smallfly]

Hey @UnicycleBloke! Thanks for the YAGL tool! I used it to decompile the NewGRF "Larger ship" by Bilbo. Then I encoded the .yagl with with YAGL and tested in game. Worked fine BUT when uploading to BaNaNaS, BaNaNaS does not allow the upload and says "["File 'fast_ships.grf' failed validation", 'fast_ships.grf: Unexpected end-of-file.']" I have attached the .yagl file as well as the grf generated by YAGL.

[UnicycleBloke]

To answer your question on Discord, yagl does support both Container1 and Container2 formats. I checked and you would only need to change one line in the YAGL code to change the format. I think someone suggested that BaNaNaS does not support Container1. The default in yagl is Container2.

The other thing I noticed while debugging is that the GRF version is given as GRF6. The default in yagl is GRF8. I have no idea what BaNaNaS expects in this regard, but maybe try GRF8.

Please let me know if making these changes fixes the issue.

[smallfly]

Just switched to "Container2" and BaNaNaS was happy Thanks!

[warp]

Hello Alan,
First of all, thank you for the useful tool
I'm using the latest version 1.2.2
I just decoded a GRF and without any editing of the created yagl file wanted to encode it back to the grf.
But there is an issue by encoding.

Code: Select all

ERROR in record #6844: YAGL parser error: Unexpected match token: ',' at line 108802 column 36 in yagl\testgrf.yagl
  [at line 40 in source file C:\Projects\yagl\records\TokenStream.cpp]

The said line is as follows:

Code: Select all

random_switch<Trains, 0x52, Consist, BackwardFromSameID[0x00]> // Action02 random

The problematic ',' is the one after the 'Consist'

According to the documentation https://github.com/UnicycleBloke/yagl/b ... _random.md should not give any problem with it.
If I'm not wrong -It seems like, it is perhaps just a parser problem.

[UnicycleBloke]

Thanks for letting me know. It is a bit odd because the code looks OK. I'll have to debug into this. Could you share the specific GRF? Perhaps I am looking at a newer version of the code, but I haven't touched it for a long time.

If you are curious:

Code: Select all

random_switch<Trains, 0x52, Consist, BackwardFromSameID[0x00]> // Action02 random

Code: Select all

    is.match_ident(RecordName(record_type())); // Consumes "random_switch"
    is.match(TokenType::OpenAngle); //Consumes "<"
    m_feature = FeatureFromName(is.match(TokenType::Ident)); // Consumes "Trains"
    is.match(TokenType::Comma); // Consumes first ","
    m_set_id = is.match_uint8(); // Consumes the value 0x52
    is.match(TokenType::Comma); // Consumes second ","
    random_desc.parse(m_type, is); // Consumes "Consist" and sets m_type to RandomType::Consist - or should

    // It seems like this condition is false for some reason. 
    if (m_type == RandomType::Consist)
    {
        is.match(TokenType::Comma); // Consumes third ","
        consist_desc.parse(m_method, is); // Consumes "BackwardFromSameID"
        is.match(TokenType::OpenBracket); // Consumes "["
        m_count = is.match_uint8(); // Consumes the value 0x00 
        is.match(TokenType::CloseBracket); // Consumes "]"
    }

    is.match(TokenType::CloseAngle); // Consumes ">"

I'm sure there are smarter, more data-driven approaches to parsing, but I found the conditional nature of at least some of the records quite hard to deal with. I'd likely have spent ages debugging my abstraction.

[warp]

...
Could you share the specific GRF?
...

sure, the GRF is Japan Set3 Trains (GPL2)
Thanks for looking after that issue.

warp

[UnicycleBloke]

Aha! My commit history shows I fixed this almost a year ago! I had a flurry of working on the code for a bit, but other things became more important. I recall I was persuaded to support multipart YAGL files (some kind of trickery with GCC to concatenate files) - that may not work properly. Anyway, I should at least make a new release. For now I pushed a new tag 1.2.3 if you want to build the code yourself.

Code: Select all

/mnt/c/Projects/yagl/build$ ./yagl -d ../grfs/jptrains.grf

yagl (Yet Another GRF Language) v1.2.2-16-g903c367-m
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Reading GRF:      ../grfs/jptrains.grf
Writing YAGL:     ../grfs/sprites/jptrains.yagl
Output directory: ../grfs/sprites
Image base:       ../grfs/sprites/jptrains

Reading GRF...
Number of records: 18910
Writing YAGL and other files...
Writing sprite sheet: ../grfs/sprites/jptrains-8bpp-normal-0.png...
Writing sprite sheet: ../grfs/sprites/jptrains-8bpp-normal-1.png...
Writing YAGL script...
mnt/c/Projects/yagl/build$ ./yagl -e ../grfs/jptrains.grf

yagl (Yet Another GRF Language) v1.2.2-16-g903c367-m
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Reading YAGL:     ../grfs/sprites/jptrains.yagl
Writing GRF:      ../grfs/jptrains.grf
Source directory: ../grfs/sprites
Image base:       ../grfs/sprites/jptrains

Lexing YAGL...
Parsing YAGL (567735 tokens) ...
Opening sprite sheet: ../grfs/sprites/jptrains-8bpp-normal-0.png...
Opening sprite sheet: ../grfs/sprites/jptrains-8bpp-normal-1.png...
WARNING: Sprite #00004918 contains 1 pure white pixels. Its YAGL rectangle may be misaligned or too large.
    The first is at [346, 1572] in sprite sheet jptrains-8bpp-normal-1.png
Creating back up GRF: ../grfs/jptrains.grf => ../grfs/jptrains.grf.bak
Writing GRF...
/mnt/c/Projects/yagl/build$

[warp]

...
For now I pushed a new tag 1.2.3 if you want to build the code yourself.

That's great, many thanks for the quick fixing the issue.
Unfortunately I have problem to compile the new version on my system. I'm on Windows 10 and I do not have any MS compiler installed. Instead I use MSYS2 environment. But it seems I'm not able to compile. I got this:

Code: Select all

User@PC UCRT64 /e/Projects/OpenTTD/NewGRF/_yagl-1.2.3/build
$ cmake ..

-- Building for: Ninja
-- The CXX compiler identification is GNU 15.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: D:/usr/msys64/ucrt64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:249 (message):
  ERROR: The directory given by VCPKG_DIR does not exist - specify with
  -DVCPKG_DIR=<dir>

-- Configuring incomplete, errors occurred!

User@PC UCRT64 /e/Projects/OpenTTD/NewGRF/_yagl-1.2.3/build
$

I have no idea what to do with that VCPKG

[UnicycleBloke]

There are some build instructions in the repo about setting up vcpkg for yagl. I'm a vcpkg beginner myself and haven't used it elsewhere. I found it a bit fiddly and fragile. WSL is a *lot* simpler. The only dependency is libpng.

Hmm... I note that my instructions for vcpkg say they didn't work (I followed them on my new PC). I would like to sort this out, but I'm neck deep in another project at the moment.

EDIT: MSYS2? Never used it, but does that even need vcpkg? Isn't it a Linux-ish command line but builds Windows executables. I own my ignorance.

[warp]

There are some build instructions in the repo about setting up vcpkg for yagl. I'm a vcpkg beginner myself and haven't used it elsewhere. I found it a bit fiddly and fragile. WSL is a *lot* simpler. The only dependency is libpng.

Hmm... I note that my instructions for vcpkg say they didn't work (I followed them on my new PC). I would like to sort this out, but I'm neck deep in another project at the moment.

EDIT: MSYS2? Never used it, but does that even need vcpkg? Isn't it a Linux-ish command line but builds Windows executables. I own my ignorance.

Today I gave it an another go, but still face some problems.
The cmake config seems to be made for full git repo package. But the v1.2.3 is a separate package and the make process fails.

Code: Select all

User@PC  UCRT64 /e/Projects/OpenTTD/NewGRF/_yagl-1.2.3/build
$ cmake ..
-- Building for: Ninja
-- The CXX compiler identification is GNU 15.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: D:/usr/msys64/ucrt64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.1s)
-- Build files have been written to: E:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/build

User@PC  UCRT64 /e/Projects/OpenTTD/NewGRF/_yagl-1.2.3/build
$ ninja
[1/140] C:\Windows\system32\cmd.exe /C "cd /D E:\Projects\OpenTTD\NewGRF\_yagl...yagl_version.cpp -P E:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/yagl_version.cmake"
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-- Repo version:
[2/140] Building CXX object CMakeFiles/yagl_lib.dir/application/CommandLineOptions.cpp.obj
FAILED: CMakeFiles/yagl_lib.dir/application/CommandLineOptions.cpp.obj
D:\usr\msys64\ucrt64\bin\c++.exe  -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/. -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/records -IE:/Projects/OpenTTD/NewGRF/_yagl
-1.2.3/records/actions -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/records/descriptors -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/records/graphics -IE:/Projects/Open
TTD/NewGRF/_yagl-1.2.3/records/features -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/utility -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/application -IE:/Projects/Open
TTD/NewGRF/_yagl-1.2.3/tests -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/version -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/third_party/catch2 -IE:/Projects/OpenTTD/
NewGRF/_yagl-1.2.3/third_party/png++ -IE:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/third_party/cxxopts/include -std=gnu++17 -MD -MT CMakeFiles/yagl_lib.dir/applicati
on/CommandLineOptions.cpp.obj -MF CMakeFiles\yagl_lib.dir\application\CommandLineOptions.cpp.obj.d -o CMakeFiles/yagl_lib.dir/application/CommandLineOptions.cpp
.obj -c E:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/application/CommandLineOptions.cpp
In file included from E:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/application/CommandLineOptions.cpp:21:
E:/Projects/OpenTTD/NewGRF/_yagl-1.2.3/utility/FileSystem.h:23:14: fatal error: features.h: No such file or directory
   23 |     #include <features.h>
      |              ^~~~~~~~~~~~
compilation terminated.
... CUT ...

Maybe I will wait until you have some time and compile the v.1.2.3

[UnicycleBloke]

Ah. Sorry. It looks like something to do with a feature test for std::filesystem which might now be redundant or done differently in your toolchain. I recall it was irritating at the time that this was even necessary.

[UnicycleBloke]

https://github.com/UnicycleBloke/yagl/r ... tag/v1.2.3

Apologies for the delay in releasing this version. I actually fixed the random action bug a year ago, but then got distracted.

Please let me know if there are any issues. The only way this thing ever moves forward is in response to user feedback. I''m happy to fix bugs and incorporate additions to the NewGRF specs, but someone needs to shout at me so I notice. I'll get an email if you add a comment here.

I have also added a kind of "info" feature to make the various property names used in YAGL a bit clearer. For example:

Code: Select all

PS C:\Projects\yagl\build> .\Release\yagl.exe -i Feature:Ships

yagl (Yet Another GRF Language) 1.2.3-0-g903c367
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Properties for feature 'Ships':
    0x00  introduction_date: date(1920/1/1);
    0x02  reliability_decay_speed: 0;
    0x03  vehicle_life_years: 0;
    0x04  model_life_years: 0;
    0x06  climate_availability: null;
    0x07  loading_speed: 0x00;
    0x08  sprite_id: 0x00;
    0x09  is_refittable: false;
    0x0A  cost_factor: 0x00;
    0x0B  speed_2_kmh: 0x00;
    0x0C  cargo_type: 0x00;
    ...
 

It can also list all the names YAGL used for string codes:

Code: Select all

PS C:\Projects\yagl\build> .\Release\yagl.exe -i StringCodes

yagl (Yet Another GRF Language) 1.2.3-0-g903c367
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Basic string control codes:

    Code  NArgs  Name          Description
    ----  -----  ------------  -----------
    0x01    01   x-off         X offset in next byte of string (variable space)
    0x0D    00   new-line      New line
    0x0E    00   small-font    Set small font size
    0x0F    00   large-font    Set large font size
    0x1F    02   xy-offs       X and Y offsets in next two bytes of string
    0x22    00   dq            Double quote
    0x7B    00   sd            Print signed dword
    0x7C    00   sw            Print signed word
    0x7D    00   sb            Print signed byte
    ...
 

If this proves useful, it might be possible to add more such guidance. Please let me know.

There is also now support for **bash**-style comments. I understand NML files can be composed from multiple PNML files using **gcc** (viewtopic.php?t=83212). I don't know much about this usage, but was asked to add something similar to YAGL. The parser interprets bash comments inserted by **gcc** to keep track of the original source file and location (PYAGL?) in case of errors. At least, that's the idea. This is experimental and I can't really imagine anyone using YAGL in this way... But if you do try it out, please do let me know if it's broken.

[warp]

Hello Alan,

many thanks for releasing the v1.2.3 and for committing yourself to look after it in the future

warp