YAGL - Yet Another GRF Language (maybe)

[UnicycleBloke]

This should result in a ridiculously high $6.3 billion for cost. But no matter what I set base cost to, it doesn't affect cost factor. When I adjust cost factor between 00 and FF, it ranges appropriately from $0 to $97k.

I'm afraid I don't actually know much about creating GRFs. I have focused on understanding the details of the specs to create a tool rather than how to use them productively. With that in mind, is it possible that I am reading some numbers incorrectly due to a misreading of the spec. Could you let me have the original GRF and whatnot so I can investigate? Happy to do this via PM.

Looking at the BaseCosts, there are many different values, and these are indexed with the instance_id in the YAGL. Could it be that the index 0x0000 you are using does not relate to the base cost you need to modify? https://newgrf-specs.tt-wiki.net/wiki/BaseCosts. For example, FIRS has the following code in it. I think I have interpreted the meaning correctly, but the settings look fishy compared to the default values in the specs:

Code: Select all

// Record #1598
properties<GlobalSettings, 0x0030> // Action00 <<<---- funding industries
{
    // instance_id: 0x0030
    {
        cost_base_multipliers: 0x0A;
    }
}
// Record #1599
properties<GlobalSettings, 0x003A> // Action00 <<<---- build raw industry
{
    // instance_id: 0x003A
    {
        cost_base_multipliers: 0x0A;
    }
}
// Record #1600
properties<GlobalSettings, 0x0031> // Action00 <<<---- remove industry
{
    // instance_id: 0x0031
    {
        cost_base_multipliers: 0x0A;
    }
}

In time, I would like to replace a lot of numbers like this with more descriptive names... And I must admit that the layout I've adopted leaves something to be desired. It is verbose. It groups properties by instance, which works well for vehicles and the like, but the global settings are less clear. I have an alternative in mind, but this is a problem for another day.

[wallyweb]

This should result in a ridiculously high $6.3 billion for cost. But no matter what I set base cost to, it doesn't affect cost factor. When I adjust cost factor between 00 and FF, it ranges appropriately from $0 to $97k.

Here are some thoughts on costs. It doesn't point to your problem, but it might give you thoughts on where to look:

The cost based multiplier affects all costs in a feature.
It can be reset in one GRF but if not reset in a subsequent GRF of the same feature, the subsequent GRF will prevail.
The NweGRF specification gives us the formula. It also discusses rounding errors.
If you are using more than one ship GRF, what happens when you move the last one to above the first one?

The cost factor affects each vehicle individually. It references the cost based multiplier, but it does not change the cost based multiplier.

I got into this when New Map Features and then JGRPP extended support to add three new bridge IDs. Definitely a mind bender.

[kamnet]

Okay, perhaps I should avoid using a global base cost modification then, it's probably not achieving what I want.

I'm also going to try to not clog this with project-specific questions...

Anyhow, my next question - is it possible to support \w<date> in date fields?

[UnicycleBloke]

Anyhow, my next question - is it possible to support \w<date> in date fields?

I'm not exactly au fait with NFO escape codes. Does this one create a word value, which you write in decimal? \w1956 => 0x07A4 => A4 07 (little-endian)? I guess \b and \d are similar.

YAGL allows you to specify numbers in one of several formats. See https://github.com/UnicycleBloke/yagl/b ... _syntax.md. The four supported formats are decimal, hexadecimal, octal and binary. Hexadecimal numbers are always preceded with 0x to distinguish them. Octal numbers always begin with 0. Binary numbers always begin with 0b. The decimal number 123 is 0x7B in hex, 0173 in octal, and 0b01111011 in binary. The decimal number -1 is 0xFFFF in hex (for a word), and 0b11111111 in binary (for a byte). You can use these formats interchangeably. If the GRF expects a byte, and you enter a value larger than 255, you should get an error when you encode the YAGL. Similar for words and double words. Not sure anyone ever uses octal, but I added it anyway. I thought binary could be useful for properties which are sets of flags.

For now, I have generally dumped numbers as hex because this seemed the closest to NFO for those familiar with that, and it was an easy default while I got on with other things. My vision is to gradually replace at least some of these with more helpful formats for dates, enumerations, flags and so on. I did the climate availability as a flag set (a bit like NML) as a demo, but there are soooo many properties... Some properties, such as special effects on vehicles, are quite complex. I'll have to think of something for those too.

[kamnet]

Action 0, Property 00 is defined as:

Date of introduction
The date is specified as number of days since 1920 where TTD counts 365,25 days in a year. It will have a random number from 0 to 511 days added to it at the start of every new game...
The \w<date> escape sequence is especially useful here. See the discussion of escape sequences for further information.

Discussion of escape sequences goes on to explain:

<date> is a date in one of the four formats YYYY-MM-DD, YYYY/MM/DD, DD-MM-YYYY, or DD/MM/YYYY. In all cases, leading zeros may be omitted.

It would be insanely helpful if one could drop a date format into the code and YAGL perform the calculations, as grfcodec does, rather than try to calculate days by hand and account for leap years, etc.

Calculation in decimal can be done using an online tool. Dates forward from 1920 work out fine, but negative decimals for dates earlier than 1920 do not seem to work.

[UnicycleBloke]

Perfect. Just what I needed to see. Couldn't find it myself. I will see what I can do. I am theoretically at work right now, so I'll have a look this evening.

[UnicycleBloke]

OK. So I've had a dig around the grfcodec source to see what it does, and have implemented something vaguely similar to support years and dates.

Code: Select all

    
\b<dec>     Just enter a number in decimal, as already mentioned.
\b<year>    "year(<year>)" is translated to a number which is <year> - 1920. This only works where a byte value is expected.
\b*<dec>    Just enter a number in decimal.
\w<dec>     Just enter a number in decimal.
\wx<hex>    Just enter a number in hexadecimal.
\w<date>    "date(<yyyy>/<mm>/<dd>)" is translated to a number which is the number of days since the start of 1920 (where a word is expected).
\d<dec>     Just enter a number in decimal. 
\dx<hex>    Just enter a number in hexadecimal.
\d<date>    "date(<yyyy>/<mm>/<dd>)" is translated to a number which is the number of days since the start of year zero (where a double word is expected).

This only works when encoding YAGL. The decoder will simply stick numbers into the output (as now). I could fix this, but it is more complicated as I need to identify just those properties where a date is valid, and then make changes to the way each of those specific properties is handled. In time.

I must admit to being a bit confused by having multiple date formats in the binary data. What are the rules for using them? What happens for dates before 1920 with the word format? What is the earliest date that the game supports?

I tested the changes with the following YAGL:

Code: Select all

properties<Trains, 0x0087> // Action00
{
    // instance_id: 0x0087
    {
        introduction_date: date(1920/1/1);      // Resolves to 0 days since 1920 is the base.
        long_introduction_date: date(1920/1/1); // Resolves to 701265 days since year zero.

It encoded just fine, and when I decoded it again, I got this output:

Code: Select all

properties<Trains, 0x0087> // Action00
{
    // instance_id: 0x0087
    {
        introduction_date: 0x0000;
        long_introduction_date: 0x000AB351; // This matches the documentation.

So that seems to basically work. I'm not super-happy with this solution because you could use "date()" and "year()" in places for which this doesn't really make sense. They just calculate numbers, so it's not a big problem.

[wallyweb]

I must admit to being a bit confused by having multiple date formats in the binary data. What are the rules for using them? What happens for dates before 1920 with the word format? What is the earliest date that the game supports?

The details are here.

[UnicycleBloke]

The details are here.

That's not really what I meant. I understand - I think - how to calculate the numbers that go into the GRF (after looking at grfcodec source). But I'm unclear about how dates earlier than 1920 work in practice. All vehicles have both a short and a long introduction date. The short one is defined as days since 1920 and the long one as days since 0. I guess this is kind of historical, because OTTD started in 1920 or something, but OpenTTD has extended this. I am worried about negative numbers. So, is it the case that short dates only make sense for dates after 1920?

I have similar questions regarding years available, which appears to have a byte format in some cases and a word format in others.

[wallyweb]

The details are here.

That's not really what I meant. I understand - I think - how to calculate the numbers that go into the GRF (after looking at grfcodec source). But I'm unclear about how dates earlier than 1920 work in practice. All vehicles have both a short and a long introduction date. The short one is defined as days since 1920 and the long one as days since 0. I guess this is kind of historical, because OTTD started in 1920 or something, but OpenTTD has extended this. I am worried about negative numbers. So, is it the case that short dates only make sense for dates after 1920?

I have similar questions regarding years available, which appears to have a byte format in some cases and a word format in others.

Thinking BC/AD, only AD is supported so there are no negative numbers.
The short format is the number of days since 1920. This is historical: Transport Tycoon and TTDPatch.
The long format is the number days since year 0.
Action0 Vehicles probably explains it best.

[UnicycleBloke]

Action0 Vehicles probably explains it best.

Yeah. I looked at that. I think I have it now. Dates are positive only. Dates earlier than 1920 can only be specified using the long date format. I imagine "years available" are similar.

Thanks.

EDIT: Just looking at the OpenTTD source really clarifies things.

[UnicycleBloke]

So I've improved the date handling a bit. The long dates used for Trains, Vehicles, Ships, Aircraft, Objects, RailTypes, RoadTypes and TramTypes are now *decoded* into a date format like this:

Code: Select all

// Record #472
properties<Trains, 0x0086> // Action00
{
    // instance_id: 0x0086
    {
        long_introduction_date: date(1883/1/1);
        model_life_years: 110;
        retire_vehicle_early: 0x3B;

The expected format is date(Y/M/D). This replaces the use of numbers I had before (that was a place holder). I've tested it a bit with dutchtrains.grf. All the dates seemed match the NML source, but please let me know if you get funny numbers: it was a bit of a faff to get the numbers to reliably translate to dates and back again. Number 0 means 1 Jan 0000. Number 701265 means 1 Jan 1920 (as in the specs). The historical short date format used for vehicle types also works, and the offset from 1920 is managed internally.

I guess I'll look at availability years next...

[Michi_cc]

This should result in a ridiculously high $6.3 billion for cost. But no matter what I set base cost to, it doesn't affect cost factor. When I adjust cost factor between 00 and FF, it ranges appropriately from $0 to $97k.

You are not setting the proper base cost for ships. If you look at https://newgrf-specs.tt-wiki.net/wiki/BaseCosts, you'll find ship purchase at index dec 19/hex 13, which is the instance id you'd have to use (and not 0).

[UnicycleBloke]

I have spent the last little while adding some regression tests to the project to facilitate refactoring. I found a few little bugs and inconsistencies along the way, and things that had not been fully implemented (like station layouts), but nothing catastrophic. The testing includes sample YAGL text for every type of block that is supported, and every known property for every feature in Action00. It should now be trivial to add more examples which exercise different features of the NewGRF specifications.

Using sample YAGL got me to wondering whether I could include a limited form of documentation directly within the executable so that, for example, you could get it to print out an example Action0A or whatever, or to list the permitted values for the various enumerations such as feature type: Trains, Aircraft, ... Still thinking about that.

Anyway... please download and let me know if there are any issues, feature requests or whatever.

Al

[UnicycleBloke]

I've fixed a couple obscure-ish issues related to missing sprites (thanks for the heads up, kamnet) and sound files stored in the graphics section of Container2 GRFs.

Also added nicer decoding for visual effect property to split out type, position and wagon power (only for trains so far, but easy to extend):

Code: Select all

visual_effect: effect(SteamPuffs, 0x08, Disable);

I probably ought to return to working on the documentation...

yagl.zip

(644.39 KiB) Downloaded 394 times

[kamnet]

Using yagl (Yet Another GRF Language) v0.4-100-g5c0b8b3.
yagl -d <x>.grf works fine.
yagl -e <x>.grf does not produce any errors, but also doesn't produce a file.

[UnicycleBloke]

Using yagl (Yet Another GRF Language) v0.4-100-g5c0b8b3.
yagl -d <x>.grf works fine.
yagl -e <x>.grf does not produce any errors, but also doesn't produce a file.

That seems very strange indeed. I've just rebuilt this commit and tried it with several GRFs. It seems to work fine. Are you seeing this with one particular GRF? Or any GRF?

EDIT: I subsequently downloaded the image above and tried that too, just in case. It seems to work. Something pretty bad would have to be wrong for me to not write the GRF. I'm wondering if it has somehow been placed in a different location that expected - also quite surprising. Or maybe it's something to do with file permissions?

The code unconditionally does the following final step during encoding.

Code: Select all

// Write out the GRF file ...
std::cout << "Writing GRF..." << std::endl;
std::ofstream os(options.grf_file(), std::ios::binary);
grf_data.write(os);

The value of options.grf_file() - the relative path to the GRF output file - should be printed as part of the earlier output. What does it say? I get something like "Writing GRF: ..\..\grfs\AuzLandscapeObjects.grf".

I'll have to check, but if this fails to open the file there may be no exception. Then the write might appear to succeed, but all the data would effectively be sent to /dev/null. This seems the most plausible answer.

EDIT2: To answer my own question, the file stream does not throw an exception if it fails. There are any number of reasons why this might fail. Could you try to create a file called <x>.grf in the relevant folder and see what the OS says? And I'll have to check that opening the file succeeded.


Al

[yorkvisk]

I have recently been working on a tool which reads a GRF file into memory, and then writes it out using a human-readable text format (at least, hopefully more readable than NFO), along with sprite sheets. This is not yet complete, but is coming along quite well. It's largely a matter of filling in the blanks now. I'm aiming for complete coverage of the NewGRF specs, which might entail a bit of digging in the OTTD source and/or questions.

For an example, the Action08 pseudo-sprite is currently rendered as:

Code: Select all

grf // Action08
{
    grf_id: "\xFB\xFB\x06\x01";
    version: 8;
    name: "Dutch Trainset 2.1.0";
    description: "{lt-gray}Dutch Trains for OpenTTD {new-line}{black}First vehicle: 1839.{new-line}{new-line}(c)Dutch Trainset Team {new-line}License: GPLv2 or higher. {new-line}See readme for details.";
}

It looks vaguely reminiscent of NML but is most definitely *not* NML. That would be a reverse engineering task beyond my skills, and probably impossible.

For another example, a VarAction02:

Code: Select all

switch<Trains> // Action02 variable
{
    this_set_id: 0xEC;
    test: (year_of_construction & 0xFFFFFFFF);
    switch (test)
    {
        0x00000000..0x000007BE: 0x00F0; 
        0x000007BF..0x000007C5: 0x00F4; 
        0x000007C6: 0x00F5; 
        0x000007C7..0x000007CE: 0x00F4; 
        0x000007CF..0x000007D3: 0x00F8; 
        0x000007D4: 0x00F1; 
        0x000007D5..0x000007D6: 0x00F8; 
        0x000007D7: 0x00EC; 
        default: 0x00EB;
    }
}

I've pulled out the name of the variable being tested here, though this idea might not work out as well as I'd like. It would be nice to format the values in the ranges to match the variable (decimal in this case 0x7BFE = 1982), but that's a refinement for later. Nothing about the text format is set in stone: I'm currently just trying to get something reasonable done for every type of pseudo-sprite.

A secondary, but maybe more interesting, goal for this tool is that it should be able to read its own text files and sprite sheets back into memory, and then write out the data as a new GRF file. This is why the output has a lot of apparently superfluous braces and semicolons. This feature has become very important in my mind, but is much less complete than writing the text file. I pretty much know how to do it, but my approach to parsing the text is a bit tedious and verbose, and there is quite a lot to do. Again, it's largely a matter of filling in the blanks.

Anyway... the reason for posting is that I've been working on this for a little while now, and I kind of wanted to give a heads up. And maybe get some direction from feedback. Obviously NML (or M4NFO, I guess) is the way to go for most GRF development, but I wondered if there is still a place for something more low level. The main reason I started this was curiosity about how NML constructs map on to the NewGRF specs. And then I got a bit carried away...

Hi, I can't install Yagl. I'm using the latest termux with Ubuntu 20, Cmake (latest), g++ 9, I can't install.
I tried the normal termux with Clang 15.0 and also could not, detail: all dependencies libpng/zlib/python are install.

It happens this error when try to give the command cmake .. ¦ within the build directory

Code: Select all

-- The CXX compiler identification is Clang 15.0.7
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /data/data/com.termux/files/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:130 (if):
  if given arguments:

    "EQUAL" "1"

  Unknown arguments specified


-- Configuring incomplete, errors occurred!
See also "/storage/emulated/0/download/yagl/build/CMakeFiles/CMakeOutput.log".

[UnicycleBloke]

Thank you for trying this out. I was pretty sure the build was working under Linux. Oh well...

I got the same error on my system. The issue relates to a variable which should default to FALSE if not defined on the command line. One of the many vagaries of CMake that pickle my noggin. I fixed this and immediately hit a compilation error due to a missing #include. Hard to see how that would have compiled in the past. It compiles for me using GCC 9.4.0 under WSL.I haven't tried clang. Note that you will need to install libpng-dev if you don't have it already (https://github.com/UnicycleBloke/yagl/b ... d_linux.md).

Please pull the latest from https://github.com/UnicycleBloke/yagl and see if that works for you.

Al

[JustMoreMemes]

I've found that trying to open a lot of NewGRFs just results in an error, rather than extracting anything. But not everything is effected, older NewGRFs tend to open up fine. Do you know what could be causing this? Here is the readout I get from decoding two different NewGRFs:

Microsoft Windows [Version 10.0.19045.4291]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>cd C:\Users\Fierce PC\Desktop\Programs\NewGRF Creation\YAGL

C:\Users\Fierce PC\Desktop\Programs\NewGRF Creation\YAGL>yagl.exe -d dropjffork.grf

yagl (Yet Another GRF Language) v0.4-0-g13d5ee5
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Reading GRF: dropjffork.grf
Writing YAGL: sprites\dropjffork.yagl
Output directory: sprites
Image base: sprites\dropjffork

Reading GRF...
Number of records: 608
Property error: Unknown property: property=0x16 [at line 259 in source file C:\Projects\yagl\records\features\Action00GlobalSettings.cpp]

C:\Users\Fierce PC\Desktop\Programs\NewGRF Creation\YAGL>yagl.exe -d hoverbus.grf

yagl (Yet Another GRF Language) v0.4-0-g13d5ee5
Copyright 2019 Alan Chambers (unicycle.bloke@gmail.com)
Released under GNU General Public License version 3

Reading GRF: hoverbus.grf
Writing YAGL: sprites\hoverbus.yagl
Output directory: sprites
Image base: sprites\hoverbus

Reading GRF...
Number of records: 17
Writing YAGL and other files...
Writing sprite sheet: sprites\hoverbus-8bpp-normal-0.png...
Writing YAGL script...

C:\Users\Fierce PC\Desktop\Programs\NewGRF Creation\YAGL>