Page 2 of 3

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 10 Feb 2020 19:23
by andythenorth
Hi :)

I don't have any feedback on the project, but I didn't want this thread to pass by with only cricket noises.

It's an impressive thing to have done :D

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 10 Feb 2020 22:33
by UnicycleBloke
I don't have any feedback on the project, but I didn't want this thread to pass by with only cricket noises.
Thanks. I really appreciate that.

In fact, I have recently been speaking to wallyweb, who gave it a quick spin, about how to get a little feedback. To this end, I have switched to working in Windows 10 and created a self-contained binary which hopefully people can run without any trouble. I mostly develop embedded firmware, so I'm a bit new to this whole release-online-for-multiple-platforms thing. For some reason I have not yet determined, the Release build (smallish) is unhappy, but the Debug build (largish) is fine: so I have released the larger image... https://github.com/UnicycleBloke/yagl/releases/tag/v0.3. This was built with Visual Studio 2019 for x64 if it matters.

I'm calling this release an Alpha because there are sure to be a ton of little things wrong with it. I'm pretty sure the coverage of the NewGRF specs is complete or very nearly so, including some stuff which I had to find in the OpenTTD source, but I did spot a lacuna just yesterday...

During development, I have focused on reasonably sizeable GRFs such as FIRS in order to get better coverage, but suggest playing with something small at first. In truth, it seems more likely to be useful for the smaller projects which people might be creating in NFO at the moment, but I'm only guessing. I feel confident that it will fall over pretty soon on some silly oversight. Obviously there is not much in the way of documentation: I'm hoping that the YAGL output is at least half-way comprehensible.

Anyway, please give it a whirl and let me know when it goes bang.


Al

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 11 Feb 2020 08:14
by jfs
If a Release build is broken but a Debug build works, my first suspicion is that you depend on some undefined behaviour, which causes code to compile to one thing in a debug build, and something else when you turn on optimizations.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 11 Feb 2020 08:52
by UnicycleBloke
undefined behaviour
Maybe. The particular error code I get seems related to heap corruption. This is not generally a feature of my code, but no one is perfect. I have used shared_ptr and STL containers throughout, and perform no direct resource management. I'll see if I can track it down this evening.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 11 Feb 2020 22:13
by UnicycleBloke
jfs wrote:undefined behaviour
Right first time: twice incrementing a counter between sequence points in the chunked tile decoder. At least it wasn't too hard to find.

This release build executable should work:
yagl.zip
Zipped Windows 10 64-bit EXE
(396.22 KiB) Downloaded 234 times
Please let me know if not.

The repository is at https://github.com/UnicycleBloke/yagl.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 20 Feb 2020 03:49
by Gadg8eer
Wait, so is this a method of decompiling NFO, a new programming language for OpenTTD mods, both, or nether? It looks impressive.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 20 Feb 2020 09:03
by UnicycleBloke
Gadg8eer wrote: 20 Feb 2020 03:49 Wait, so is this a method of decompiling NFO, a new programming language for OpenTTD mods, both, or nether? It looks impressive.
Thanks very much. :)

yagl is a tool for both compiling and decompiling GRFs from and to a new script language, YAGL. This is broadly comparable in function to grfcodec, which compiles and decompiles GRFs from and to NFO. Unlike NML, YAGL is a low level script: there is a direct one-to-one correspondence between blocks in YAGL and the equivalent records in NFO. The difference is that YAGL is intended to be more human readable than NFO, and things like the sizes of structures, lengths of arrays, lengths of strings, and so on are worked out automatically.

My original goal was just to decompile GRFs created with NML to see what was inside them in terms of the low level NewGRF specs, but I got carried away. I think the most likely use case would be to decompile an existing GRF, make a few mods (e.g add or fix some translations, or whatever), and then re-compile. If you have the NML source for the GRF, using nmlc would probably be the better choice. There is no reason why you could not create a whole new GRF written in YAGL, if you were so minded. This would be equivalent to creating a GRF in NFO. The major barrier to this is that I have not yet written any significant documentation. I'm thinking about that... Also yagl does not have any of the linting capabilities that grfcodec does. I'm not sure to what extent they are needed. Adding specific checks for things wouldn't be too hard.

Since yagl has the ability to read and write a GRF into memory, it would in principle be quite simple to read and write NFO too (at lot easier than YAGL was, at any rate). I don't know if that would be useful at all.

Please have a go with it, and let me know how it works out. There are sure to be some bugs. If you simply decompile/recompile your favourite GRF, and then it no longer works in openttd, I'd very much like to know about that. As far as I can tell, all the discrepancies between the before and after GRFs are innocuous.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 21 Feb 2020 00:09
by kamnet
I would love to see some documentation and tutorials developed. I simply have not been able to wrap my head around NML, I seem to have a little easier time with NFO, but that's hard for a lot of people to work on. Perhaps this can be a nice in-between and help me step up into NML, or maybe just be able to start cranking out some projects faster.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 22 Mar 2020 21:40
by UnicycleBloke
kamnet wrote: 21 Feb 2020 00:09 I would love to see some documentation and tutorials developed.
I have been a bit distracted but have started a little documentation. It's mostly a pretty thin thing at the moment, barely a beginning with some structure but lots of empty pages, but I would be very grateful for feedback. These pages and a few others are worth a look:
https://github.com/UnicycleBloke/yagl/b ... s/intro.md
https://github.com/UnicycleBloke/yagl/b ... _syntax.md
https://github.com/UnicycleBloke/yagl/b ... ction01.md

The key thing to note is that YAGL blocks directly map onto NFO records. If you understand NFO, my hope is that using YAGL will be a breeze. For this reason, I am hoping that the documentation need not be too much more than a gloss over the existing NewGRF specs on which I based my software. I at least need to explain the structure of each block. And I use a whole bunch of enumerations with keywords like "Aircraft" where NFO would have the number 03 for that feature. I need to explain those, too.

But even before all that is complete there is nothing preventing you from using grfcodec to compile one of your existing NFO projects, and then decompiling with yagl just to have a look. It will be a lot easier to know what to add if people actually have a go and then ask questions, or complain, or whatever. :)

I have attached the latest Windows version of the executable, which includes a number of minor changes and fixes related to 32bpp decoding:
yagl.zip
(379.33 KiB) Downloaded 114 times

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 05 Apr 2020 18:25
by UnicycleBloke
Work continues - slowly - on the documentation. With Wallyweb's help, I found a little bug which prevented non-vehicle GRFs from working (a slight misreading of the spec on my part). This was simple to fix and is included in the attached Windows build.

Just for fun, I tested this build by making a small modification to Dutch Trainset 2: https://www.tt-wiki.net/wiki/Dutch_Trainset_2. This has been my default test bed all along.

First I decoded the GRF into YAGL and a sprite sheet:

Code: Select all

yagl.exe -d dutchtrains.grf 
Then I edited the name of the "NS 5500 (Steam)" train a little in the file sprites/dutchtrains.yagl:

Code: Select all

// Record #988
strings<Trains, default, 0x013A> // <feature, language, first_id> Action04, Default
{
    /* 0x013A */ "{black}NS 5500 (Steam {green}locomotive {red}(added in YAGL){black})"; 
}
// Record #989
strings<Trains, de_DE, 0x013A> // <feature, language, first_id> Action04, German
{
    /* 0x013A */ "NS 5500 (Dampf)";
}
Then I re-encoded the GRF:

Code: Select all

yagl.exe -e dutchtrains.grf 
Using the revised GRF in game, I got this:
trains.png
trains.png (20.57 KiB) Viewed 5694 times
Not that I'm recommending changing the works of others. Just trying to show how it could work with something you've already written.

As ever, please give it a try and let me have any feedback. Same for the fledgling documentation.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 05 Apr 2020 19:33
by andythenorth
Continuing to watch this with interest :)

It's not something I can use, I've too much work tied up with nml, but it's nice to see a new toolchain develop.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 05 Apr 2020 19:54
by kamnet
I got a bunch of errors trying to run the program.

The code execution cannot be processed because MSVCP140_CODECVT_IDS.dll was not found.
The code execution cannot proceed because MSVCP140.dll was not found.
The code execution cannot proceed because VCRUNTIME140_1.dll was not found.
the code execution cannot proceed because VCRUNTIME140.dll was not found.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 10:18
by UnicycleBloke
kamnet wrote: 05 Apr 2020 19:54 I got a bunch of errors trying to run the program.
Bugger! These are Visual Studio re-distributable files. I was pretty sure that using a static build would eliminate this sort of thing.

This page appears to contain what you need: https://support.microsoft.com/en-gb/hel ... -downloads. The software was built with Visual Studio 2019.

Please let me know if installing the relevant package fixes the issue, and I'll update the documentation.


Al

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 11:22
by UnicycleBloke
Oops. Further investigation showed that I had not set up the static build options as I thought. The attached image only depends on kernel32.dll, which is part of Windows.
yagl.zip
(575.69 KiB) Downloaded 146 times
Sorry for the confusion.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 13:47
by UnicycleBloke
andythenorth wrote: 05 Apr 2020 19:33 It's not something I can use, I've too much work tied up with nml, but it's nice to see a new toolchain develop.
Sure. But I hope you'll give it a spin. My original goal was to look inside the GRFs generated with NML. I thought it would be interesting to see how the NML constructs are mapped onto the underlying NewGRF specs used for NFO. I wondered if it might help with debugging or something.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 15:38
by kamnet
UnicycleBloke wrote: 06 Apr 2020 13:47 Sure. But I hope you'll give it a spin. My original goal was to look inside the GRFs generated with NML. I thought it would be interesting to see how the NML constructs are mapped onto the underlying NewGRF specs used for NFO. I wondered if it might help with debugging or something.
I'm using it to help demystify uncommented NFO source code, especially when the only thing that exists is the compiled GRF. There's a lot of NewGRFs whose authors, experience and documentation have disappeared, leaving behind only the finished product.

Right now I'm working on Zepherys' Ekranoplan file. I hacked on it a few years ago and was able to introduce some new variables and make some minor adjustments, but much of the code was just unreadable for me. Having it decoded into much more human-readable YAGL, I've been able to make some significant improvements. Hopefully within the next few days I will be able to provide a better set with some commented code.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 17:22
by UnicycleBloke
kamnet wrote: 06 Apr 2020 15:38 Having it decoded into much more human-readable YAGL, I've been able to make some significant improvements.
You just made my day! Let me know if there is anything I can help with or add to the software to make life easier.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 20:19
by kamnet
Right now I'm getting stuck trying to increase the cost of purchasing a vehicle. I discovered the base cost property, and at first increasing it worked, and then it apparently stopped working, so I'm not sure what the issue is.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 06 Apr 2020 23:12
by wallyweb
kamnet wrote: 06 Apr 2020 20:19 Right now I'm getting stuck trying to increase the cost of purchasing a vehicle. I discovered the base cost property, and at first increasing it worked, and then it apparently stopped working, so I'm not sure what the issue is.
You might need to consult MacBeth's three witches who noted toil and trouble.
The cost of each vehicle type can be adjusted in two places:
Action 0 for global settings
Action 0 for the vehicle type

Code: Select all

08 	Global Settings - Cost base multipliers (08)
00 	Trains - Cost factor (17)
01 	Road Vehicles - Cost factor (11)
02 	Ships - Cost factor (0A)
03 	Aircraft - Cost factor (0B)
and then we encounter the esoteric and all pervasive rate of inflation.

Re: YAGL - Yet Another GRF Language (maybe)

Posted: 07 Apr 2020 07:38
by kamnet
Here's what I have so far:

Code: Select all

properties<GlobalSettings, 0x0000> // Action00
{
    // instance_id: 0x0000
    {
        cost_base_multipliers: 0xFF;
    }
}

Code: Select all

properties<Ships, 0x0000> // Action00
{
    // instance_id: 0x0000
    {
        introduction_date: 0x0000;
        reliability_decay_speed: 32;
        vehicle_life_years: 30;
        model_life_years: 40;
        climate_availability: Temperate | Arctic | Tropical | Toyland;
        sprite_id: 0xFF;
        is_refittable: true;
        cost_factor: 0xFF;
        speed_2_kmh: 0xFF;
        cargo_type: 0xFF;
        cargo_capacity: 0x0F28;
        sound_effect_type: 0x07;
        refit_cargo_types: 0x00000000;
        callback_flags_mask: 0x00;
        refit_cost: 0x32;
        ocean_speed_fraction: 0x00;
        canal_speed_fraction: 0x5B;
        miscellaneous_flags: 0x02;
        refittable_cargo_classes: 0x01D1;
        non_refittable_cargo_classes: 0x0000;
    }
}
This should result in a ridiculously high $6.3 billion for cost. But no matter what I set base cost to, it doesn't affect cost factor. When I adjust cost factor between 00 and FF, it ranges appropriately from $0 to $97k.