SquirrelGRF: Squirrel scripting for GRF-like files

[cirdan]

As we all know, there are a number of ideas and suggestions that keep appearing on a regular basis in the forums regarding OpenTTD. One of them, whose last respawn was a few days ago, is the request to have an easier way of coding NFO, and it goes, more or less, as follows: Coding NFO is cumbersome and has a steep learning curve, since it involves reading the bare specs and typing hex codes by hand; why don't we make a compiler/interpreter/whatever that inputs a higher-level description and outputs the corresponding GRF?

We have already heard many reasons why some consider this to be unfeasible; I will not repeat them here. Whether one agrees with them or not, the fact remains that nobody has stepped out and written such a program, not even started to work on it. Therefore, I think it is fair to assume that, even if the task could be theoretically possible, it poses practical difficulties that so far have stopped anyone from undertaking it.

So, what I am hereby proposing is that we devise, instead of some kind of program that helps us write NFO, a new extension system in OpenTTD to achieve the same goals that a GRF has (change or add vehicles, stations, buildings, etc.) only that in a more human-friendly fashion. The idea is not to drop GRF support--we want it to make use of the many excellent GRFs already available--but to add a new way in which such GRFs can be created from now on, one that is easier and does not scare newcomers off. This would mean that people wishing to make GRFs for OpenTTD would need to spend less time coding, and particularly much less in the early stages of learning.

While this idea had been lurking in my head for quite some time, it was the recent addition of Squirrel to OpenTTD (for NoAI) what finally set me off to writing an implementation, which I am attaching as a patch against r15084. The patch gives OpenTTD the ability to read a new kind of GRF file, a "SquirrelGRF" file, which is nothing more than a standard Squirrel text file that uses a set of predefined functions to tell OpenTTD what to do or to change, similarly to actions in a NewGRF file. For example, the NewGRF file whose NFO coding is

Code: Select all

0 *  4  02 00 00 00
1 * 23  08 06 10 32 54 76 "Name" 00 "Description" 00
2 *  8  00 00 01 01 00 09 c2 01

would be coded in SquirrelGRF as

Code: Select all

GRFInfo (0x76543210, "Name", "Description");
RailVehicleChangeSpeed (0, 450);

Note that the attached patch is only a proof of concept, and is rather hackish. It should be seen as a quick-and-dirty implementation of the idea, to show that it can actually be done, but so far the only functions implemented are precisely GRFInfo() and RailVehicleChangeSpeed(), so you won't be able to code complex GRFs with it. The attached sample.nut file contains the above code in a working state though, so loading it will give your Kirby Paul Tanks a noticeable speed boost (note: rename sample.txt to sample.nut; the forums do not allow a .nut extension in attached files).

Any comments or suggestions about the idea or the patch are welcome.

[Conditional Zenith]

Well the first thing that comes to mind is that GRFs written this way will not work with TTDPatch. The capability to run the old "newGRFs" should of course remain, say you say. Other than that, I know little about GRFs so I can't say much.

[ThunderAI]

Well the first thing that comes to mind is that GRFs written this way will not work with TTDPatch. The capability to run the old "newGRFs" should of course remain, say you say. Other than that, I know little about GRFs so I can't say much.

I dont see why it needs to remain at all. It is called advancement and if TTDPatch wants to load the GRFs then they should add the ability in.

[Gremnon]

The newgrf_ports branch uses a specialised version of grfs to add airports in - does that mean it should be backported to TTDP?
If someone wants to do the same, but for Squirrel scripts instead of airports, by the same logic... I see no reason TTDP should have to support that too. It would simply disable it because it's not supported, whereas in OTTD - if it were implemented - it would work perfectly fine.

Food for thought.

[FooBar]

The capability to run the old "newGRFs" should of course remain, say you say.

I dont see why it needs to remain at all.

Are you volunteering to rewrite all existing grfs?


I don't really see that this SquirrelGRF makes things much easier. If I compare

Code: Select all

RailVehicleChangeSpeed (0, 450);

To the NFO equivalent:

Code: Select all

-1 * 0 00 00 01 01 00 09 \w450

then sure, the squirrel version looks more easy, but if I'd like to change speed, power, traction type and weight, I'd much rather write

Code: Select all

-1 * 0 00 00 04 01 00 09 \w450 0B \w100 19 08 16 \b30

than

Code: Select all

RailVehicleChangeSpeed (0, 450);
RailVehicleChangePower (0, 100);
RailVehicleChangeTractiontype (0, 2);
RailVehicleChangeWeight (0, 30);

If you'd comment the NFO version properly, it's just as easy to read as any other code.

Besides, I'd rather invest in a seperate program that converts Squirrel to NFO than using Squirrel directly in OpenTTD.

Just my €0,02...

[cirdan]

Are you volunteering to rewrite all existing grfs?

No one is asking to drop NewGRF support from OpenTTD; in fact, I explicitly advised against that in my post. Existing GRFs should continue to work in OpenTTD.

I don't really see that this SquirrelGRF makes things much easier. If I compare

Code: Select all

RailVehicleChangeSpeed (0, 450);

To the NFO equivalent:

Code: Select all

-1 * 0 00 00 01 01 00 09 \w450

then sure, the squirrel version looks more easy, but if I'd like to change speed, power, traction type and weight, I'd much rather write

Code: Select all

-1 * 0 00 00 04 01 00 09 \w450 0B \w100 19 08 16 \b30

than

Code: Select all

RailVehicleChangeSpeed (0, 450);
RailVehicleChangePower (0, 100);
RailVehicleChangeTractiontype (0, 2);
RailVehicleChangeWeight (0, 30);

If you'd comment the NFO version properly, it's just as easy to read as any other code.

Again, no one will force you to use the new scheme. You are free to continue writing NFO and NewGRF.

Also, where did you get the documentation for those RailVehicle* functions? Wherever it was, I'm sure it also mentions

Code: Select all

RailVehicleChangeStats (0, 450, 100, 2, 30, ...);

Besides, I'd rather invest in a seperate program that converts Squirrel to NFO than using Squirrel directly in OpenTTD.

Do not let any of us stop you. In the meanwhile, the other option is already available.

[FooBar]

Don't take my reply too personal cirdan...

No one is asking to drop NewGRF support from OpenTTD; in fact, I explicitly advised against that in my post. Existing GRFs should continue to work in OpenTTD.

I'm aware of your statement, but the way I read it, ThunderAI asks to drop NewGRF support. Read his post again and you'll see...

Again, no one will force you to use the new scheme. You are free to continue writing NFO and NewGRF.

That's why I think a program that converts Squirrel to NFO is more useful, as one could use it for TTDPatch as well. I think most NewGRF developers still attach value to TTDPatch support which would keep them from using your new system. While I have no idea of the technical side of your implementation, but if your current system translates the SquirrelGRF to NFO which then gets interpreted by the game, the step to a seperate application might not be that big...

Also, where did you get the documentation for those RailVehicle* functions? Wherever it was, I'm sure it also mentions

Nowhere. Since you stated that GRFInfo() and RailVehicleChangeSpeed() are the only implemented functios, I had to make the others up.

And RailVehicleChangeStats (0, 450, 100, 2, 30, ...);? Are you kidding? I thought your system was meant to make things more easy.. NFO at least gives a hint on what a certain value does, but this is just an array of numbers, which probably need to be in a specific order as well. Don't get me wrong here, I'm all in favour of a system that is easier to use and I most likely will use the most simple version (due to my laziness), but what I'm seeing now just scares off newcomers just as any other system we currently have.

Maybe I should say something useful as well...
The key to any programming or scripting language is documentation. As long as that is easy to understand, the language (how complicated it may be) is easy to use as well. The documentation is one thing that makes NFO difficult to use because it's hard to understand. Don't let that become a problem of your SquirrelGRF language!

[cirdan]

Don't take my reply too personal cirdan...

I didn't, really. But I must have some communication problem since this is not the first time someone thinks so...

No one is asking to drop NewGRF support from OpenTTD; in fact, I explicitly advised against that in my post. Existing GRFs should continue to work in OpenTTD.

I'm aware of your statement, but the way I read it, ThunderAI asks to drop NewGRF support. Read his post again and you'll see...

Hmmm, as I read it, he is saying that adding a feature to OpenTTD should not be avoided just because TTDPatch doesn't, or cannot, have it. Then again, perhaps my interpretation is wrong.

That's why I think a program that converts Squirrel to NFO is more useful, as one could use it for TTDPatch as well. I think most NewGRF developers still attach value to TTDPatch support which would keep them from using your new system. While I have no idea of the technical side of your implementation, but if your current system translates the SquirrelGRF to NFO which then gets interpreted by the game, the step to a seperate application might not be that big...

I agree with you: A program to translate Squirrel (or any other language, preexisting or not) to NFO would be better. But I consider that to be beyond my possibilities, so I settled for the second best thing.

And RailVehicleChangeStats (0, 450, 100, 2, 30, ...);? Are you kidding? I thought your system was meant to make things more easy.. NFO at least gives a hint on what a certain value does, but this is just an array of numbers, which probably need to be in a specific order as well. Don't get me wrong here, I'm all in favour of a system that is easier to use and I most likely will use the most simple version (due to my laziness), but what I'm seeing now just scares off newcomers just as any other system we currently have.

Of course, the API is not settled yet. I'm not trying to push the idea as it is; I'm open to suggestions and improvements. Regarding this particular matter, a solution could be

Code: Select all

RailVehicleChangeStats (0, RR_SPEED, 450, RR_POWER, 100, RR_TRACTION, 2, RR_WEIGHT, 30, RR_END);

My intention isn't to change things that work, and that particular way of doing things in NFO (prop/value pairs) can be useful.

To sum up: I'm not trying to sell you this new-thing-that-will-solve-all-of-our-problems. It's just a new idea to discuss, to decide whether we would be better off with it (or something similar) or not. I didn't provide a patch to have it set in stone, but just because there are people around here who like to bash new ideas, good or not, and "code it yourself" is one of their favourite arguments.

[FooBar]

But I must have some communication problem since this is not the first time someone thinks so...

Might be me as well. English is not my first language (it comes only third) and as a result I sometimes have difficulties expressing myself or interpreting others. To me it looked like you were "not amused" with my reply and that's why I wrote not to take it too personal...

Anyways...

Hmmm, as I read it, he is saying that adding a feature to OpenTTD should not be avoided just because TTDPatch doesn't, or cannot, have it.

To that I completely agree. Even currently OpenTTD has some things Patch doesn't have and TTDPatch has some things Open doesn't have. No big deal.
It's just that I interpreted ThunderAI's reply as follows:
ThunderAI wrote "I dont see why it needs to remain at all". Now "it" can refer to anything, but since he used the word "remain" and Conditional Zenith wrote "The capability to run the old "newGRFs" should of course remain", I thought Thunder was referring to that. And thus I thought that Thunder was volunteering to convert all existing newgrfs because he otherwise wouldn't wouldn't have written what I think he was writing.

Oh boy, this gets more difficult every word. Didn't I say I had problems expressing myself?

A program to translate Squirrel (or any other language, preexisting or not) to NFO would be better. But I consider that to be beyond my possibilities, so I settled for the second best thing.

Ok, so I understand that your system changes the gamestate directly without using NFO as an intermediate step. Well, since your system is included in OpenTTD directly, I think that's the better option anyway as it reduces the amount of calculations needed.

RailVehicleChangeStats (0, RR_SPEED, 450, RR_POWER, 100, RR_TRACTION, 2, RR_WEIGHT, 30, RR_END);

Ah, that makes more sense already. Can I suggest something then? I don't see the use of the "RR_" prefixes, because the function already has "Rail" in it's name.
From a user point-of-view, the following might be even easier to understand, but I don't know if that's possible within the Squirrel language:

Code: Select all

RailVehicleChangeStats (ID:0, SPEED:450, POWER:100, TRACTION:2, WEIGHT:30)

But since the vehicle ID isn't really a property like the other values, something like the following might be better, where the second parameter of the first function is optimal and refers to the number of consecutive IDs to change (defaults to 1):

Code: Select all

RailVehicleModifyID (0 [, 1])
RailVehicleChangeStats (SPEED:450, POWER:100, TRACTION:2, WEIGHT:30)

So one tells the system which vehicle needs to be changed and then all following code is applied to that vehicle, so this could be valid as well:

Code: Select all

RailVehicleModifyID (0 [, 1])
RailVehicleChangeStats (SPEED:450, POWER:100)
RailVehicleChangeStats (TRACTION:2, WEIGHT:30)

I'm not trying to sell you this new-thing-that-will-solve-all-of-our-problems.

Don't worry. If you're asking money, I'm not buying

Like I said, I'm all in favour of a system that's more easy to use, so I'm just trying to make sure that your system is indeed easier to use

[Yexo]

I'd be nice if this was extended so that at least a big part of the current grf language was implemented.

@FooBar:
The closest squirrel can come to your examples is this:

Code: Select all

local vehicle_ids = [0, 1, 4];
local new_stats = {speed = 450, power = 200};
RailVehicleChangeStats(vehicle_ids, new_stats);

Of course this is the same:

Code: Select all

RailVehicleChangeStats([1], {speed = 200});

if you just want to change a single vehicle.

[michael blunck]

Just do

s/squirrel/lua/

and use the search function.

HTH
regards
Michael

[Yexo]

Just do

s/squirrel/lua/

Why would one do that? OpenTTD already has squirrel support atm.

and use the search function.

HTH
regards
Michael

To find what? I haven't seen any start of an implementation before, only ideas.

[FooBar]

The closest squirrel can come to your examples is this:

Code: Select all

local vehicle_ids = [0, 1, 4];
local new_stats = {speed = 450, power = 200};
RailVehicleChangeStats(vehicle_ids, new_stats);

Of course this is the same:

Code: Select all

RailVehicleChangeStats([1], {speed = 200});

if you just want to change a single vehicle.

That's actually pretty close. I could live with that. Is that 'local' mandatory? I'm used to PHP myself and am therefore not used to declaring the scope of a variable.

Maybe I should read the NoAI specifications, as that (1) teaches me a bit of Squirrel and (2) teaches me a bit on how to write an AI myself...

[cirdan]

@FooBar:
I'm going to make you a confession. Up to the day before yesterday, I had absolutely no experience with Squirrel. So I've just followed the path of least resistance to a minimally working implementation.

I'd be nice if this was extended so that at least a big part of the current grf language was implemented.

@FooBar:
The closest squirrel can come to your examples is this:

Code: Select all

local vehicle_ids = [0, 1, 4];
local new_stats = {speed = 450, power = 200};
RailVehicleChangeStats(vehicle_ids, new_stats);

Of course this is the same:

Code: Select all

RailVehicleChangeStats([1], {speed = 200});

if you just want to change a single vehicle.

This seems like nice and simple syntax indeed. I wish I knew more about Squirrel...

Just do
s/squirrel/lua/

Interestingly enough, my initial plans were to implement "LuaGRF". But now Squirrel is already being used by OpenTTD, so I thought I would keep things simple.

On the other hand, I have never heard of an implementation of something similar, whether in Lua or other language. Is there one?

[michael blunck]

[again and again ..]

To find what? I haven't seen any start of an implementation before, only ideas.

To find out that all those arguments have been presented ad infinitum, whether for XML, LUA or Squirrel.

See e.g. http://www.tt-forums.net/viewtopic.php? ... 9&start=80, posts by Patchman, Rubidium, belugas, DaleStan, peter_1138 and eis_os.

regards
Michael

[cirdan]

See e.g. http://www.tt-forums.net/viewtopic.php? ... 9&start=80, posts by Patchman, Rubidium, belugas, DaleStan, peter_1138 and eis_os.

There are certainly quite a number of points raised in that thread; I'll try to give my opinion about them.

One of the complaints in that thread, expressed several times by different people, is that there is no working implementation. Well, that's precisely what I'm offering. It's not a complete implementation, but at last it shows that it can be done.

The other issue is efficiency. I must admit that I do not have profiling runs to show, so I cannot counter this argument. If callbacks are used intensely, this may very well be a serious problem, even if the code can be pre-compiled as bytecode. So, no, I'm not going to say that "efficiency is not important"; if having a text GRF slows down the game (other than at load time), that is not generally acceptable. On the other hand, as I said, I'm not advocating replacing NFO with this, but rather that they could live side by side, so GRFs that would be CPU hogs as text can still be coded in NFO.

The question is: Does making GRF coding easier, for at least some (not necessarily all) GRFs, warrant the addition of an interpreter in OpenTTD?

[Yexo]

The question is: Does making GRF coding easier, for at least some (not necessarily all) GRFs, warrant the addition of an interpreter in OpenTTD?

Seeing squirrel is already part of OpenTTD, I'd say yes.

[Rubidium]

When a compiler for <something> to NFO/GRF can't be made due to time constraints, you think that you'll finish adding an API as effective as NewGRF to OpenTTD?

[cirdan]

When a compiler for <something> to NFO/GRF can't be made due to time constraints, you think that you'll finish adding an API as effective as NewGRF to OpenTTD?

I honestly don't know. The API can be grown over time, much like (if I am not mistaken) NewGRF support in OpenTTD was added in steps. And, as the discussion is developing, it seems that for full NFO support the best option is already NFO, but perhaps a reduced API would be enough for a number of GRF coders.

(By the way, I didn't know that the obstacle to write a compiler for <something> to NFO was time constraints.)

[frosch]

The other issue is efficiency. I must admit that I do not have profiling runs to show, so I cannot counter this argument. If callbacks are used intensely, this may very well be a serious problem, even if the code can be pre-compiled as bytecode.

To give you some numbers I did for george to optimise his grfs:
http://www.math.tu-clausthal.de/~mtce/o ... 081222.txt
http://www.math.tu-clausthal.de/~mtce/o ... tailed.txt
Note that those are only industries, there are a lot more houses. And some vehicles callbacks are called even more often.

Also note that there are fundamental differences between newgrf callbacks and an imperative language.
- There are no global storage variables.
- I.e. houses have only one 8 bit storage, which can only be written in a few strictly defined situations.
- There are no API functions at all to get e.g. some random value. You can always only access a few variables, which might be restricted to reading/writing in various cases.
And to be honest, your example shows only how to realise some simple action0. And you can already find two languages which can deal with that in the TTDP graphics forum. Defining a callback is something completely different, especially as writing down which variables the callback shall use, what to compute with them, and what to return, is hardly the difficult part of newgrf.
I like the analogy of a microcontroller when talking about newgrfs: E.g. though microcontrollers can be programmed in C or in assembler or whatever, no language can help you to setup a timer or a PWM generator correctly. (Edit: Or as that might sound to easy. Imagine to setup a USB port on a descriptor level)

In summary it is quite wrong to think of newgrfs as of some interpreted language. It is more like some language that initializes peripheral elements of some microcontroller.