SquirrelGRF: Squirrel scripting for GRF-like files

[DaleStan]

I think the NFO way of doing this makes sense here

Except that that isn't the NFO way of doing it. Every time a sprite gets drawn, the sprite-selection "callback"[0] is called. Every single time. Which means your bug simply can't happen -- moving involves drawing the sprite in the new location, so the VM asks (again) what sprite should be used. And, since the vehicle has been moved, it is no longer at the end, so the GRF won't tell the VM to draw the end-of-consist sprite.

[0] In usual programming terms, it is a callback. In NFO terms it is not.

EDIT: Also, I think it's interesting how many times this has been said, and by how many different (not-me) people, in the last three pages. Each quote comes from a separate post. Two people are quoted twice, everyone else is quoted once.

Besides, I'd rather invest in a seperate program that converts Squirrel to NFO than using Squirrel directly in OpenTTD.

I think a program that converts Squirrel to NFO is more useful

A program to translate Squirrel (or any other language, preexisting or not) to NFO would be better.

External something->NFO converter is IMO the way to go

A compilator of >something< to NFO would make more sense.

A tool that converts "something" into NFO is the way to go.

I agree that a something-to-NFO converter would be the best option

the only way this project could survive is as a standalone project that can output NFO (or GRF, but why reinvent the wheel?) files.

If replacing hex numbers with text is the goal, doing a to-NFO compiler is much wiser.

One post in three, not counting mine on either side.


Finally:

The biggest issue here is the frequently-quoted observation from Patchman -- Why not NFO?

From my point of view, that question amounts to: Who needs a C compiler when we have hex editors?

The ELF specs are there, the processor ABI and specs are there, the library APIs are there; so who needs a toolchain?

Ignoring for the moment that you aren't writing a toolchain, NFO is portable. ELF is not.

[michael blunck]

I think it's interesting how many times this has been said, and by how many different (not-me) people [...]

Obviously, everybody understanding .nfo and/or building compilers, are reserved, to say the least. But all those who don´t understand anything about .nfo and/or compiler building are enthusiastic - as always.

Just one example:

This patch is interesting because It could add some new features on OTTD.
One new feature could be coding fixed EMUs and DMUs.
[emph. mb]

This is possible since years - even in OTTD. See e.g., Example4: using Callbacks 10, 16, 1D and 23 which shows the coding of a 4-part EMU (in .nfo).

If replacing hex numbers with text is the goal, doing a to-NFO compiler is much wiser.

Even something like M4 would be able to do this, as has been demonstrated already years ago. No compiler needed.


At last, let me add my final contribution:

A(n external) compiler to .nfo would be the way to go. If at all.

EOD
regards
Michael

[Korenn]

One post in three, not counting mine on either side.

That's nice, but seeing as we've already had the discussion that saying it is useless as long as nobody actually does it (which was your own argument!), there's no reason to stop cirdan from taking a different approach and seeing how far it can go.

So far your predictions on why it can't work have all been proven wrong, so I don't see why it'll fail anytime soon. cirdan, keep at it!

[Expresso]

The performance problem might be solved by having openttd actually interpret as much as possible and converting it to internal structures. This should be easy for creating vehicles (construction/replacing), but more difficult for animations and production callbacks (there where it matters). You're going to need some native code which can do the same as production and animation callbacks, from there on it seems pretty clear sailing... as long as you keep performance issues to a minimum.

Since vehicle construction is quite dependent on user input, I don't see any problems with scripting that (translation: no profiling needed for that).

Lets just say we want to construct an ICE3 at full length:
- Because its looks are pretty constant, you don't need to convert all of the script to an internal representation.
- On each component of the train you just set the appropriate values (tractive effort, hp, speed, sprite, cargo and capacity).
- If the train goes beyond a certain length (16 parts in case of that ICE).
Now, the ICE3 gets an animation (opening and closing doors at a station):
- the container for the sprite (in the openttd engine) should contain the information which sprites belong to what action (running, opening/closing, loading, unloading, loading stage) how many animation frames there are and what type of animation it is (ping-pong, random, unique random, loop, backwards loop).
-- This means that the sprite manager should keep track of which sprites are animated and how many frames there are... but this is implemented in openttd, isn't it?
- The only thing which the vehicle needs to know while being displayed is what it is doing and at which frame it is - quite easy to implement.
Sound:

In the above mentioned situations the information only needs to be delivered ONCE, if an interpreter is needed for things like these (be it NFO or squirrel), I suggest to fix that as it provides a gigantic opportunity to increase performance.

[cirdan]

In its current state, SquirrelGRF looks mainly like a straight translation of NFO, only with fancy text strings instead of hex numbers. And I can't imagine the benefits of adding an essentially equal second method of doing the same as the NewGRF-system getting even close to outweight the drawbacks.

The problem here is the way OpenTTD organises its data. Spritesets, spritegroups.. you name it, everything was designed to hold data as defined in a NFO file. Doing things differently would require major code restructuring. It's not impossible, but it's difficult in a week and a half.

And, regarding outweighing the drawbacks, this is how you prevent passenger carriages from being attached to a consist in SquirrelGRF:

Code: Select all

function attach (wagon)
{
        return (GetCargoType(wagon) == 0) ? 0xFD : 0xFF;
}

SetRailVehicleStats (9, { wagonattach = attach });

Do you think that its NFO counterpart is as legible as this?

...

Thank you for your ideas. I'm afraid, however, that some of them may be too advanced for the current state of the patch.

Finally:

The biggest issue here is the frequently-quoted observation from Patchman -- Why not NFO?

From my point of view, that question amounts to: Who needs a C compiler when we have hex editors?

The ELF specs are there, the processor ABI and specs are there, the library APIs are there; so who needs a toolchain?

Ignoring for the moment that you aren't writing a toolchain, NFO is portable. ELF is not.

Oh, I see, it's only a matter of portability. In fact, I now realise that C is chosen over manually writing machine code because of its portability alone. Ease of use has nothing to do with it.

And it was an example. It was there to hint an intelligent person in the right direction, not to be taken literally.

EDIT: Also, I think it's interesting how many times this has been said, and by how many different (not-me) people, in the last three pages. Each quote comes from a separate post. Two people are quoted twice, everyone else is quoted once.
...

Please don't quote me out of context. What I said, and I stand by my words, is that

I agree that a something-to-NFO converter would be the best option, but I disagree with you in that it only has advantages. In fact, I see one big disadvantage: no implementation and no working plan. While this does not, by itself, make it impossible, I do consider it far more difficult than what I'm doing; and the amount of thought that the idea has been given on this forum, without any visible results, reassures me on this idea. Of course, I encourage anyone willing to prove me wrong to do so.

On second thoughts, I now realise that I was wrong. The best option is actually a device that reads your mind and writes the NFO file for you. That makes SquirrelGRF the third best option in my list (but still the only one with an implementation).

I like to speak through my code. I had an idea that (in my opinion) could benefit the community, so I wrote a patch for it and posted it for public review. I'm fine if it's disliked or nobody finds it interesting; I've had far more important patches torn apart to pieces in other projects. What bothers me from the discussion in this thread is that many people are opposing the idea just because they can think of a better option which no one has seen working. Sometimes the best is the enemy of the better. As i said, I like to speak through my code; I wish that those supporting the "best option" did the same.

That's nice, but seeing as we've already had the discussion that saying it is useless as long as nobody actually does it (which was your own argument!), there's no reason to stop cirdan from taking a different approach and seeing how far it can go.

I'm glad to see that there are still some people who get the point.

So far your predictions on why it can't work have all been proven wrong, so I don't see why it'll fail anytime soon. cirdan, keep at it!

It's funny indeed to see how some people's positions have changed from an initial "you're a moron and what you're trying to do can't be done" to "even if you've done it we still don't want it". And remember Maier's Law: If the facts do not conform to the theory, they must be disposed of.

----------

The only new feature in today's patch is the implementation of the wagon-attach callback. Not much, but it required understanding large parts of the code. Also, NFO uses the same codepath to determine the sprite to be used for a vehicle and to obtain a callback value. Since this is a mess, I had to design something new. (Technical part: The spritegroup defined in an Action 3 is really a multiplexing function that has to decide the real reason why it was invoked. But why do that when you can have multiple entry points?)

The attached sample file won't let you attach passenger carriages to your Ginzu A4s.

[maquinista]

Very good.

Also, I think that some constants could help:

Code: Select all

0xFD : 0xFF
// or
WAGON_DISALLOW : WAGON_ALLOW

[Ralph]

I think the NFO way of doing this makes sense here

Except that that isn't the NFO way of doing it. Every time a sprite gets drawn, the sprite-selection "callback"[0] is called. Every single time. Which means your bug simply can't happen -- moving involves drawing the sprite in the new location, so the VM asks (again) what sprite should be used. And, since the vehicle has been moved, it is no longer at the end, so the GRF won't tell the VM to draw the end-of-consist sprite.

The point I was trying to make is that using a callback (1D in NFO) when the user assembles their trains is simpler than trying to encode complex behaviour on how the vehicles can be used when first defining the vehicles.

As to the example I assumed that once a vehicle type or individual vehicle has been assigned a sprite it will continue to use that until told otherwise. (I do not think that cirdan has yet implemented a way to update the sprite of an individual wagon in the game the way I used the function in the example).

[SirXavius]

I like to speak through my code. I had an idea that (in my opinion) could benefit the community, so I wrote a patch for it and posted it for public review. I'm fine if it's disliked or nobody finds it interesting; I've had far more important patches torn apart to pieces in other projects. What bothers me from the discussion in this thread is that many people are opposing the idea just because they can think of a better option which no one has seen working. Sometimes the best is the enemy of the better. As i said, I like to speak through my code; I wish that those supporting the "best option" did the same.

Cirdan, you're obviously intelligent and very confident in your abilities and the direction you want to take with this thing. I have been watching this thread and your progress from afar hoping you get a fully functional package, because i'm about to sit down in the next couple of weeks and see if i can learn nfo coding. I shudder at the idea of undertaking a project and making myself feel stupid because i can't understand a simple language, or simple instructions to a complex language. I am all for anyone making my own work easier for me.

From one colleague to another, i would ask that you not let these some of these guys deter or discourage you. You can tell which ones are trying to be helpful, and which ones can't help but be negative. Plus, not just the users and coders we have now, but there is a future cadre of new OTTD players, some of them computer-literate or artistic, who will find this old game and like the fact that you can customize many facets of the game, and will need a tool to make it much easier than what we have now, to do just that. I've seen many projects fail because of discouragements (as opposed to brainstorming ways around barriers both real and perceived), and other projects succeed only because of encouragements and cheers. So keep up the hard work, and remember you have many folks waiting to admire you for your persistence and your success. (No pressure! )

[DaleStan]

Ignoring for the moment that you aren't writing a toolchain,

Oh, I see, it's only a matter of portability. In fact, I now realise that C is chosen over manually writing machine code because of its portability alone. Ease of use has nothing to do with it.

No, it's a matter of "You're not writing a toolchain. Nor are you writing part of a toolchain." If you were, I wouldn't be complaining.

Sometimes the best is the enemy of the better.

And this is one such place. Have you ever seen the merely good go into OpenTTD?

So far your predictions on why it can't work have all been proven wrong, so I don't see why it'll fail anytime soon. cirdan, keep at it!

Um... No. I specified four things. One has been touched, but not completed. The other three have remained completely untouched. And I have several more for when those are completed. (Action F comes to mind, as do callbacks 1E and 2D.)

It's funny indeed to see how some people's positions have changed from an initial "you're a moron and what you're trying to do can't be done" to "even if you've done it we still don't want it".

When did I ever say that I wanted this? My position has always been that I don't want this. But if I had started out with that argument, would that have been more likely to dissuade you?

[Michi_cc]

And, regarding outweighing the drawbacks, this is how you prevent passenger carriages from being attached to a consist in SquirrelGRF:

Code: Select all

function attach (wagon)
{
        return (GetCargoType(wagon) == 0) ? 0xFD : 0xFF;
}

SetRailVehicleStats (9, { wagonattach = attach });

Do you think that its NFO counterpart is as legible as this?

No, of course not. But that's not the point, a to-NFO compiler can do legible as well. The point is that something that adds lines over lines without any technical benefit has a chance near zero to ever make it into trunk.

-- Michael Lutz

[maquinista]

And, regarding outweighing the drawbacks, this is how you prevent passenger carriages from being attached to a consist in SquirrelGRF:

Code: Select all

function attach (wagon)
{
        return (GetCargoType(wagon) == 0) ? 0xFD : 0xFF;
}

SetRailVehicleStats (9, { wagonattach = attach });

Do you think that its NFO counterpart is as legible as this?

No, of course not. But that's not the point, a to-NFO compiler can do legible as well. The point is that something that adds lines over lines without any technical benefit has a chance near zero to ever make it into trunk.

-- Michael Lutz

I see lots of technical benefits on squirrel. NFO is difficult. In Spanish set I only can copy the code of other trains and adapt it carefully. Also, I can't add new features, only new trains or translations. Squirrel could make it much easier.

[Expresso]

<steam venting>
Something to actually make life easier on the modders comes along, and since it competes with what is currently used to mod openttd (and actually has the potential at becoming better and more usable) it seems to be oppressed immediately by some people.

Somebody is trying to help out with improving openttd, but that is not seen by some people. Instead, those people panic and start attacking the help. This is just sickening.
</steam venting>

Cirdan has mentioned that squirrelgrf wouldn't replace NFO (at least for the time being). As a toolchain it's not an option, because you still get to deal with all the issues and limitations of newgrf.

If you want to give feedback on something, why not give positive feedback instead of negative feedback? Although it's easier to be negative, it'll be more rewarding to be positive.

[CommanderZ]

But michi_cc is so correct.

I'm not criticizing the implementation, I'm criticizing the whole idea of building a new modding "API " pararrel to NFO. Devs already said they don't want it, so it won't be trunked. If it won't be trunked, nobody will make these SquirrelGRFs. If nobody supports it, it won't give devs a reason to even think about it.

[Darkvater]

I guess I missed a big part of the conversation but why not create this SquirrelGRF as an external tool that writes NFO code from SquirrelGRF? It can then readily be used in OpenTTD without any modifications and leave raw GRF code for the parts that are as of yet unsupported.

A bit like GRFMaker but more powerful (I surmise).

[Hyronymus]

I guess I missed a big part of the conversation but why not create this SquirrelGRF as an external tool that writes NFO code from SquirrelGRF? It can then readily be used in OpenTTD without any modifications and leave raw GRF code for the parts that are as of yet unsupported.

A bit like GRFMaker but more powerful (I surmise).

Good idea, especially if it bypasses the shortcommings of GRmaker.

[Michi_cc]

If you want to give feedback on something, why not give positive feedback instead of negative feedback?

cirdan explicitly asked about dev opinions, I'm giving him some.

To be clear, I have absolutely nothing against somebody writing such a patch, but it has to be clear that with the current design trunk inclusion is probably very unlikely. If that doesn't bother you, sure, go ahead.

-- Michael Lutz

[NukeBuster]

I guess I missed a big part of the conversation but why not create this SquirrelGRF as an external tool that writes NFO code from SquirrelGRF? It can then readily be used in OpenTTD without any modifications and leave raw GRF code for the parts that are as of yet unsupported.

A bit like GRFMaker but more powerful (I surmise).

Because the goals of squirrelGRF lay beyond the current NewGRF implementation.

[maquinista]

But michi_cc is so correct.

I'm not criticizing the implementation, I'm criticizing the whole idea of building a new modding "API " pararrel to NFO. Devs already said they don't want it, so it won't be trunked. If it won't be trunked, nobody will make these SquirrelGRFs. If nobody supports it, it won't give devs a reason to even think about it.

But We can patch the trunk with the Squirrel API and create a mod like MiniIn. Also, NOF is supported by very few people.

Also, I will try code the Spanish Narrow gauge trains with squirrel (not now, I would do It when squirrel API was more completed).

[AndyLandy]

NFO exposes the ability to specify an essentially infinite number of graphics/sounds depending on any zero or more of:
1) Game date
2) Vehicle age
3) Vehicle speed
4) Time since last service
5) Vehicle purchase date
6) Date of last service
7) (Trains only) Presence/absence of certain types of wagons in certain positions.
(Trains only) Count of wagons, of any number of specified types.
And dozens of other variables.

For example, I can choose a different breakdown sound if the engine is not pulling a caboose. I can select a lower maximum speed if the last vehicle is not one of: {caboose, passenger car, mail van}. I can choose different graphics for a vehicle built (or last serviced) before/after 1 Jan 1980, but only for @CARGOS -- the others always use a third set of graphics. And I can make the built/last serviced choice the user's. Can you invent an API that's powerful enough to expose all those choices?

Very probably. In fact, I suspect it would be possible to create a completely callback-free description language that would allow access to all these features. Such a language would be more like a config file than a programming language, but shouldn't need to lose any of the complexity. Whether it's possible to write something like this can be converted to NFO, or whether you'd need to change the inner workings of (Open)TTD(Patch) I'm not sure.

You could also do away with the base-value/multiplier way of specifying running costs. Simply specify an initial running cost and OpenTTD can work out

Why does OpenTTD have to do this? An NFO compiler could do it just as well, and this avoids breaking with the current codebase wherein every cost in TTD, Patch, and Open is base/multiplier.

I guess the question here is "If the NFO compiler could do this, why doesn't it? Why laden the user with this, when the computer could do it for them.

You could also get away from the "plane speeds are an 8-bit number, multiplied by 8" thing, and just specify a top speed in MPH and be done with.

Again, why burden OpenTTD with this when any other program could do it just as statically and just as well?

The honest answer is "I don't care whether it's OpenTTD, GRFCODEC or whatever that does this, just as long as it's not me doing it manually" -- I'm a firm believer in making the computer do as much of the work as I can.

[cirdan]

...

Thanks for your ideas. Of course, having OpenTTD parse the Squirrel (text!) file into internal structures is the first thing to do and, if I'm not mistaken, that's precisely what sq_compile() does: turning the text file into some sort of compiled bytecode (someone more knowledgeable about Squirrel please correct me if I'm wrong). Once converted to bytecode, an activation function is called when the game is started, and after that the Squirrel code is only used for callbacks (Squirrel functions registered to be called for certain events), and it is essentially up to the Squirrel virtual machine to run them, however fast it does so. In a sense, NFO is not too different from compiled bytecode, so, once compiled, there shouldn't be much difference between NFO and Squirrel. Of course, NFO is bytecode tailored to the task at hand, so it should be faster, but I hope that the slowdown in Squirrel is not noticeable or at least acceptable.

Also, I think that some constants could help:

Thanks. The constants will be available in the next version of the patch.

No, of course not. But that's not the point, a to-NFO compiler can do legible as well. The point is that something that adds lines over lines without any technical benefit has a chance near zero to ever make it into trunk.

Doesn't making GRF coders' lives easier count as a technical benefit? Some of them have been asking for something like this.

Also, I'm trying to keep a balance between adding new features to the patch and keeping it maintainable, which means touching as little as possible from the rest of the code. For me, it's not as easy as it seems; for instance, every time I go through the sprite resolver code I feel the urge to rewrite it, because I think it is a mess. It also means that I won't add revolutionary features that need major code engineering.

Back on the issue of the benefits, what kind of technical benefits would you expect from this patch?

While at it, what makes a patch appear on the devs' radar?

I guess I missed a big part of the conversation but why not create this SquirrelGRF as an external tool that writes NFO code from SquirrelGRF? It can then readily be used in OpenTTD without any modifications and leave raw GRF code for the parts that are as of yet unsupported.

In my humble opinion as an experienced programmer, because that would be reinventing the wheel (a language-to-bytecode compiler, when we already have several generic ones available). And because that would reinforce the limitations of NFO code (such as the multiplexing thing I mentioned earlier). Again, in my humble opinion, NFO is not the language of the future.

cirdan explicitly asked about dev opinions, I'm giving him some.

And I thank you for doing so; this wouldn't have been the first time that I'm met with silence after asking for opinions.

To be clear, I have absolutely nothing against somebody writing such a patch, but it has to be clear that with the current design trunk inclusion is probably very unlikely. If that doesn't bother you, sure, go ahead.

I take note of this. Certainly, this is not a patch designed to be maintained out-of-tree; its usefulness as such is limited, at best, and the burden of keeping it in sync is likely to be considerable.

Which design, if any, would make trunk inclusion more likely?