GRL (GRF Language)

[AndersI]

1) How do you conditionally skip the action 6?

Either you don't, or you add something like the C construct (cond?value:Parameter(2)) which will put a heavier load on the compiler.

1b) (Corollary) How do you specify the value that is used when the 6 isn't?

As above.

2) What if you want to modify something that is implied by the grl, not explicit?

You don't. As I said earlier, is it really necessary to produce a HLL that can decode all existing GRF/NFO files, or is it enough with a HLL that can be used to encode any future GRF:s?

[DaleStan]

It is enough to encode all future grfs, but if something has been done once, it'll probably be done again, so anything that has been done requires serious consideration; both whether or not it's necessary for GRL to support it and whether or not there a GRL-supported and functionally equivalent form.

[Alberth]

First of all thank you all for thinking along wih me. It really helps geting the design right faster.

Yesterday after writing the replies I played some Mars scenario, and thought a lot about Action6 and how to deal with it.

First item at the agenda is the decision whether or not to provide Action6 at user level. I think now that it should be available. The main reason is that it makes GRL more compatible with NFO, in particular when somebody invents yet another use for this primitive, besides the current GRF resource management. (Which will happen one day after I implemented GRF resource management if you believe in Murphys law.)
At the same time, I recognize that most people really don't want to play (or rather, shouldn't play ) with such dangerous toys, so at some point in the future, GRL should provide a nice primitive for doing GRF resource management.

The second problem is addressing of the fields. Fields at the outermost level are not a problem, but fields buried in some nested nodes, in particular lists of these things, are another matter. For example, If the next sprite is an Action6, and I want to change the 3rd offset, how do I express that in a "offset = ....."?
The answer was surprisingly simple and elegant, add a label!
Thus, fields at the outermost level in the next sprite can be accessed with something like "offset = feature" (for the 'feature' field), and fields buried in a (sub)node get an additional label prefix, like "offset = third:offset". Last but not least, I will throw in an additional offset so you can access eg the 2nd byte in the field (not sure whether it is ever needed, but it won't hurt).

The third item is the algorithm. It turns out that it is a three step process. A GRL primitive (action) first converts its fields and (sub)nodes into a flat (not nested) list of NFO fields. The latter are things with a numeric value and a natural length (ie the length one would get if no other requirements are set). Also included in the NFO fields are their labels (so the Action6 before it can find out which field the user wants), and some type information ('this field is a byte/word/double/extended byte'). Most primitives can decide this list of NFO fields on their own. The exception to this rule is an Action6 primitive, which needs the NFO field list of its successor. This means that the first step may cause recursive calls downwards in the list primitives to compute the NFO field lists. Since the list of GRL primitives is finite, the recursion is also finite. (In practice, I'd be surprised if it is ever longer than 3 or 4.)

Once the NFO list has been created for a primitive, it should be frozen. This means that all natural length are fixated to fixed sizes. Once this is done, the offset of all fields become known. (In other words, an Action6 needs a frozen NFO list of the next sprite!)

The third step of generating a sequence of bytes in NFO format from the NFO field list is quite trivial.

Why the difficult three steps? The reason is that a user may write an Action6 that modifies 2 places in the next sprite, first near the end of the sprite, and then near the start of the sprite. Since the 2nd modification may modify the size of the accessed field, the offset of the first modification may also change! As a result, an Action6 should first decide the sizes of the accessed fields (ie change the 'natural length's), then freeze the NFO list of the next sprite, and at that point it can get the offsets (and complete its own NFO field list with values and give it to its previous sprite if necessary).

I think this is a correct solution. The next sprite gives a desired size, and the Action6 can still change it in any order to whatever size it feels like, and still everything will be consistent, since once frozen, the NFO field list cannot be modified any more.

[Alberth]

You guys are much faster than me, I cannot keep up with your discussion (and starting tomorrow that will get much worse, since I have to go back to work). However, it is a great discussion, thanks all!!

This sounds like the right approach to me. Something like

Code: Select all

        id = Byte(Parameter(2)),

or

Code: Select all

        value = 0020+Word(Parameter(2)),

Now the Action6 is implicit instead of explicit. There are of course many other possible ways to use Action6, but who says that your GRL must be able to express existing GRF/NFO files? Isn't it enough if GRL can be used to create new GRF:s?

I like your syntax ideas although at this moment I have no idea what 'Parameter' exactly means since I haven't yet studied GRF parameters. I will keep this for later reference.

With respect to your question, in my previous post I said that I will give the Action6 primitive to the user precisely for reasons expressed by DaleStan. However, I also feel that I should provide 'easier' primitives that makes eg resource management more in reach for every-body. That is something for the future.

The more I am learning, the more I get the feeling that there are two possible paths. Either be as expressive as NFO which gives you less higher-level primitives, or go for some sub-set of NFO in a higher-level approach. NDL is an example of the latter. and there are probably several others that I am not aware of.

The trade-off is the size of the sub-set versus the level of abstraction.

For now, I tend to try covering the entire NFO, since getting rid of manually encoding byte sequences seems like a bigger benefit than trying to cover some useful subset.
In addition, I believe that having a subset implies that some user will get into trouble because the subset is not covering all that he wants. At that moment, the user can either wait for the developer to update the tool, or swich to some lower-level language (ie NFO at this moment). The grand trick is now that you'd want to provide such a user with a gradual fallback. Obviously, we'd need something in-between the high-level subset thingie and the bottom to make this happen.

(in short: First get a good base covering everything, then try to gradually upgrade to higher level primitives that co-operate in a good way with lower level primitives seems like a good path to me.)

[Alberth]

Well, compiler should know what is expected (whether extended byte, word, byte, etc...) and use it, UNLESS user specifies otherwise. Therefore if user says dword, put there a dword even if spec says byte

Wow, nice one.

What's even better, with my 3-step process this one is not difficult any more, I just need to inject the explicit size information in the NFO field list.
(now I 'just' need to implement the 3-step process first... ).

[Alberth]

If there would be some feature not possible to be done using GRL and you need that feature, you have to stick with old-style NFO for your entire project. This would be quite a bit of disadvantage ....

As the grl-bytes is included in the definition, you should be able to express anything a plain NFO can (by using plain old NFO commands).

Well, yes and no. Encoding action 6 with grl-bytes (currently the only way to do it, but I expect this will change) also requires that the following sprite be a grl-bytes.

Not exactly, you can cheat/use the computer.

Run the ./grlenc convertor, and you get the NFO byte sequence of the next sprite, so you can count your offset. Fill that number as unsigned numeric value in 'offset' et voila, you have a working Action6 with correct offsets.
One problem remains with this approach, namely you cannot fix the size of the extended bytes.

These questions come from a convoluted double-tac-nuke I pulled some years ago: I wrote an action 6 that made three modifications, except that the third modification used parameter FF. (Making it the terminator, not a modification, and making the following 3 bytes "extraneous".) I then used another 6 to change the FF to the desired parameter number, and a 9 to skip the first 6 if I didn't want the second 6 to make its third modification at all.

Something like

Code: Select all

* 0   09 00 01 \7> 02 01
* 0   06 00 01 07 FF
* 0   06 01 01 08 01 01 0C FF 01 05 FF

(The offsets in the second 6 may not have any realistic use. I know the one I wrote did what I intended for it to do.)

Oh you're so evil

With my 3-step process, I would expect this to work with a warning/error that the first FF looks an awful lot like the list termination byte.
We shall see....

[Alberth]

It is enough to encode all future grfs, but if something has been done once, it'll probably be done again, so anything that has been done requires serious consideration; both whether or not it's necessary for GRL to support it and whether or not there a GRL-supported and functionally equivalent form.

I couldn't agree with you more here.

To give a direction, I think that GRL should be able to handle it, one way or another (without fallback to grl-bytes). Once there is a good base, we can think about making new primitives at a higher level that gives better/more compact notation for what many people typically do in GRF (for example, have automatic assignment of Action2 cargo-id's rahter than having to specify this manually can be one step, have a nested case statement that automagically decodes to a set of *-graphics GRL primitives (and then to Action2 NFO bytes) is a possible second step)).

Going up in abstraction level too fast gives the risk that you're loosing users because you don't entirely cover their needs.

[Eddi]

Going up in abstraction level too fast gives the risk that you're loosing users because you don't entirely cover their needs.

i recognise this as a side hit against me

it is true that a top-down approach like mine cannot cover the whole base initially. i am aware of this, and try to push the development forward as fast as possible to a broad support base, which is typically limited by infections of "real life".

on the other hand, a bottom-up approach like yours may run into problems, because you have to deal with a lot of legacy constructs, that the users will want to mix, you won't just get away with "deprecating" the whole base-constructs when people have built up a significant amount of code using them.

[AndersI]

If we look at NFO code as 'machine code', then the bottom-up approach is mostly like an assembler where there's often a 1:1 relationship between mnemonics and machine instructions. Assemblers have evolved into macro assemblers and almost high level languages, and so could GRL. Assembly programming gives greater freedom, mostly at the cost of longer development time.

The top-down approach is the compiler, compiling a high level language into (a subset of) the machine code. Using a language suitable for the task at hand normally gives the benefit of faster development, and sometimes (depending on the complexity of the language) is also easier to learn. The resulting GRF would probably not be minimal (or optimal), but could equally well get the job done.

Are there, with todays computers and the current TTDPatch design, any problems with 'fat' GRF files? Does minimizing/optimizing give any notable benefit for the end user?

Looking at repeated requests for coders in many graphics threads, it seems learning ease and development speed should be the most important characteristics of a 'higher level NFO', if the goal is to actually produce more GRF files, faster.

[michael blunck]

Using a language suitable for the task at hand [...]

Problem is that we didn´t find (or develop) such a language, although we´ve been discussing that particular problem for years now ...

Are there, with todays computers and the current TTDPatch design, any problems with 'fat' GRF files?

Sprite limitation comes to mind, especially when programming stations.

[...] the goal is to actually produce more GRF files, faster.

What for? Where should originate the needed graphics from?

IMO, this is a more complex problem. Personally, although I´m fit in writing NFO, releasing sets isn´t equally fast - not at all. Even not if all of the graphics have been drawn. (There´s a common misunderstanding about NFO: most "coding" action is copy&paste, but far more time is used for including sprites, e.g., getting offsets right.).

So, coding and even drawing graphics isn´t the most time-consuming action. That would be the "finishing" of a set (laying out the structure of the set, how to handle all those intricacies, balancing, those numerous test runs, re-drawing sprites which don´t look that good in game as thought before, packaging of sprites, optimising the code, ...) which is a long-lasting process and it usually happens that during that process there are new ideas arising ...

regards
Michael

[Alberth]

Looking at repeated requests for coders in many graphics threads, it seems learning ease and development speed should be the most important characteristics of a 'higher level NFO', if the goal is to actually produce more GRF files, faster.

Maybe I am too cautious. On the other hand, apparently nobody has yet succeeded in writing an 'easy and fast to learn' language that is usable for a wide audience, otherwise nobody would be programming NFO any more, except one compiler author, a few users that do highly advanced GRF wizardry, and the people going GRF development.
I wouldn't have started GRL in that case.

So where is that nice, easy to learn, powerful language that every-one wants?

If I may believe comments by DaleStan in other (previous) discussion-threads there have been plenty of attempts (even today, there is NDL and NFO2 as alternative development thread), so it is not a lack of willing coders.
Another reason may be that the compilers cannot keep up with GRF development, ie GRF is a moving target, and users want to have the latest/newest features. I cannot give an answer to that question, I have been around a few months only.
The third reason that I can think of, is that users do very complex, widely varying things, which makes a (generic) high level language highly complex or impossible to implement.

High level primitives may be powerful and easy to use, but they are also small, they are very specific for one purpose. High-level, general purpose primitives do not exist (they would become unmanagably complex). Thus, you have to provides lots of high-level primitives before you get any form of user base, which is a huge investment.

This can be reduced if you pick the primitives exactly right. Trouble is that you need to have intimate knowledge of your users and their wishes and desires w.r.t. the 'programs' they make, to pull that off (in my experience you need to know even better than they know).
I am not the right person for that job. I have been around only a few months, written 0 real-life NFO files, cannot draw anything except squares, circles, and lines in diagrams, and have no clue what you can do by combining all the available primitives. (ie I have just been given with new building blocks and learning about the shape of all the blocks. How to make a shed from the blocks is far beyond me, let alone the big buildings that other people are making. I have no chance what-so-ever deciding about the proper form of pre-fabricated modules for them.)


I understand the problem, I just don't think there is a quick answer, at least not from me. One of the more fundamental problems may be that there is no general agreement of what a "easy to learn and fast to develop" language really is.
Just a set of examples "I would expect to write this program text down for specifying this animated wagon/plane/whatever" would already be progress as it narrows down the options ("make something that is nice and easy" is so ambigious that every-body has a different, non-compatible idea about it)

The NDL project may be better to your liking, since it takes a compiler-like approach.

[AndersI]

Problem is that we didn´t find (or develop) such a language, although we´ve been discussing that particular problem for years now ...

Could it be that we are looking at the 'problem' at a much too deep level, existing NFO, the machine code, instead of attacking it from above: "What do we want to accomplish when creating a GRF? How can we do that in the easiest way?"?

[...] the goal is to actually produce more GRF files, faster.

What for? Where should originate the needed graphics from?

You ignored a very important if there. Also, I wrote earlier: "judging by the number of requests for coders..."

but far more time is used for including sprites, e.g., getting offsets right.

This particular problem is quite elegantly solved in GrfMaker, at least for trains and road vehicles. Maybe that part should be isolated out into a separate program?

which is a long-lasting process and it usually happens that during that process there are new ideas arising

So you need a development process that makes changes very easy to implement.

So where is that nice, easy to learn, powerful language that every-one wants?

?? You are writing one variant right now, aren't you? GrfMaker is one such attempt, NDL another, and so on. I don't think you'll ever find one language that everybody loves - just look at all the other language wars there are out there. It's especially difficult to convert the people that already have a religion, sorry, language (process, whatever). We need diversity, and hopefully the good ones will flow up to the top.

Another reason may be that the compilers cannot keep up with GRF development, ie GRF is a moving target, and users want to have the latest/newest features.

Some users want to have the latest features, obviously. Beginners don't need that, as long as there is a migration/evolution path that can be followed. A simple "sprites in on one end, GRF out in the other" would help some graphics artists to see their creations in game earlier, keeping up the interest in the project better than six months waiting for a coder (admittedly, in six months anyone should be able to code a 'put graphics in game' GRF him/herself with Notepad). The step to creating a 'set', with cargo sprites, livery overrides, changes as time progresses, dependency checks and all that other fancy stuff is quite a bit larger.

users do very complex, widely varying things, which makes a (generic) high level language highly complex or impossible to implement.

Only if you insist in modeling the machine code (NFO) directly into your HLL. We do actually program computers in C++, Pascal, Perl, etc. etc. nowadays, without seeing the need for a 1:1 relation between HLL and machine code.

One of the more fundamental problems may be that there is no general agreement of what a "easy to learn and fast to develop" language really is.

A good start could be: If it can easily be done by the computer, do it with the computer.

The human should not have to
- count sprite lines, characters in a string, bytes in a sprite line, etc.
- convert numbers between bases
- convert numbers to multiple bytes
- care about endian-ness
- know the internal structure of the machine code
- calculate offsets
- memorize a lot of things such as 'magical numbers', it should be possible to look things up quickly instead (that's more a question of an IDE than the actual language).
- ...

[DaleStan]

If I may believe comments by DaleStan in other (previous) discussion-threads there have been plenty of attempts (even today, there is NDL and NFO2 as alternative development thread), so it is not a lack of willing coders.
Another reason may be that the compilers cannot keep up with GRF development, ie GRF is a moving target, and users want to have the latest/newest features. I cannot give an answer to that question, I have been around a few months only.

There have, and NFO is a moving target.

Moving quite slowly right now, but moving nonetheless.

[Alberth]

There´s a common misunderstanding about NFO: most "coding" action is copy&paste

The sentence is now completely out of context, but I find this one intriguing.
May I read this as creating a web-page with these fragments (or even better in the NewGraphicsSpecs) will end the need for GRL, NDL, GRFmaker, and others?

[Alberth]

Problem is that we didn´t find (or develop) such a language, although we´ve been discussing that particular problem for years now ...

Could it be that we are looking at the 'problem' at a much too deep level, existing NFO, the machine code, instead of attacking it from above: "What do we want to accomplish when creating a GRF? How can we do that in the easiest way?"?

Possibly, it is a natural concept that once you can do A, you want B, then C, etc, and before you know it, the have the entire complexity of the GRF/NFO language coming down at you.

If you want to try this, I'd skip "How can we do that in the easiest way?" at first, since that is a less than trivial question.

Another reason may be that the compilers cannot keep up with GRF development, ie GRF is a moving target, and users want to have the latest/newest features.

Some users want to have the latest features, obviously. Beginners don't need that, as long as there is a migration/evolution path that can be followed. A simple "sprites in on one end, GRF out in the other" would help some graphics artists to see their creations in game earlier, keeping up the interest in the project better than six months waiting for a coder (admittedly, in six months anyone should be able to code a 'put graphics in game' GRF him/herself with Notepad). The step to creating a 'set', with cargo sprites, livery overrides, changes as time progresses, dependency checks and all that other fancy stuff is quite a bit larger.

One of the problems here is lack of an adequate guide into the primitives, and how to use them (especially combining them). NewGraphicsSpecs is more a reference (ActionX has this encoding, and does this) than a user's guide (to get your sprites moving combine this and this action in this way).
There are a few beginners guides, but they are very basic. For example, there is no guide what I can do with an industry (animation, production cycles, callbacks and how they are set up, etc). This information has to be deduced from the forums or from careful reading of NewGraphicsSpecs.

Combined with the console-bound 'grfcodec', and those alien hexadecimal numbers, I can understand that most graphics artist are not even attempting to write an NFO file themselves. (I don't know what GrfMaker does, I don't have a Win* system, nor do I have MS Word.)

One of the more fundamental problems may be that there is no general agreement of what a "easy to learn and fast to develop" language really is.

A good start could be: If it can easily be done by the computer, do it with the computer.

The human should not have to
- count sprite lines, characters in a string, bytes in a sprite line, etc.
- convert numbers between bases
- convert numbers to multiple bytes
- care about endian-ness
- know the internal structure of the machine code
- calculate offsets
- memorize a lot of things such as 'magical numbers', it should be possible to look things up quickly instead (that's more a question of an IDE than the actual language).
- ...

Hmm, looks like the requirement list of GRL to me...

[DaleStan]

One of the problems here is lack of an adequate guide into the primitives, and how to use them (especially combining them). NewGraphicsSpecs is more a reference (ActionX has this encoding, and does this) than a user's guide (to get your sprites moving combine this and this action in this way). There are a few beginners guides, but they are very basic.

Because beginners require the basics, and keeping anything non-basic up-to-date (or even generating a reasonable example for something like var 7E) is horrendous.

Combined with the console-bound 'grfcodec',

The GRF format is public knowledge. The only reason GRFCodec is still a console program is that no one thinks it's worth the hassle to write a portable GUI version.

[michael blunck]

> There´s a common misunderstanding about NFO: most "coding" action is copy&paste, but far more time is used for including sprites, e.g., getting offsets right.

This particular problem is quite elegantly solved in GrfMaker, at least for trains and road vehicles. Maybe that part should be isolated out into a separate program?

I meant the relation, I didn´t mean that positioning sprites would be too complicated in the first place. And even TTDPatch has a tool included for it (grf-author helper by Oskar).

> coding and even drawing graphics isn´t the most time-consuming action. That would be the "finishing" of a set [...] which is a long-lasting process and it usually happens that during that process there are new ideas arising

So you need a development process that makes changes very easy to implement.

Well, yes. But

laying out the structure of the set, [...], balancing, those numerous test runs, re-drawing sprites which don´t look that good in game as thought before, packaging of sprites, optimising the code, ...

what kind of "development process" would help with this? And OTOH, a "long-lasting" development process isn´t bad at all. Sometimes a problem may well lie around for a very long time before it get´s a sudden solution ( I do have such with NewStations ATM). We´re not commercial here.

The human should not have to
- count sprite lines, characters in a string, bytes in a sprite line, etc.
- convert numbers between bases
- convert numbers to multiple bytes
- care about endian-ness
- know the internal structure of the machine code
- calculate offsets

Most of these should be handled already by DaleStan´s nforenum?

regards
Michael

[DaleStan]

The human should not have to [hyphens changed to numbers -- DaleStan]
1) count sprite lines, characters in a string, bytes in a sprite line, etc.
2) convert numbers between bases
3) convert numbers to multiple bytes
4) care about endian-ness
5) know the internal structure of the machine code
6) calculate offsets

Most of these should be handled already by DaleStan´s nforenum?

Info Version 7, supported by both GRFCodec and NFORenum, makes the first four of those and part of 5 complete non-issues: \b127, \d1990/01/01, \b*255, \7>, \2u>, \D=, &c. except for getting sprite 0 right, which only NFORenum will fix automatically. GRFCodec will tell you the correct change, so if you make your sprite 0 as "0 * 0 \d0", the change becomes the required value.

One idea I've had for some time now, but never actually implemented, is to allow RPN in the \[bwd] escapes, and then use extraneous numbers on the RPN stack to cause NFORenum (GRFCodec? both?) to emit Action 6s, eliminating 6 as well. (Of course, these 6s would only support 1, 2, and 4 byte overwrites.)