GRFCodec feature discussion thread

[DaleStan]

I thought about it a little more, and I would be a whole lot happier mnemonicing \2r as reset (or rst), not return. Operation 0F doesn't "return" in the NFO sense of the word, but it does reset the state of the variational, as if the portion from the first <var> to the 0F <op>, inclusive, didn't exist.

I'll hold off on committing this for a while, but those of you who want to start using them now can apply these diffs and compile.

I'll probably add support for \2sto and \2rst before committing, and make them be the forms that grfcodec generates.

[RSpeed tycoonfreak]

I have a problem, I want to encode a file but it says "cannot read truecolour pcx files!"
Is it possible to check repair the pcx file with grfcodec?

[DaleStan]

Not unless you want to fix it yourself. The code can be found at [url]svn://svn.ttdpatch.net/misc/grfcodec[/url].

The correct solution is to decode one of the trg?[r].grf files, and apply the palette from the resultant trg?[r].pcx file to your PCX file. If your editor can't write paletted PCX files, then it's time you get a new editor. I recommend GIMP.

EDIT to replace LART-post with answer.

[DaleStan]

It went and happened again, except this time it's Csaboka.

Csaboka just added a new operator (10) to variational action 2, and I'm wondering what escape sequence to use for it. This time, I don't have much in the way of ideas. So far, the best I've thought up are 'pst' or 'psto'.

Or maybe it's time I give the idea of writing something that looks like C code some more serious thought. Anyone care to wrangle some bison?

[Csaboka]

I'm sorry for breaking NFORenum again; I don't plan to add any other breaking change in the near future.

About the C code idea: I know it's easy for me to say (you have to implement it after all ), but I like the idea. Most of the action2 operators are close to C operators, especially that the stores has the semantics of the C = operator, and 0F could be the semicolon or the colon operator, whichever you prefer. The difference would be that your expressions wouldn't have precedence rules, but would be read left to right (maybe excluding the = operator). The problems are unsigned operators, and min/max. Unsigned operators seem to be solvable easily by adding an "u" or something. Min/max would be trickier, since people would like to use them as functions, not as binary operators, and that would complicate the parsing.

[Csaboka]

After some thinking, here is my first proposal for the grammar of the expressions. I couldn't cope with an = operator, so I decided to replace it with an -> operator that gets the destination on its right hand side. I also couldn't cope with min and max, those are left as binary operators, however strange they seem. I hope this can still be a good starting point.

(Nonterminals are in angle brackets, terminals are quoted)

Code: Select all

<expression>::= <primitive_expr> (<operation>)*
<operation>::= ("->" <lvalue>) | (<operator> <primitive_expr>)
<lvalue>::= ( "reg" | "persreg" ) "[" <primitive_expr> "]"
<operator>::= "+" | "-" | "*" | "%" | "/" | "%u" | "/u" | "&" |
              "|" | "^" | "min" | "max" | "umin" | "umax" | ","
<primitive_expr>::= <const> | <variable> |
                    "(" <variable> [">>" <const>] ["&" <const>]
                    [ ["+"|"-"] <const> ["/"|"%"] <const> ]
<variable>::= "var"<const> | "var"<const> "("<const>")" |
              "reg" "[" <const> "]" | "persreg" "[" <const> "]"

(It looked better on paper, I hope it can be followed...) Const is any integer constant either in decimal (minus sign allowed where appliable) or in hexadecimal, with the prefix "0x". I haven't tested the grammar too much, but I think it would work nicely. I hope it isn't ambiguous, it would take more testing to find that out.

For those who haven't learned formal languages, the expressions would work like this:

• Expressions are interpreted left to right, without any of the usual precedence rules. Comma can be used to separate calculations, and is compiled to operator 0F. Parentheses can used only for specifying varadjust.
• To store a value in a register, you use the -> operator, followed by either "reg[xx]" or "persreg[xx]". The operator returns its left operand, so you can continue calculating with that, or use the comma operator to start with a clean slate.
• If you need constants in the expression, you can write them normally, the compiler will automatically turn them into reads of var1A with a correct and-mask
• Variables that don't need parameters can be used as varxx (xx is a hex number), parametrized variables can be used as varxx(yy) (yy is a constant). To read registers, you can use reg[xx] instead of var7d(xx), and persreg[xx] instead of var7c(xx). Other aliases can also be added if needed.

And finally, an example may make it more clear. Let's calculate the square of the Euclidean distance between the center of the closest town and the current industry. (Assume that the XY of the town is already copied into register 0, since we can't access town and industry vars at the same time.)

Code: Select all

(reg[0] & 0xFF) - (var80 & 0xFF) -> reg[1],
(reg[0] >> 8 & 0xFF) - (var80 >> 8 & 0xFF) -> reg[2],
reg[1] * reg[1] -> reg[1],
reg[2] * reg[2] + reg[1]

(You can't change the order in the last step because precedency is ignored)
For comparison, this is how it currently would look like (assuming dword size):

Code: Select all

    7D 00 20 \dxFF
\2- 80 20 \dxFF
\2s 1A 20 \d1
\2r 7D 00 28 \dxFF
\2- 80 28 \dxFF
\2s 1A 20 \d2
\2r 7D 01 20 \dxFFFFFFFF
\2* 7D 01 20 \dxFFFFFFFF
\2s 1A 20 \dx1
\2r 7D 02 20 \dxFFFFFFFF
\2* 7D 02 20 \dxFFFFFFFF
\2+ 7D 01 20 \dxFFFFFFFF

Even with DaleStan's escapes serving as mnemonics, it still mostly a bunch of hex digits, and you have to specify everything, can't have defaults.

I'm open to suggestions, tell me what you think about it.

[DaleStan]

After some thinking, here is my first proposal for the grammar of the expressions.

Thanks. I was attempting to construct something FAR more complex (with named and sized temporary variables, for starters), and stalling out completely.
This might represent some happy medium. I think the assignment and function-call definitions are valid, but someone with more experience should probably check it.

I've added <XY>, which is the xy-offset parameter for many industry and industry-tile variables, as two comma-separated values, and "opXX"/"funcXX", so new operators can be used immediately upon introduction.

Code: Select all

<block>::= "{" (<expression> ";")+ "}"
<expression>::= ( <primitive_expr> (<operation>)* )
              | ( <lvalue> "=" <expression> )
              | ( <func> "(" <arg_list> ")" )
              | ( "(" <expression> ")" )
              | ( "(" <expression> ")" <operator> "(" <expression> ")" )
<arg-list> ::= ( ( <primitive_expr> ",")* <expression> ( "," <primitive_expr> )+ )
             | ( ( <primitive_expr> ",")+ <expression> ( "," <primitive_expr> )* )
             | ( <expression> ( "," <expression> )+ )     (?)
<operation>::= <operator> <primitive_expr>
<lvalue>::= ( "reg" | "persreg" ) "[" <primitive_expr> "]"
<operator>::= "+" | "-" | "*" | "%" | "/" | "%u"
            | "/u" | "&" | "|" | "^" | "," | "op"<const>
<func>::= "min" | "max" | "umin" | "umax" | "func"<const>
<primitive_expr>::= <const> | <variable> |
                    "(" <variable> [">>" <const>] ["&" <const>]
                    [ ("+"|"-") <const> ] [ ("/"|"%") <const> ]
<variable>::= "var"<const> | "var"<const> "(" (<const>|<XY>) ")" |
              "reg" "[" <const> "]" | "persreg" "[" <const> "]"

Eliminating the comma operator would remove the ambiguity that can currently appear in function calls, but there's still some ambiguity in the length of <expression> in assignment. Don't Do That (TM) might be a good rule for now, but eventually precedence would have to be introduced.
I also don't like the idea of

Code: Select all

min(reg[0]=$COMPLICATED;reg[0]*reg[0], $FOO);

Semicolons have no business being in the middle of function calls.

The last option for <arg-list> requires the use of one register to encode. I'd probably use EF, to leave the high 16 registers for free for coder's use. Similarly, the last option for <expression> requires arbitrarily many registers, and I'd probably default to using E0..EF, and abort if more than 16 registers are required.

Given the above, you can, if you so desire, type something like func00( <several expressions to sum go here> ); This will have the effect of adding all the expressions, but I don't see any point in adding a explicit "add" mnemonic. You can also do the same with func01 or func06 (subtract and divide), for example, with undefined results. If you want to be an idiot, I won't stop you. I will make it difficult, though.

I suppose I need to go read the flex and bison docs now.

[Csaboka]

Sounds very nice The only drawback I can see is that your version allows more things than the original action2 syntax (<expression> <op> <expression>, for example), so you can't get the original expression back from a GRF file, unless you have the sources; my version is roughly a one-to-one mapping. This analogous to C and ASM - you can get ASM back from the binary, but not C. This isn't necessarily a bad thing, of course; I don't think many people modify NFO decoded from a production GRF.

The last option for <arg-list> requires the use of one register to encode. I'd probably use EF, to leave the high 16 registers for free for coder's use. Similarly, the last option for <expression> requires arbitrarily many registers, and I'd probably default to using E0..EF, and abort if more than 16 registers are required.

I'd suggest to make these things optional (disabling them would of course make the more advanced expressions invalid). I'm sure there are coders who don't want things to happen behind their backs.

Oh, and there is one problem I noticed in my grammar after submitting, and it affects your version as well. It isn't exactly a bug, but it surely breaks the expectations of coders. A simple expression like

Code: Select all

var0a & 0xFF

will be compiled to two variable reads (0A and 1A) linked via operator 0B. To get the intended one-variable version, you need

Code: Select all

(var0a & 0xFF)

which is a rather subtle difference.
Similarly,

Code: Select all

var0a >> 8

is invalid, since the right shift operator is valid only if it's in parentheses. The working version is

Code: Select all

(var0a >> 8)

but I doubt anyone would think this is OK.

I guess we need a special case for expressions that contain one variable read only, or maybe for the first variable read in expressions.

EDIT: actually, your grammar is ambiguous at this point.

Code: Select all

(var0a & 0xFF)

can be interpreted as an <expression> in parentheses, or as a <primitive_expr>. I don't know how bison works, but if it's possible to resolve ambiguities manually, it should prefer the second interpretation over the first.

[DaleStan]

I'd suggest to make these things optional (disabling them would of course make the more advanced expressions invalid). I'm sure there are coders who don't want things to happen behind their backs.

I've added the concept of directives, which may be used to provide such setup info, eg:
#size (byte|word|dword)
#vars (this|related)
#tempregs (none|<byte>|<byte>..<byte>)

The defaults would be
#size byte
#vars this
#tempregs E0..EF

The current parser shouldn't require parens around primitives (in fact, I think it forbids them), but won't know for sure until I have a functional lexer and driver for testing.

I've eliminated the argument-separator vs comma-operator ambiguity by changing those expressions to "non-comma" expressions, which are exactly like regular expressions except that comma is not valid in <operator>s not enclosed by parentheses.

[Csaboka]

The current parser shouldn't require parens around primitives (in fact, I think it forbids them), but won't know for sure until I have a functional lexer and driver for testing.

Well, you do need parens around primitives, if there are more than one.

Code: Select all

(reg[0] & 0xFF) - (var80 & 0xFF)

is very different from

Code: Select all

reg[0] & 0xFF - var80 & 0xFF

Will your code detect which expressions can be turned into primitives without the coder telling explicitly? What about the >> operator, which is valid for primitives only?

Oh, BTW, we're getting very far from the title of the topic. Could you rename it to something more general like "GRFCodec feature discussion"?
Or should I split this discussion into a separate topic?

[michael blunck]

> I'm open to suggestions, tell me what you think about it.

OK, here are my 2cc.

Well, first of all, ".nfo" is complicated, but it is in two ways: one is the visual form of the syntax but the other is the complex underlying structure of TTD resp the Patch´s implementation.

> Even with DaleStan's escapes serving as mnemonics [...]

Well, from the beginning, I didn´t count on this. IMO, those mnemonics aren´t better than the original .nfo "mnemonics". In most cases, those are even better because they´re more memorizable (or even more ergonomic). And then these "new" mnemonics only cover a part of .nfo´s syntax, i.e. IMO in the first place they further add to the complexity of natural .nfo syntax (because you´ll have to use both, see your examples) and hence doesn´t make life easier.

> here is my first proposal for the grammar of the expressions [...]

Well, meanwhile the whole thing gets even more complex than Dale´s initial approach. I doubt its usefulness for coders if you don´t take into account some basic principles:

- you´ll have to be complete, meaning that the new syntax should be stand-alone, there´s not much use if we´ll have to use both forms in the end,
- the syntax should be easier memorizable than original .nfo,
- you should avoid to add to .nfo´s already existing complexity.

I know this whole thing is far from being easy. Some time ago, I had a go on it myself, but eventually scrapped it ...

regards
Michael

[Csaboka]

With all respect, I don't think your concerns are justified.

- you´ll have to be complete, meaning that the new syntax should be stand-alone, there´s not much use if we´ll have to use both forms in the end,

I think both my and DaleStan's proposals are complete - you can express any calculation that is possible with a variational action2. The opXX and funcXX identifiers are a safety solution that ensures that new operators can be used even before GRFCodec is updated for them.

- the syntax should be easier memorizable than original .nfo,

I don't know about you, but I have a horrible memory for numbers. Everything that's expressed with words or well-known symbols is easier to memorize for me than a bunch of numbers. Is "0A 08 FF" easier to read/write than "var0A >> 8 & 0xFF"?

- you should avoid to add to .nfo´s already existing complexity.

I don't think we're adding complexity. DaleStan's version actually hides some of the complexity because it can interpret expressions that cannot be coded directly and convert them to valid ones. You can just throw an expression at it and let it handle the complexity of figuring out how to calculate it.

My ultimate dream would be an NFO that reads like a normal programming language instead of hex, but I've already explained that some time ago and the idea was turned down. It would reduce the "syntax complexity" you're talking about, and the coder could concentrate on what the code actually does. There is a reason why we're coding TTDPatch in assembly language and not machine code, even though you still need to know a lot about your CPU either way...

[michael blunck]

I think both my and DaleStan's proposals are complete - you can express any calculation that is possible with a variational action2. [...]

Well, seems I expressed myself badly concerning this point. Let´s try again: IMO there´s not so much sense in only transforming var.action2s into a new syntax and rely on traditional .nfo for everything outside the scope of var.action2s. That´s what I meant with "adding complexity". In fact, you´ll have to learn a set of two different "mnemonics".

>- the syntax should be easier memorizable than original .nfo,

I don't know about you, but I have a horrible memory for numbers. Everything that's expressed with words or well-known symbols is easier to memorize for me than a bunch of numbers.

Yes, I fully understand. But IMO, the thing which has to be memorized in .nfo in the first place isn´t "numbers" but "well-known symbols" resp. "patterns". (E.g. a pattern like "var-shift-mask". If, in the end, the coder is unsure about whether a "number" (a var in this case) in a pattern should read 42, 43 or 44, this may be looked up in the docs, the same way as it´d be needed in your proposal.)

Is "0A 08 FF" easier to read/write than "var0A >> 8 & 0xFF"?

Yes indeed, the pattern "0A 08 FF" is much more easier to memorize than the pattern "var0A >> 8 & 0xFF". It´s both shorter and it´s by means simpler, talking alone about the number of special signs in the latter pattern.

>- you should avoid to add to .nfo´s already existing complexity.

I don't think we're adding complexity.

Well, yes. Regarding my statement above, the addition of another different syntax to the already existing .nfo syntax can be best described by "adding complexity". Isn´t it?

My ultimate dream would be an NFO that reads like a normal programming language instead of hex, but I've already explained that some time ago and the idea was turned down.

Well, not by me. That´s exactly the way I was thinking about a couple of years ago (when .nfo was even simpler as it is today).

It would reduce the "syntax complexity" you're talking about, and the coder could concentrate on what the code actually does.

Yes. I second this.

regards
Michael

[eis_os]

How to hide the corner cases in a good way?

The problem while you can do A LOT with nfo currently there are to many restrictions, odds and I call the them corner cases where some stuff isn't allowed or does something different because a bit is overload to actually allow even more functionality.
NFO grows exuberantly. While it's good to have even more and more functions the handling will get worse. (I do think I told that already several times. )

If you put an layer on top you will get questions why the compiler does this or that wrong or don't do it the way it should do it. The result will be again a hack to the nfo syntax in some form to allow this to work correctly. Meaning it gets even worse.

From my standpoint, creating a compiler on top of the current nfo system won't be good...

I propose some NFO 2 which work in a more traditional system like a VM that can do real stuff and stop the madness of the totally different ways. Then you can write an compiler on top of it and won't need to deal with all the specials you have.

[DaleStan]

The current parser shouldn't require parens around primitives (in fact, I think it forbids them), but won't know for sure until I have a functional lexer and driver for testing.

Well, you do need parens around primitives, if there are more than one.

Code: Select all

(reg[0] & 0xFF) - (var80 & 0xFF)

is very different from

Code: Select all

reg[0] & 0xFF - var80 & 0xFF

Hm... yeah.

There are three solutions. Make that an error, set precedences so that the second behaves as the first, and set precedences so the second behaves as

Code: Select all

(reg[0] & 0xFF - var80) & 0xFF

I'd guess that the second option is most likely to be the one intended, and that the first option is the hardest. I believe that the second option is the one that actually happens, but I don't know for sure.

I think both my and DaleStan's proposals are complete - you can express any calculation that is possible with a variational action2. [...]

Well, seems I expressed myself badly concerning this point. Let´s try again: IMO there´s not so much sense in only transforming var.action2s into a new syntax and rely on traditional .nfo for everything outside the scope of var.action2s. That´s what I meant with "adding complexity". In fact, you´ll have to learn a set of two different "mnemonics".

*We* *may* to learn a new set of mnemonics. We may also choose to stick to the old ones.

People who don't already know how to write an advanced variational 2 can choose to learn either format. Yeah, so you have to know NFO if you want to debug the compiler, but that's just the same as knowing machine code to debug NASM or gas, or knowing assembly to debug gcc.

Is "0A 08 FF" easier to read/write than "var0A >> 8 & 0xFF"?

Yes indeed, the pattern "0A 08 FF" is much more easier to memorize than the pattern "var0A >> 8 & 0xFF".

But which is easier to read? If you're looking at the middle of a long action 2, how do you know that "0A 08 FF" is not nvar and a callback result?
The only way I know is to read the entire sprite up to that point.

If, instead, you're looking at "var0A >> 8 &0xFF" (or, in a byte-sized varaction, "var0A >> 8") it is guaranteed that you're looking at a variable access, and no further thought is necessary.

As implied above, the spec for a simple variable access is:

Code: Select all

varXX[(<param>)] [& <mask>] [(+|-) <add>] [(%|/) <divmod>]

If you want to add fifteen to variable 43, type "var43 + 15", and you will get exactly what you should: "43 40 FF [FF [FF FF]] 0F [00 [00 00]] 01 [00 [00 00]]" (Which one is generated depends on what size you've specified.)

It´s both shorter and it´s by means simpler, talking alone about the number of special signs in the latter pattern.

It's definitely not shorter for the wider varactions, and and "simpler" does not necessarily mean "easier to remember".
Maybe you have memorized the byte-codes for all seventeen varaction2 operations, all thirteen action 7/9 conditions, all thirteen action D operations, and all seven GRM operations. More power to you. But I haven't, and I think most people can't.
Maybe you can remember which of bit 6 and bit 7 indicate a shift-and-add-divide operation. I have to look it up every time.
I think reducing the number of documentation-accesses is a noble goal.

Nobody's forcing you to use this. Use the byte-stream format and be done with it.
If you have constructive criticism (eg "I think the grammar handles <foo> incorrectly -- I would expect <bar>, but am getting <baz>.") then by all means, go for it.
If all you have is "This is stupid and I won't ever use it, then go say that someplace else. Or better yet, not at all.

>- you should avoid to add to .nfo´s already existing complexity.
I don't think we're adding complexity.

Well, yes. Regarding my statement above, the addition of another different syntax to the already existing .nfo syntax can be best described by "adding complexity". Isn´t it?

Hands up, everyone who can make sense of "var0A + 1" on the first try?
Hands up, everyone who can make sense of "0A 40 FF FF 01 00 01 00" on the first try?

There's "complexity" and there's complexity. By your definition, C is adding complexity to ASM, because you can put assembly code in C programs, and, in fact, there are some things that require assembly code. (Some assembly interfacing, runtime-generated var-args calls, explicit segment registers, &c.) If this is indeed what you mean, then I'm going to have to throw out your definition as completely absurd.

The problem while you can do A LOT with nfo currently there are to many restrictions, odds and I call the them corner cases where some stuff isn't allowed or does something different because a bit is overload to actually allow even more functionality.
NFO grows exuberantly.

And this is different from machine code exactly how? Every time Intel or AMD release a new processor, they add new instructions, and compilers have to be updated if they want to take advantage of this new functionality.

But the old code, both high-level, assembly, and machine, remains valid.

[Csaboka]

@Michael: I can't really follow you. You first argue that using a human-readable longhand for expressions is worse than raw hex, then you agree with me about how a human-readable NFO format would be a good thing.

There are three solutions.[...]

I think we're talking about two different things here. My original idea was left-to-right interpretation without precedence, so "var0a + var0b * var0c" would do (var0a+var0b)*var0c. The only place where parens are allowed would be the primitive values, that is, <valuable><varadjust> pairs, since those happen independently from the rest of the calculation. I thought you kept this part of my proposal, except allowing parens anywhere, in which case the contents of the paren is calculated in a separate var before using it. The coder would still know the order of the operations, and no precedence handling is required.

My only problem was that in this case, you can't tell whether things like "(var0a & 0xF)" are meant to be a <variable><varadjust> pair or an expression that uses operator 0B.

So, just to avoid talking past each other: do you want the compiler to handle precedence normally, or go left-to-right, except where parenthesed explicitly (i.e. follow my proposal)?

[Patchman]

My only problem was that in this case, you can't tell whether things like "(var0a & 0xF)" are meant to be a <variable><varadjust> pair or an expression that uses operator 0B.

It doesn't matter to the grf coder how this is resolved, so if it can be done as a varadjust, you should generate that, and otherwise use operator 0B. Similar for addition.

[DaleStan]

I think the target I'm aiming for is "left to right, unless parenthesized explicitly, or the right-hand side is a <variable>".

So, all sequences that can be parsed as "varXX[(<param>)] [>> <shift>] [& <mask>] [(+|-) <add>] [(%|/) <divmod>]" have implicit parens, and everything else is explicit or left-to-right.

<param>, <shift>, <mask>, <add>, and <divmod> are all integer constants; anything that isn't would indicate that the previous operator should be encoded as an <operator> byte.

[Patchman]

I think left-to-right would be very counterintuitive though.

[AndersI]

I applaud any initiative that makes NFO easier to read. There's another factor involved here too - most of you have been around for quite a while, started understanding/memorizing NFO when it was simpler, and learnt the new things one by one when they were developed. When a coder newbie (such as I) tries to dive into it now, there's just *too much* to learn before getting over the threshold. The Wiki is great, but there is soo much to read...

As NFO code is positional, you can't understand any byte sequence taken out of context - the full 'line' has to be read and fully understood (misremembering a param size makes you read all the rest of the line wrong). The possibility to have a character string as "string" in the NFO instead of hex bytes is one example of a small thing that gives much back (I don't know how 'small' it was to code, but as an example, OK?).

Any system that removes (part of) this positionality and makes it possible to understand subsequences is a Good Thing(tm) in my book.

That is not 'adding complexity', it is 'adding simplicity'.

The more of the NFO that can be written in 'plain text' the better, the important thing is to add those things in a consistent way, not overloading well known (for whom? programmers?) names and operators with a new meaning.