Handling of unexpected sprites (Split from OpenGFX)

[DaleStan]

What Raumkraut said about HTML.

377 Doesn't recognise legal action; does not attempt to process, does not issue error, skips to next line
378-383 (same as previous line)

What happens to this too-intelligent parser when it hits a recolor sprite and parses it as an action 0?
What happens when a bare real-sprite grows a defined meaning?

There is great safety in by-the-book parsing. And, with no incorrect common practices to accommodate, why encourage or reward anything other than strict conformance?

I think it's fine if it tries to skip things it doesn't understand to see if there's something else it does understand.

And it is also fine for the Intel 386 to skip instructions it doesn't understand (such as CMOVxx) in an attempt to find instructions that it does understand?

... html ...

Invalid example. NFO is not a markup language.
There's a reason NFO coders are called coders. They emit code. Not markup language. Markup can, to some extent, be fudged if something unknown, unexpected, or not to the users wishes is encountered. This is why <NOFRAMES> and <NOSCRIPT> were invented.
NFO code, like machine code, C, assembly, Java, and everything else in that class, cannot. If it's unexpected or unknown, you bail out.
You don't ignore the bytes 0F 0C (an undefined opcode on all known x86 processors[0]).
You don't ignore the call to malloc because the appropriate library isn't available. Nor do you assume the presence of a semicolon where one isn't.
You don't skip the pushad instruction because you're assembling for the 8086.
And you don't ignore sprites, or bytes, because you were expecting to see something else.

[0] Note that this is not UD2, the instruction that is defined to raise an undefined opcode exception, but is an opcode that is actually not defined.

[richk67]

Any chance of moving this NFO discussion off the 8bpp graphics topic.

[LordAzamath]

I already reported one post yesterday night so this discussion could be split off.. I'm sure mods will notice it soon

[FooBar]

... Do you really want to sentence GRF creators to the same hell as web developers have been living for the past 10 years?

Hell? I don't feel I live in a hell (yes, I'm a web developer too). The problem in webdeveloping is that browsers are incompliant.
In general: if the parser is compliant, but also accepts incompliant stuff, you don't have to go through hell as a developer as long as you stick to the rules.

From a programming perspective, the issue in question - extraneous sprites - should probably raise a warning (during compilation and/or interpretation), rather than an error or being ignored altogether.

I'd prefer a warning over an error any time, if processing of the code doesn't result in an immediate crash of the program. And yes, a warning is better than ignoring stuff, but I don't see why a program should terminate with an error message if it already knows how to ignore certain things (in this case bare real sprites).

What happens to this too-intelligent parser when it hits a recolor sprite and parses it as an action 0?

If that recolour sprite is in a place where a recolour sprite could be expected, it should recognise it as a recolour sprite and parse it as such. If the recolour sprite is in an invalid location, it could as well try to parse it as an action 0, coming to the conclusion that it's invalid.

What happens when a bare real-sprite grows a defined meaning?

If that happens, the parser needs to be changed in order to understand that new meaning. This means that the code fragment from my previous post can't be parsed properly. Which isn't a problem, because that fragment was invalid in the first place. The author just got lucky that it was accepted by some older parser.

There is great safety in by-the-book parsing.

Oh yes indeed. Everyting that's not in the book is invalid by default.

why encourage or reward anything other than strict conformance?

I don't consider that as a reward or encouragement, but just as getting lucky.

And it is also fine for the Intel 386 to skip instructions it doesn't understand (such as CMOVxx) in an attempt to find instructions that it does understand?

What else should it do? Explode? I don't think a processor can generate error messages all by itself. Maybe it could return an error code or something to the program from which it received the instructions. Then it's up to the program to handle that error, either by issueing a warning, but it could as well just ignore it if it's programmed to do so.

And it is also fine for the Intel 386 to skip instructions it doesn't understand (such as CMOVxx) in an attempt to find instructions that it does understand?

Invalid example. The Intel 386 is a piece of hardware while NFO is a software programming language.

Invalid example...something else.

Those parsers (probably) all parse by the book. If they were smarter than the book alone, they could as well attempt to understand what the programmer was trying to accomplish. If the results are unexpected, the programmer knows he (or she for that matter) made a mistake. It would be nice if the parser has some kind of debugging mode which then includes hints for the programmer on where to look to fix his code.

Also, please note that I don't have to be right. I'm just explaining my logic. I'm fine with things going by the book, but I also would appreciate a little user friendlyness. Let's say I need to develop a parser that can parse a date in yyyy/mm/dd format. I decide to make it user friendly to also accept dd/mm/yyyy dates. If I now feed a dd/mm/yyyy date to my parser, that date is in fact invalid according to the book, but it still gets accepted.
Now the openttd parser encounters some real sprites which shouldn't be there. It recognises the lines a real sprites, but knows those shouldn't be there. Now I really don't care if it skips those or marks the grf as invalid. The ones who do care should visit http://bugs.openttd.org/ and file a proper bugreport. I don't see it as a bug, but as a feature; others apparently think otherwise, which is perfectly fine, but that doesn't render my logic invalid. It's just my logic which differs from someone else's logic. The same way the TTDPatch parser differs from the OpenTTD parser.



Also I would like to apologise to all users who have seen the 'new post flag' raised on this topic the past few days just to discover something completely different. I'd appreciate if some moderator could split this discussion into a new topic.

[DaleStan]

If that recolour sprite is in a place where a recolour sprite could be expected ...

You missed the question completely.
An action 5 or A is skipped.
There follow some number of real sprites, and then a pseudo sprite that starts with the byte 00. What should the parser do with that sprite? If it is supposed to determine whether or not the sprite is a valid action 0, explain how it does this. If it uses "expectations", what are those expectations? If it uses both, what does it do if one says to parse it as an action 0 and the other to skip it as if it were a recolor sprite?

What happens when a bare real-sprite grows a defined meaning?

If that happens, the parser needs to be changed in order to understand that new meaning.

And previously accepted GRFs suddenly become invalid. This is *not* a feature. Cf. the racket when TTDPatch started enforcing that the feature bytes must match in 1/2/3 chain. Nevermind that the documentation has always said this is required.

I don't consider that as a reward or encouragement, but just as getting lucky.

What is getting lucky? Managing to count to eight?

And it is also fine for the Intel 386 to skip instructions it doesn't understand (such as CMOVxx) in an attempt to find instructions that it does understand?

What else should it do? Explode?

Yes. Well, not quite. It should, and does, raise a #UD (an undefined opcode exception). In the absence of something that can handle this -- usually a debugger -- this exception, like all other exceptions, terminates the program.

but it could as well just ignore it if it's programmed to do so.

And how does it know how many bytes to skip?

And it is also fine for the Intel 386 to skip instructions it doesn't understand (such as CMOVxx) in an attempt to find instructions that it does understand?

Invalid example. The Intel 386 is a piece of hardware while NFO is a software programming language.

Well, then, instead of "Intel 386" take "a software emulation of an Intel 386".
I'm comparing any generic hardware or software which behaves as an Intel 386 processor, and therefore interprets x86 machine code, to OpenTTD and TTDPatch, which interpret NFO code.

I also would appreciate a little user friendlyness.

I would too. Friendly to the *users*. This requires that NFO coders get beat over the head with as many errors as possible as early as possible, so that they get fixed before the users realize it's even possible for such errors to exist.
Or, if you want friendly to the coders, then I (and I doubt I'm alone here) want to know everything that could possibly be wrong with my code *now*. Actually, yesterday would be preferable, but now will work. This way I get to look all smart, rather than having my users complain that I released something broken. Fudging my code so it almost works unless I look at exactly the right/wrong location is *not* friendly. It's hell.

[FooBar]

You missed the question completely.

Sorry 'bout that, I either misread or misunderstood.

What should the parser do with that sprite?

Because it's not contained in an action 5 or A, it should attempt to process it as an action 0. It will fail in doing so. Now in your logic, the parser should issue an error. In my logic I'd say that the parser knows that the sprite in question is invalid. Now it could issue that error, but it could as well skip to the next sprite because it already knows the sprite is faulty.

And previously accepted GRFs suddenly become invalid.

I believe that every code ever written by the book should always be parsed correctly. If the code is not written by the book, the author will just get lucky if the code will be accepted, but it's his problem if it isn't accepted anymore in a new revision of the parser. Then he should've written the code by the book in the first place.

What is getting lucky?

Seeing something work what in theory shouldn't work.

And how does it know how many bytes to skip?

In case of NFO, the complete line. In other languages this might be different, but in most cases this would be "skip until encounter of semicolon not included in quotes".

*doesn't have anything to say on intel emulation*

if you want friendly to the coders, then I (and I doubt I'm alone here) want to know everything that could possibly be wrong with my code *now*.

I actually agree to that. Here comes the html example again. In case of html, I'm happy as soon als the w3c validator reports 0 errors AND the page displays correctly in both IE and Fx. In case of NewGRF I'm usually happy as soon as NFORenum reports 0 errors or warnings AND the grf works correctly in both TTDPatch and OpenTTD.

In case of OpenGFX, I'm happy as soon as I can see how things look ingame, I don't care about code correctness and compatibility with Patch [0] at this point because the project is still in development and the partial sets are only in the last place meant for "production environments". As soon as we'll get to a final state, code correctness and compatibility will be the important things, now it's just getting the sprites ready.


[0] Actually, I do care about compatibility with Patch, otherwise I would not bothered to created and release the bugfix version.

[DaleStan]

In my logic I'd say that the parser knows that the sprite in question is invalid.

How does it know that such a sprite is invalid? Is it assuming the sprite is invalid because the GRF is obviously invalid, or is there some other reason?

if you want friendly to the coders, then I (and I doubt I'm alone here) want to know everything that could possibly be wrong with my code *now*.

I actually agree to that. Here comes the html example again. In case of html, I'm happy as soon als the w3c validator reports 0 errors AND the page displays correctly in both IE and Fx.

But wouldn't it make you even happier if you knew that neither browser would automagically wrap the file in <HTML> tags? One more place that could catch one more stupid error.

Also, ...

[FooBar]

How does it know that such a sprite is invalid?

Quite simple. Let's take Patch as an example. NewGRF error code 1 means "A real graphics sprite was encountered where a pseudo sprite was expected ". So it already knows that there's a real sprite where it shouldn't be.
Now instead of displaying such an error message, Open ignores the sprite and proceeds with the next.

But wouldn't it make you even happier if you knew that neither browser would automagically wrap the file in <HTML> tags?

Don't care, because I try not to write invalid code. But it could help users who do not care about whether they are writing correct code or not.

Also, ...

Good! I didn't mind the old behaviour, but I don't mind this new either.
I guess somebody has been reading our little discussion

[ISA]

...............

[LordAzamath]

[offtopic]
Damn.. Can't you stay at least on topic in a topic which got split because off topic?
[/offtopic]

[DaleStan]

How does it know that such a sprite is invalid?

Quite simple. Let's take Patch as an example. NewGRF error code 1 means "A real graphics sprite was encountered where a pseudo sprite was expected ". So it already knows that there's a real sprite where it shouldn't be.

Weren't we talking about a recolor sprite/action 0?

[FooBar]

Right.

Ofcourse, at first it doesn't know and tries to parse it as a normal action 0. At which it will fail. Now the parser comes to a point where it could/should[0] issue an error because of the fact that it cannot parse the action. I don't know which error code Patch would return, but possibly it's 6 ("The given property number in action 0 is invalid"). Now, when a program (in general) is about to return an error code including the line on which the error occurred, it could as well not return the error code but resume processing from the next line.

[DaleStan]

Ofcourse, at first it doesn't know and tries to parse it as a normal action 0. At which it will fail.

That is rather unlikely. I ran all the recolor sprites in ttdpbase, ttdpbasew, trg1, trg1r, and newstatsw through NFORenum as action 0s. This is essentially all live recolor sprites. Of those 568 sprites, a whopping 9 (~1.6%) came out as invalid. The remainder are all by-the-book valid, though they may or may not make any actual changes.

[FooBar]

Let me ask you something...
Do you understand my logic by now?

Because in that case we maybe might discussing how a parser should work so that I can stop explaining my logic. Which might be flawed when it comes to parsers. I'm only trying to explain why I think that it could be possible for one of these to skip a line of code rather than report an error on it.


Anyways...

Ofcourse, at first it doesn't know and tries to parse it as a normal action 0. At which it will fail.

Well, maybe it should figure out first what kind of action 0 it is. As soon as it knows that it can tell whether it's legal for that type of action to be in that position. Now continue reading my previous post, replacing "cannot" with "shouldn't".

[DaleStan]

Let me ask you something...
Do you understand my logic by now?

What I understand is that you are under the impression that NFO is like HTML, and should therefore behave like HTML. It is not and it should not. HTML is supposed to be subject to the users' wishes (including such "wishes" as using links/lynx) first and the creator's wishes second. NFO is subject only to the creator's wishes. The only place the users' desires enter into the equation is whether or not the GRF file gets loaded.

Ofcourse, at first it doesn't know and tries to parse it as a normal action 0. At which it will fail.

Well, maybe it should figure out first what kind of action 0 it is. As soon as it knows that it can tell whether it's legal for that type of action to be in that position.

All actions 0-5, A, E, F, 11-13 are legal in all locations except for:
1) Within a data block.
2) Before the action 8.
All other actions are valid in all locations except "within a data block".
Action 8 has the additional restriction that exactly one must appear.
Action 11 has the additional restriction that at most one may appear.

Since both action 0 and all data blocks (where you would find a recolor sprite) are invalid before the action 8, we can assume that the suspect sprite is not before the action 8, which leaves only "within a data block". A data block is something introduced by an action 1/5/A/11/12. So, we are right back where we started: A sprite that is technically valid as both a recolor sprite and an action 0. The only differentiating factor between the two is context. Context that just finished telling you that the preceding sprite was invalid, and is now completely clueless, since it doesn't know what sort of data should follow an invalid sprite.

[FooBar]

Well, I can follow you there, but then try to explain why OpenTTD managed to skip a bunch of unexpected real sprites? Apearantly it wasn't utterly clouless about what to do next...

[DaleStan]

That's your job, I'm afraid. My case is that it should never have done that, and the fact that it did is an aberration.

[peter1138]

Well, I can follow you there, but then try to explain why OpenTTD managed to skip a bunch of unexpected real sprites? Apearantly it wasn't utterly clouless about what to do next...

It knows what a real sprite looks like as they don't start with 0xFF like pseudo sprites, and it needs to know how to skip expected sprites...

[ISA]

does it look like this?

As I once mentioned that Oil rig don't appear, the problem still exists! Other stuff works fine now! I'm using TTDPatch nightly 2020
What would You devs need more to fix the problem?

[FooBar]

While this is not quite the right topic to ask, I'll give you an answer anyways

Because OpenGFX' final format (replacements for trg*r.grf) is getting shape, the partial newgrfs are likely not to be supported any more. You also might want to try opengfx newindustries 0.10.1.