GRFCodec feature discussion thread

[DaleStan]

I thought about $ too, but I thought they were harder than necessary to produce on UK keyboards. It appears, however, that I was mistaken. Noted and changed.

[George]

Sorry, I'm a bit confused.
This page http://www.ttdpatch.net/grfcodec/ specifies "Get the most recent GRFCodec toolset, version 0.9.10", but files in the archive are dated
15/08/2006. Is that the latest version of the codec? If not, where can I download the latest version?

it does not understand \2sto and \2rst, while NFO renum generates them

[DaleStan]

That would be because NFORenum matches what's stored in Subversion, not what was compiled a few months before \2sto and \2rst were added. Search and replace is your friend, until I figure out a decent method of keeping the parsers from desyncing.

Does any one else machine-parse NFO? If so, my current thought is to add another comment between the "Info version" and "Format:" lines. The easiest way for me to do that would probably be to print a decimal count followed by printf(" %02x:%s",byte,string); for each escape (probably without the backslashes). To use standard RFC nomenclature, writers MUST generate a correct count, but readers MAY ignore it.

This would, of course, only be used for escapes that the hard-coded parser does not already know.

EDIT: No, before Format, not after.

[George]

That would be because NFORenum matches what's stored in Subversion, not what was compiled a few months before \2sto and \2rst were added. Search and replace is your friend,

about 10 times a hour or even more often Fortunately I use NFORenum together with my own parser, so it is only 2 strings of code

until I figure out a decent method of keeping the parsers from desyncing.
Does any one else machine-parse NFO? If so, my current thought is to add another comment after the "// Format:" line. The easiest way for me to do that would probably be to print a decimal count followed by printf(" %02x:%s",byte,string); for each escape (probably without the backslashes). To use standard RFC nomenclature, writers MUST generate a correct count, but readers MAY ignore it.
This would, of course, only be used for escapes that the hard-coded parser does not already know.

Can't you simply realize GRFCodec 0.9.11?

[DaleStan]

Can't you simply realize GRFCodec 0.9.11?

No, because it would break again. And again. And again. I want to fix this once and for all, rather than having to shovel new versions of both out every time a new escape sequence appears.

You can also turn off NFORenum's generation of escape sequences (which still won't make U+E0XX come out quoted.)

[George]

Can't you simply realize GRFCodec 0.9.11?

No, because it would break again. And again. And again. I want to fix this once and for all, rather than having to shovel new versions of both out every time a new escape sequence appears.
You can also turn off NFORenum's generation of escape sequences (which still won't make U+E0XX come out quoted.)

May be you could merge GRFcodec and NFOrenum into one package then?

[Wile E. Coyote]

I couldn't find that by search, so sorry if I'm asking something which is allready asked.

Is it possible to add binary format to GRF Codec? There are escape sequences for <dec>, <year>, <hex> and <date>. <bin> could be useful, for example, for properties which are set by bitmask, to avoid calculating, it'll be easier just to type binary number than to convert to hex and if some mistake is made, to convert to binary format again.

[DaleStan]

After more thought, although I can fix the operand and condition escapes, I wouldn't be able to fix any other escapes that way. This is an improvement, but not perfect.

I can put binary in, but I'm not sure how useful it is. You don't have to do much calculation to convert bitmasks to hex, and \wb1000100000000 is fraught with chances to type too many zeros.
For completeness, maybe I should add octal too (\<bwd>o<oct>) but I feel like that's not often used.

In any case, here's a GRFCodec that will read all current escape sequences. It'll also generate an extra three lines of header. An RFC-style specification for the NFO header is available on request. NFORenum will scramble the lines slightly, but will not break things; the former is a bug I intend to fix. I'm entirely thrilled with that line order for the header, so I've not committed the code; there's a patch instead.

[athanasios]

Thanks Dale.

[Wile E. Coyote]

Wow, what a speed! Dale is God!!! Thanx!
And yes, I think too there will be probably no use of octal format.

I'm probably boring, but I have one more idea (not sure is it allready on TODO list?), altough I don't know is it possible (it's probably too much complicated if it's possible): to make GRF Codec "smarter", i.e. to, for example, decode strings only where they are expected to appear (in Action 4, 8, etc. and not in Action 2 chains, now when they are decoded, lots of strange characters appear instead of normal hex numbers).

[DaleStan]

Wow, what a speed! Dale is God!!! Thanx!

I think there's a misunderstanding. Binary didn't go in. This is up to date as of just before your request. Sorry to be misleading.

I'm probably boring, but I have one more idea (not sure is it allready on TODO list?), altough I don't know is it possible (it's probably too much complicated if it's possible): to make GRF Codec "smarter", i.e. to, for example, decode strings only where they are expected to appear (in Action 4, 8, etc. and not in Action 2 chains, now when they are decoded, lots of strange characters appear instead of normal hex numbers).

It is not on the todo. That's what NFORenum is for. On the other hand NFORenum and GRFCodec are, more and more, starting to share code, so maybe it's time for the two to be merged. But I don't want to think about how hard this job would be.

[DaleStan]

This is similar to the above preview executable, but will correctly read the output from NFORenum r2100. See my latest post in the NFORenum thread for notes on that.

Also, I am still open to suggestions for the missing escapes. Because I've changed how they're parsed, longer escapes can be read with a sane quantity of work.

RPN in escapes is in the TODO. \<bwd>b (binary encoding) is unlikely unless patches for both NFORenum and GRFCodec are provided.

[Rubidium]

Attached is a build log with warnings and a diff to fix those warnings. The log is of a compile of GRFCodec r2103 by GCC 4.3 on an x86_64 machine on Linux (Debian/SID).

[planetmaker]

Hi,
when opening the depot and thus the purchase screen for trains OpenTTD 16254 issues the following line in the console:

Code: Select all

dbg: [sprite] Ignoring 12 unused extra bytes from the sprite from 2ccdj at position 168866

Code: Select all

renum -w 141 2ccdj.nfo | tee -a renum.log
echo Running GRFCodec. . .
grfcodec -c -e -p 2 2ccdj.grf

The resulting newgrf doesn't issue an error when compiled without parameter -c

I'm using grfcodec 2103E:

Code: Select all

~/ottd/grfdev/grfcodec> make clean && svn up && make grfcodec
rm -rf *.o *.os *.obj *.OBJ *.exe *.EXE *.map *.MAP *.bin grfmrg.ah
At revision 2112.
[CPP] grfcodec.o
grfcodec.cc:102: warning: deprecated conversion from string constant to ‘char*’
grfcodec.cc: In function ‘int main(int, char**)’:
grfcodec.cc:792: warning: deprecated conversion from string constant to ‘char*’
grfcodec.cc:813: warning: deprecated conversion from string constant to ‘char*’
grfcodec.cc:834: warning: deprecated conversion from string constant to ‘char*’
[CC] grfcomm.o
[CPP] pcxfile.o
pcxfile.cc: In member function ‘int pcxfile::subofsx(int, int)’:
pcxfile.cc:446: warning: suggest explicit braces to avoid ambiguous ‘else’
pcxfile.cc: In member function ‘int pcxfile::subofsy(int, int)’:
pcxfile.cc:458: warning: suggest explicit braces to avoid ambiguous ‘else’
[CPP] sprites.o
[CPP] pcxsprit.o
[CPP] info.o
info.cc: In member function ‘virtual void infowriter::newband(pcxfile*)’:
info.cc:278: warning: suggest parentheses around && within ||
[CC] error.o
[CC] getopt.o
[CC] path.o
[CPP] readinfo.o
[CPP] file.o
[LD] grfcodec
~/ottd/grfdev/grfcodec> grfcodec
-bash: /usr/local/bin/grfcodec: No such file or directory
~/ottd/grfdev/grfcodec> ./grfcodec
GRFCodec version 0.9.10 r2103E - Copyright (C) 2000-2005 by Josef Drexler
.
.
.
~/ottd/grfdev/grfcodec> svn diff
Index: Makefile.setup
===================================================================
--- Makefile.setup      (revision 2112)
+++ Makefile.setup      (working copy)
@@ -18,7 +18,7 @@
 # If the boost folder auto-detection fails, set your boost include folder here.
 # If boost can be found in your standard include search paths, setting this to
 # any non-empty value will cause make to assume that boost is present.
-#BOOST_INCLUDE = 
+BOOST_INCLUDE = /usr/include/boost 
 # If make is convinced that you do have boost installed when you do not,
 # uncomment the following line:
 #NO_BOOST = 1
@@ -126,7 +126,7 @@
 grfmerge.exe:  $(GRFMERGESRC:%.c=%.obj)
 
 clean:
-       rm -rf *.o *.os *.obj *.OBJ *.exe *.EXE *.map *.MAP *.bin grfmrg.ah
+       rm -rf *.d *.o *.os *.obj *.OBJ *.exe *.EXE *.map *.MAP *.bin grfmrg.ah
 
 remake: clean all

[DaleStan]

The NFO alone is not sufficient for me to watch GRFCodec break; I'll need the PCX for sure. Possibly one or both GRFs as well.

Also, it would be helpful if you could figure out the sprite number. I think I could convince GRFCodec to provide UI, but I'd might end up flailing wildly with a hex editor.

[planetmaker]

Oh, sorry, yes, the pcx are needed, of course, too. Attached is a version of the grf's archive.

With that version and OpenTTD 16254 I get

Code: Select all

dbg: [sprite] Ignoring 12 unused extra bytes from the sprite from 2ccdj at position 265148

UnfortunatelyI've no idea as to how to get an idea which sprite actually is responsible for that message and my search for such (grfcodec, grf2html) was so far not fruitful either.

EDIT: you can also obtain the grf via mercurial:
hg clone http://mz.openttdcoop.org/hg/2cctrainset
you'll need something like revision <89 or modify the apprpriate build script to include the -c again for grfcodec.

[frosch]

grf2html provides that info

[DaleStan]

grf2html provides that info

Thank you. I hadn't noticed that feature before, but it made my life far far easier.

Code: Select all

dbg: [sprite] Ignoring 12 unused extra bytes from the sprite from 2ccdj at position 265148

This should be fixed too (r2116).

EDIT: Also, the gcc 4.3 warnings have been silenced. Thanks, Rubidium.

[edorfaus]

Not sure this is the right place, but best I found anyway...

I wanted to use GRFCodec etc. on an amd64 box, but grfdiff wouldn't compile (nasm complained about elf not supporting 64-bit).

Looking into it, I found out that it must be told to use elf64 explicitly - and managed to make the NASMFORMAT var in the Makefile reflect this when the machine type is x86_64 (and only then, I believe).

This change lets it compile successfully, so I figured I'd tell you about it. (Wasn't sure where to do that though... This forum has far too many subforums and topics (not to mention posts) to go through :P )

Here's the svn diff:

Code: Select all

Index: Makefile
===================================================================
--- Makefile    (revision 2150)
+++ Makefile    (working copy)
@@ -6,7 +6,7 @@
 MAKEFILE = Makefile

 # OS dependent variables
-NASMFORMAT = $(shell [ \( $(ISCYGWIN) -eq 1 \) ] && echo coff || echo elf )
+NASMFORMAT = $(shell [ \( $(ISCYGWIN) -eq 1 \) ] && echo coff || [[ "`uname -m`" == "x86_64" ]] && echo elf64 || echo elf )
 REL_EXE = $(shell [ \( $(ISCYGWIN) -eq 1 \) ] && echo .exe )

 # Borland compiler settings

EDIT:
By the way - I noticed in the grfcodec output, and during searching have seen it mentioned, that pure white pixels are warned against... but I haven't been able to find any clear explanation for why they are considered to be a bad thing. Do you have any pointers or hints on that?

[DaleStan]

Not sure this is the right place, but best I found anyway...

I am not aware of a better place.

grfdiff and grfmerge are ancient and minimally supported, but it is nice to have them compile. However, it seems like everyone is complaining that nasm-doesn't-like-my-this, nasm-doesn't-like-my-that, nasm-doesn't-like-my-other-thing. I'm rather tempted to write a Perl script to turn it grfmrgc.bin into C and solve the problem once and for all.

By the way - I noticed in the grfcodec output, and during searching have seen it mentioned, that pure white pixels are warned against... but I haven't been able to find any clear explanation for why they are considered to be a bad thing.

1. If grfcodec is encoding pure white pixels then, 9 times out of 10, you typoed the real-sprite. I could make it more intelligent and only trigger on white pixels on the edge of the sprite, and/or only on full lines of white pixels, but ...
2. Pure white pixels do not fade out when TTD fades to black, leaving mostly-black with a few bright white dots

There is a visually identical #FCFCFC in the palette that I recommend you use instead.