In-depth question about the Newgrf-format

[octo]

Hi everybody :)

I'm currently continuing ``pasky''s work of bringing newgrf-support to OpenTTD. I hope this is the right place to ask such a question. If not, please give me a hint as to where to ask :)

The problem lies withing Action 2, with variational yet deterministic cargo-IDs. The ``varadjust'' part is either 2 or 4 byte long. But I have no idea which one to assume. I could look at the total length of the sprite to see if it was 2 or 4 bytes long, but even then I can't distinguish between cases 2 and 3, because they are both 4 bytes long. My guess is that it depends on the variable being referenced to, but I couldn't find anything like that in the spec :/

Any hints are appreciated :)
Thanks,
-octo

[Csaboka]

The length depends on the first byte; the top two bits define which function the GRF wants: 00 means just shifting and ANDing (2 bytes), 01 means shifting, ANDing, adding a value then dividing (4 bytes), while 10 means shifting, ANDing, adding a value, then calculate modulus (4 bytes). The remaining six bits define the bit number for the shift, and six bits are enough because there's no point in shifting a 32-bit value more than 31 bits right.

[octo]

Thanks a lot, that helps a lot :)

In case I have further questions: Should I come back to here, or is there an IRC channel or a mailing list, where such questions should be asked instead of here?

[Csaboka]

I think it's the best to stay here. There's a ttdpatch-devel mailing list, but it's for patch developers only, so you won't be allowed to post unless you have a lots of questions and Joseph agrees as well. IRC, on the other hand, will work only if Joseph or me happens to be there, so you may have to lurk there for a while to catch us.

I have notification on this topic, so I can answer your questions fast if I'm at my computer.

[octo]

Okay, I have another question I thought I understood how the Actions 1, 2 and 3 interact, but apparently I am wrong.

The first question: In Action 2 there are ``Graphics Sets'' and ``Set ID lists''. And then there is talk about ``Cargo IDs''. I thought they were all more or less synonyms. Meaning, you have a ``Cargo ID'' which points to either a sprite set or another cargo id, and, additionally, can point to more cargo ids with some sort of mechanism to decide which one to use. However, I saw that the IDs given in the grf-file are anything but unique. Not even the IDs given for two type 0x81 lists are always unique. What's up with that? First, I thought that the ID only matters, till a mapping is defined. But that happends at the activation-stage, so this is apparently not the case. Then I thought simple CargoIDs which only point to a sprite set get an ID, so they can be referenced by a CargoID with a list, but the lists don't have unique IDs either, so I'm really lost..
Another thing is that in action two there is one byte for the ``Set ID'' and later on (in the list definitions) there are _two_ bytes to reference to a Set/CargoID/whatever. What exactly is there referenced and can I simply ignore the MSB?

The second question is much easier (to ask and to answer): When there is both: A wagon-override for the default cargo type, and a cargo override for the default engine type, which one do I choose?

Thanks in advance,
-octo

[Csaboka]

I think there's a misunderstanding here: Cargo IDs and sprite set IDs are two distinct things: cargo IDs get defined by action 2 while sprite set IDs always correspond to the sprites in the latest action 1. AFAIK there's no such case when a cargo ID needs to point to *one* other cargo ID, this wouldn't do anything useful. There are intermediate cargo ID definitions ( 80-83 ) that can be used to choose from a list of other IDs, and there are "final" definitions ( one type per each vehicle type, plus stations, canals and town buildings) that decide the sprite set to be used, and therefore refer to sprite set IDs, not cargo IDs.

About cargo IDs being two bytes: there are two reasons for that. The first (shorter) reason is that callback results must be distinguished from normal IDs to avoid confusing the two, so normal cargo IDs are words with the high byte being zero while callback results are words with 0xFF being in the high word. The second reason is the internal handling of cargo IDs: during the initialization stage, references to cargo IDs get replaced with the sprite number of the definition of the given ID. This allows faster resolving of references and redefining cargo IDs ( if you no longer need a definition with a given cargo ID, you can safely re-use the ID since previous definitions will still point to the old one), but needs two bytes since sprite numbers are word values in TTD.

As for the second question, I haven't ever heard of cargo override, and it's isn't mentioned in newgrf.txt either. I'm not a newgrf expert, I understood most of it just some weeks ago while writing newhouses, and this may be one of the parts I skipped. You should try PMing Patchman (if he haven't already noticed this topic), he should be able to answer you.

[octo]

Hi :) Thanks for your fast response :)

I think I got it right and just am having a hard time putting it into understandable words. I'll try again though: A mapping (action 3) always refers to a ``CargoID''. A CargoID either refers to the preceeding SpriteSet, or to a list of other CargoIDs.
A mapping uses the cargo-type and, if applicable, the pulling engine to determine which CargoID to use for drawing the vehicle.

I'm afraid I still didn't fully get how the ID stuff works out though. You can re-use an ID, if you don't need it anymore. Okay, but as far as I understand it, you need it till the mapping (action 3) is processed. However, all actions 1 and 2 are only carried out during initialization, many overwriting each other, and all actions 3 are carried out during activation, after all that overwriting has taken place. It'd make much more sense (for me anyways, maybe you know a good reason for doing it otherwise) if all actions 1, 2 and 3 were handled in the same phase. Then the mapping could happen right after the CargoID was defined, and after that the ID doesn't matter anymore.

[Csaboka]

Hi Thanks for your fast response

I think I got it right and just am having a hard time putting it into understandable words. I'll try again though: A mapping (action 3) always refers to a ``CargoID''. A CargoID either refers to the preceeding SpriteSet, or to a list of other CargoIDs.
A mapping uses the cargo-type and, if applicable, the pulling engine to determine which CargoID to use for drawing the vehicle.

Yes, that's correct.

I'm afraid I still didn't fully get how the ID stuff works out though. You can re-use an ID, if you don't need it anymore. Okay, but as far as I understand it, you need it till the mapping (action 3) is processed...

You don't need the IDs during activation since all of them were replaced by sprite numbers by then. When an action 3 is processed during activation, it'll see sprite numbers instead of the IDs and therefore won't care about what the IDs themselves were.

Maybe it'll get clearer if I tell you how Patchman implemented this:

He declared an array of 256 words, which contains the ID -> sprite number assigment. During initialization, it's cleared before processing a grf file. When initialization reaches an action 2, it'll record the sprite number of the action 2 to the according slot in the array. If it finds something that references a cargo ID, it'll substitute the ID with the sprite number stored in the array (except if the original ID has 0xFF in its high byte, meaning that it's a callback result, and should be left alone). This means that it always substitutes the sprite number of the _most recent_ definition of the given ID, and TTDPatch can totally forget about cargoIDs once the initialization is finished.