Increasing number of vehicules ID in the GRF definition

[MagicBuzz]

Hello,

First of all, let me say I don't know much things about the GRF format.
The only thing I know is that the TTDPatch team is the format "owner", and OTTD have to match with it.

But in OTTD, the current limitation of 256 IDs is quite frustrating, as we can easily increase it in the OTTD source code.

Right now, as I read along the forums, only one set of vehicules can be used at the same time for a certain type of vehicules.
Because of this, we can't have big sets of vehicules, nor using by exemple URKS set and DB set together.

So I come after participating to this thread in the OTTD forum :
http://www.tt-forums.net/viewtopic.php?f=32&t=33124

We would like for the future, the GRF allows us to use bigger vehicule ID, like 16 bits instead of 8.
It might need a GRF version bump.

The goal of this change is to let TTDPatch running "like now" with extended sets, but allow OTTD to use a wider set of options.

I think about this :

We use 2 bytes.
byte1 will contain mostly flags and some bits to extend the ID range values, while byte2 contain the "base" vehicule ID.

Let's see in detail :

byte1 = 00 / byte2 = xx :
Backward compatibility :
Let's use the current rules using the byte2 values as vehicules IDs, a new installed set will override any "standard" vehicules set (or it won't load, i don't know how it works currently)

byte1 > 00 / byte2 = xx :
If extended vehicule ID is not supported (TTDPatch until the vehicule ID can change to 16 bits) :
Forward compatibility :
byte1 bit 1 set to 1 : this set will override any loaded standard set (with current rules, using byte2 as vehicule ID, according to the byte1 bits 2-5 -vehicule type- matching the default vehicule type table http://wiki.ttdpatch.net/tiki-index.php ... VehicleIDs)
byte1 bit 1 set to 0 : this set will be ignored

Feature :
If the new system is supported :
byte1 bits 2-5 = flag indicating the vehicule type : 4 bits = up to 16 different vehicule types (road, rail, air, sea, tram, ereail, etc.) that allow creating easily new types in the future.
byte1 bits 6-8 + byte2 = extended vehicule id : 11 bits = up to 2048 different vehicules (for each of the 16 different types, so a GRF could contains up to 32k vehicules IDs plus the 256 base IDs)

How does the extended vehicules ID might work :
- If an extended vehicule ID duplicate if found for the same vehicule type, override the previous vehicule with this one (allow to make GRF fixes)
- Otherwise, just add the vehicule to the already loaded sets


How do you think about it ?

PS : This is just a draft, as I said, I don't know much how works the GRFs, some information in my post might be wrong.

[Edit] Some parts were not clear at all [/Edit]

[Villem]

Changing things in TTDPatch is not that straight forward...TTDP did not decide the ID Limit, its a inheritance from TTD.
To change that limit, i'd imagine you would need to patch incredible amounts of things cause it hasn't been done yeat, and i think this is not the first time something like this has been suggested. TTDP developers can't just whip out their magical wands, and wave them around to increase a limit..

Theres a tredenmous amount of time spent on working out how changing that limit is even possible, then it takes incredible amount of time, and effort to figure out how to get rid of all the bugs related... I'd imagine, if this thing was easy to do, it would been done already..

[Zephyris]

I think the idea is to extend the newgrf format to allow openttd to take advantage of more vehicles whilst still allowing compatibility with ttdpatch. Patching openttd to allow more vehicles is far more plausible than ttdpatch...

[DaleStan]

Changing the NFO spec is *easy*. Changing the NFO spec while maintaining backwards compatibility is only minimally harder.
Changing TTDPatch to do more than just parse and ignore such changes is the hard part.

Until that last happens, such features aren't apt to get much use, since most of the coders code with Patch as the primary target, and OpenTTD as an if-it-supports-my-code target.

[MagicBuzz]

I think the idea is to extend the newgrf format to allow openttd to take advantage of more vehicles whilst still allowing compatibility with ttdpatch. Patching openttd to allow more vehicles is far more plausible than ttdpatch...

That's exactly what I think about.

I know TTDPatch can't change the vehicule ID "like this", as it would need a lot of rewrite.
But as OTTD can use it, it should be nice that the GRF format actually allow us to use more IDs, while the backward compatibility with TTDPatch is granted.

What I detailed in the first post may just be coded in several hours in TTDPatch, as it would just change the way TTDPatch loads the GRF, not how it uses it : byte1 = 00 for "basic" GRFs (feature disabled) and first 5th bits of byte1 matched with the byte2 value for extended GRFs with backward compatibility.

Exemple : let's say "train" vehicule type is 0000 and a new vehicule type "flying saucer" is 1111

It byte1 = 00000000 then TTDPatch will load the vehicule using byte2 value for ID.
If byte1 = 00000xxx then TTDPatch won't load the vehicule.
If byte1 = 10000xxx then TTDPatch will load the vehicule as a train, using byte2 value for ID.*
If byte1 = 11111xxx then TTDPach will detect it can't hande flying sauver vehicule, and ignore it.

* : The GRF loader will have to ensure the ID value match the train type (if like I think, the vehicule type depends of the ID value). If not, it will ignore it.

[DaleStan]

What I detailed in the first post may just be coded in several hours in TTDPatch

No, it can't. Not until you, at the very least, specify the canon names for all the NFO entities whose definition you are changing. And not until provide a new format for those entities that is a format spec, not a bunch of examples. Go read the NFO spec. That level of detail is what is required. If examples are necessary, your spec is insufficiently specified.

[MagicBuzz]

Ok, I just took a look at this :
http://wiki.ttdpatch.net/tiki-index.php?page=Action0

Please don't flame me if I'm wrong, as I just don't have any experience with the GRF...

If I corretly understand this page, the EngineID can be an extended byte, ie if it's FF, then the actual value is the following word.

The only problem with this, is that I'm not sure we can use it for current vehicule features, because there is a default vehicule with ID "FF".

If action 0 vehicules features are hardcoded to read a B regardless the FF value (ie. FF isn't FFFF00 in the GRF (0xFF followed by 0x00FF with little endien format) file for action 0 / features 0, 1, 2 and 4)
If action 0, features 0, 1, 2 and 3 read a byte from the engine ID regardless the FF value (don't use the extended system) then there is a need to create a new action, let's say action "10".
This new action will support features 0, 1, 2 and 3 exactely as the action 0 does with the updates bellow)

If action 0 vehicules features already recognize B*
No need to create a new action, as the current action 0 features can handle the word based ID.

Now, we have a way to read a 16 bits based engine ID.
So we just have to use simplified rules :

Backward compatibility :
If ID <= 255 then treat this vehicule as a standard vehicule : read the byte2 as a engine ID.

Forward compatibility (GRF using extended IDs that TTDP should ignore -and if action0 handle 16 bits engine IDs-) :
If ID > 255 just ignore the vehicule and jump to the next action. It should be easy to do, as the "features 0, 1, 2, 3" won't change, they remain "as is" (then you just have to read the data but not store them to jump the the next action)

Feature (OTTD) : store the read engine ID as a word and use it as vechicule ID

Is that more clear ? If this works like I understand, then it would take only a few seconds to handle correctly this new feature

[Edit] In fact, TTDP just have to read the engine ID into a word, then check it's below or egal to 255 before converting it to a byte.

[MagicBuzz]

Damned... :/

Action 0 accepts a B* for vehicule ID.
But if I correctly understand the GRF tutorial, Action 3 that refers also to a vehicule ID accepts only B... (not B*)

If so, then there is still a need to bump GRF version. But the update isn't big : just change any engine ID "B" into a "B*", that's all.

[Csaboka]

That B* isn't for vehicles in action0; it's for sound effects. GRFs can define more than 255 sound effects, so you may need to refer to IDs above 255. I don't know how hard it would be to extend that B* to vehicle IDs, that's Patchman's area of expertise, I think.

[MagicBuzz]

That B* isn't for vehicles in action0; it's for sound effects. GRFs can define more than 255 sound effects, so you may need to refer to IDs above 255. I don't know how hard it would be to extend that B* to vehicle IDs, that's Patchman's area of expertise, I think.

Ok, so as I thought, there is a need to bump the version. In order to handle properly the B* for vehicule IDs. But I don't think it should be hard to implement in TTDP : just adding the B* ability for these actions, then check the value is bellow 256 before using it.

PS : As said since the begining of the topic, all I ask for here is a new GRF spec about how to handle "extended vehicules ID". Not any change of TTDP. Currently, this feature seems to be easy to change in OTTD, so we should introduce it. TTDP should just ignore these extended IDs as I suggested, like OTTD does with all unsupported features

[MagicBuzz]

Some news about the project in OTTD :
http://www.tt-forums.net/viewtopic.php?p=609666#p609666


I "finished" the basic part of the project : allowing OTTD to use engine IDs > 255
(In fact, I did only basic tests, you can be sure it's very buggy )

I'll now wait for the decision about adding the support of this in the GRF format (B* for engine IDs)

[Lakie]

Why do you need more than 256 vehicle type ids? Unless you have a set which isn't really usable or a lot of trains which won't be used.
I don't really see why it is necessary to break TTDpatch and OpenTTD vehicle newgrf compatibly because a poorly coded / chosen grf wants to use more than vehicle types.

Well, by that I mean sets could no longer work with TTDpatch as vehicle types would be lost (most likely to be the wagons), and don't forget vehicle ids from 0x100 up are special effects and other special classes...
Also a lot of code assumes certain ids for a certain class (ship, plane etc.) is within a certain vehicle type id range...

By Vehicle ids, I mean Vehicle Type Ids, incase anyone got confused...
Although, my understanding is basic, I'll let another dev correct me over these things.

~ Lakie

[PikkaBird]

Why do you need more than 256 vehicle type ids? Unless you have a set which isn't really usable or a lot of trains which won't be used.

I can't really understand the logic behind it either, especially not if the reason for doing it is to allow multiple train sets, etc, to be used at once; the different sets are, well, too different. However, it would be nice to have more aircraft and ship IDs to play with. Perhaps a more interesting feature would be the possibility of "porting" IDs from one vehicle type (trains, RVs) to another (ships, aircraft)?

[MagicBuzz]

My background idea is not to allow multiple sets to be used at the same time, but using sets without the current limitation of vehicules numbers.

This might allow more wagon types by exemple, as currently, if you design low speed wagons and high speed wagons, you won't be able to design middle speed wagons without compromising the number of engines.
Allowing more vehicules ID may allow us to get different wagons for different rails types also.

This could also allow to get in one GRF complete sets for multiples climates, etc.

In my tests, I duplicate the basic vehicules, but you have to keep in mind all "standard vehicules" with ID > 255 won't be visible in the release version. Only GRF will add usable vehicules.

And at last, it WON'T break the compatibility between TTDP and OTTD GRF files. I'm currently waiting to implement extended ID in the GRF reader of OTTD, till the new spec isn't validated. OTTD doesn't support many GRF things, I don't see what bugs you if OTTD supports one feature TTDP doesn't.

Any higher ID vehicule should be simply ignored by TTDP, but all lower ID vehicules will load as now. It's to the GRF author to take care his set is still usable with only lower ID vehicules.

[AndersI]

Wouldn't the easiest solution for OTTD be to have the possibility to add an ID offset (or 'page number') when loading a GRF (swtrains.grf:01 for example)? Then you could load 256 sets without ID collision, just by letting the user specify the offset in his newgrf.cfg (or whatever it's called in OTTD). Any line without the extra offset would load the GRF into 'page zero', working just as today. The vehicle ID (and all code that uses it) and the GRF loader would be the only parts that needed changing.

A bonus with this approach is that the NFO/GRF standard is not changed, only the configuration file.

Problems with this approach may be:

- a GRF can only check for other GRFs on the same page, not all loaded GRFs
- a GRF that consciously overwrites another set - for example a GRF that only changes the base year of some vehicles. The user must then be aware of this fact, and add the same offset for both of them.

[MagicBuzz]

Wouldn't the easiest solution for OTTD be to have the possibility to add an ID offset (or 'page number') when loading a GRF (swtrains.grf:01 for example)? Then you could load 256 sets without ID collision, just by letting the user specify the offset in his newgrf.cfg (or whatever it's called in OTTD). Any line without the extra offset would load the GRF into 'page zero', working just as today. The vehicle ID (and all code that uses it) and the GRF loader would be the only parts that needed changing.

A bonus with this approach is that the NFO/GRF standard is not changed, only the configuration file.

Problems with this approach may be:

- a GRF can only check for other GRFs on the same page, not all loaded GRFs
- a GRF that consciously overwrites another set - for example a GRF that only changes the base year of some vehicles. The user must then be aware of this fact, and add the same offset for both of them.

That's exactelly what I worked on today. I don't think it's a definitive solution, but a good alternative before the GRF accepts higher IDs values : with some care, we can obtain the same result.
Everything is ready in my patch except... the offset support when reading the GRF. I really need some help about it, as I can't understand anything in the GRF loader code
But I alreay did some tests by mixing different GRF (in fact, base GRF with 256-511 IDs and additional GRFs using the standard range) and it works great.
The only problemis that some GRFs change some global variable (base cost for exemple) then they are applied to the whole game. It need some work to change this.

[AndersI]

The only problemis that some GRFs change some global variable (base cost for exemple) then they are applied to the whole game. It need some work to change this.

Maybe only let page zero GRFs modify globals? One more thing for the user to remember, but might work.

[Bilbo]

The only problemis that some GRFs change some global variable (base cost for exemple) then they are applied to the whole game. It need some work to change this.

Maybe only let page zero GRFs modify globals? One more thing for the user to remember, but might work.

Not possible, since then some vehicles would be very expensive or very cheap. Some sets use this to modify base price, as GRF's contain the price only relative to base price.

[Maedhros]

A much better way would be to have a base-cost for each page, though that may not be desperately easy to code.

[Bilbo]

A much better way would be to have a base-cost for each page, though that may not be desperately easy to code.

Yes, all base costs (cost and running cost for all vehicle types), base costs for bridges if they are included (though slotting for bridges could be done later) .... perhaps more global variables are important, you have to check grf specs

The really global costs (terraforming, etc ...) should then be modifiable only by grf in slot 0