newgrf translations

[Bjarni]

was splitted out of http://www.tt-forums.net/viewtopic.php?t=25681

A suggestion: How about if OpenTTD would recognize commonly used strings in newgrf files and then would get translations for them from the lang-files?

I wondered about that as well. Say strings like diesel, electric and steam (for locomotive names) are commonly used. However my idea was turned down with arguments like "then what if somebody decides to use a set with fuelcells or batteries" and "the newgrf codec should support translations by now. Not sure if any sets use that though"
I got feedback from a non-English talking person that UKRS was unplayable due to the fact that it only contains English strings.... we should consider a solution for this, either though the newgrf codec (and grf artists/coders) or some string overwriting... The main question is how to do this without spending ages on doing so as it appears to be a lot of work

[Patchman]

In principle, the best solution would probably be separate translation grfs. These would have to be loaded before the grf they're translating, and contain string definitions like "instead of the text for ID D402, use this", or possibly somehow set a flag that all strings defined by the file will override those from the given GRF file.

I'm open to suggestions for the specifications how this should work. One way would be an action D variable that's set to the GRFID of the grf we're overriding.

These translation files could also be automatically created by some program. I had something like this which still works for vehicle names, but that doesn't extend well to the newer features where text definitions are local to the grf file.

[Bjarni]

In principle, the best solution would probably be separate translation grfs. These would have to be loaded before the grf they're translating, and contain string definitions like "instead of the text for ID D402, use this", or possibly somehow set a flag that all strings defined by the file will override those from the given GRF file.

I'm open to suggestions for the specifications how this should work. One way would be an action D variable that's set to the GRFID of the grf we're overriding.

These translation files could also be automatically created by some program. I had something like this which still works for vehicle names, but that doesn't extend well to the newer features where text definitions are local to the grf file.

I just had a little chat with MiHaMiX about this and it should be possible to extend the web translator to also handle strings from grf files. The benefit for using it would be that it's easy to manage and we already got access to a lot of translators for a whole lot of languages.
Using one file for the grfset and another one for the translations will make the grf file work as a standalone (like now), while we make it easy to add new languages without actually touching the set itself (no reencoding needed).

The ideal solution would be if we make the web translator to encode the translation grf when it's done (or when a new language is done or something) and then upload it to a place where the grfset itself is available for download. That way people can download updated translations from the same location without having to download the set itself.

Now the question is how to get this to work. We need a code, that works on both TTDpatch and OpenTTD, with UTF-8 in mind (OpenTTD will eventually support this) and in a way so we can hopefully add it to all the grf sets out there, hopefully without too much work. We also need a way to tell the web translator what strings to translate. Either this can be done with extracting them from the grf file or somebody needs to write them in a text file and upload it. Also when it is right time to start translating a grf? Is it when it's done (might be a bit late) or when it's almost done or when it's new (with the risk that it will never be done and the translators waste time).

Clearly we got a lot to work out here, but I think we can get a good result from this. With good planning we should be able to get this to work

[mart3p]

In principle, the best solution would probably be separate translation grfs. These would have to be loaded before the grf they're translating, and contain string definitions like "instead of the text for ID D402, use this", or possibly somehow set a flag that all strings defined by the file will override those from the given GRF file.

Wouldn't it be easier to load the translation file after the GRF file. The GRF file would then load its text strings in the normal manner. The translation file, which would of course have a reference to the grfid of its target file, could then load the additional translated strings. Doing it this way round would mean an additional flag would not be needed.

We need a code, that works on both TTDpatch and OpenTTD, with UTF-8 in mind (OpenTTD will eventually support this)

peter1138's UTF-8 code already works for UTF-8 GRF text. Using the utf8 branch and my Cargo Subtypes patch I was able to load the Cyrillic text strings from Planeset 1.5 - see a screenshot here

[MeusH]

How about strings commonly used in various newgrf sets?
Like "freight" "tilting" "express" "suitable for" "special features"?

It would be nice if newgrf authors somehow unified these strings so translators of OpenTTD won't have to translate strings like that many times

[DaleStan]

It would be nice if newgrf authors somehow unified these strings

How? Those strings MUST be placed in the D000..D3FF range, and that range is GRF-specific; that is, each GRF can write indiscriminately, and no GRF can retrieve a text defined by another GRF.

[Patchman]

Wouldn't it be easier to load the translation file after the GRF file. The GRF file would then load its text strings in the normal manner. The translation file, which would of course have a reference to the grfid of its target file, could then load the additional translated strings. Doing it this way round would mean an additional flag would not be needed.

One flag will be needed regardless, because the translation grf needs to specify which grf it's translating. The D000/D800 range of text IDs are grf-specific, i.e. no grf can read or write another grf's strings. So it can't translate them without having a "hook" in the game code.

This can be done either by setting a global variable (using action D or action 0) or an entirely new action.

As for whether it should be loaded before or after the grf it's translating, ideally both should work to reduce the support costs when people try it the "wrong" way. Of course when done properly the translation grf can also generate error messages if it's loaded too early/too late.

I can make it work either way, both would be relatively easy to implement I think. If loaded before it overwrites the string data and if loaded after it overrides the text IDs (which needs some extra code for the D000/D800 range, but nothing terribly complicated).

[michael blunck]

How about strings commonly used in various newgrf sets?
Like "freight" "tilting" "express" "suitable for" "special features"?

Ah, the well-proven babblefish approach.

regards
Michael

[Bjarni]

We need a code, that works on both TTDpatch and OpenTTD, with UTF-8 in mind (OpenTTD will eventually support this)

peter1138's UTF-8 code already works for UTF-8 GRF text. Using the utf8 branch and my Cargo Subtypes patch I was able to load the Cyrillic text strings from Planeset 1.5 - see a screenshot here

good. One less problem to take care of. We are making progress

How about strings commonly used in various newgrf sets?
Like "freight" "tilting" "express" "suitable for" "special features"?

Ah, the well-proven babblefish approach.

I would go for this approach only if everything else fails. We got the ability to translate all strings, then why not use it?

[michael blunck]

> We got the ability to translate all strings [...]

Well, what means "all strings"? E.g., I´m using a max number of substrings in such a way that it works out for german, english, french and spanish, but this does´t necessarily mean it´ll fit every other language.

regards
Michael

[Bjarni]

> We got the ability to translate all strings [...]

Well, what means "all strings"? E.g., I´m using a max number of substrings in such a way that it works out for german, english, french and spanish, but this does´t necessarily mean it´ll fit every other language.

regards
Michael

what I meant was that with this approach, we should be able to translate all the strings in a grf set to all the languages we want to translate to. Currently that's not possible. Granted, it was "a bit" unclear

[belugas]

Currently, OTTD does not replace internal strings by those specified in grfs.
What we do instead is load and reference them by means of GRFID and StringID, associated with the current language used by the system.
It is not entirely finished, we still have to deal with duplicates. So, i think it can handle diverse loading schemes.

What would be good is for the translation grf to specify, if possible, the string id AS WELL as the grf id it is replacing,making it possible to "unload" previous delcaration. Or not to load it, if transaltion is run afterward

[DaleStan]

What is "StringID"?

The newgrf spec speaks of TextIDs, (though on the Action4 page it's called an "offset"[0]) which are word-sized integers. In grf files, most TextIDs are in the D0XX range, and the majority of those that aren't in D0XX are in the DCXX range. Are these the same thing?

In either case, a translation GRF file would almost certainly be required to specify:
1) Its GRFID. (All GRF files must have a GRFID)
2) The GRFID of the file it is translating for. (So the appropriate set of D0XX/DCXX IDs can be changed), and
3) The TextID for each string it is translating.

[0] There are byte-sized offsets, but these are not (AIUI) TextIDs, rather they are mapped to the appropriate 80XX TextID.

[belugas]

Hehehe
Time for clarification
Yes, TextID and StringID are exactly the same. I'm just used to the term StringID, since it is how we call it. Sorry for the confusion
Let's use the term TextID, as in grf spec, from now on.

In either case, a translation GRF file would almost certainly be required to specify:
1) Its GRFID. (All GRF files must have a GRFID)
2) The GRFID of the file it is translating for. (So the appropriate set of D0XX/DCXX IDs can be changed), and
3) The TextID for each string it is translating.

On this, I do agree 100%, although the range of IDs are not much of an importance for OTTD.

[DaleStan]

I don't really care which term we use; I just wanted to make sure that "StringID" and "TextID" did indeed refer to the same thing.

I don't know how the Webtranslator works, but I do know that while constructs such as {P "" s} sort-of exist in the newgrf spec[0], {P lo la el} and its ilk do not. I'm also not quite sure how positional arguments would work. Code 9A may find itself some new work rather soon.

[0] TTD almost always provides two TextIDs, one for singular and another for plural.

[Bjarni]

I don't know how the Webtranslator works, but I do know that while constructs such as {P "" s} sort-of exist in the newgrf spec[0], {P lo la el} and its ilk do not. I'm also not quite sure how positional arguments would work. Code 9A may find itself some new work rather soon.

currently it works by reading the txt files in svn in our svn system. It will then declare english.txt as the list of strings, that should be in all translations. A translator will then get a list of missing strings in the language he signed up to translate (say in german.txt). That list displays a text field together with each line and the translator can fill it with the translated string with all the options we got for say {P "" s} and such. It will then be verified (are all {} correct} and then they will be stored to be committed next time all newly translated strings are committed to svn.
Since MiHaMiX wrote the whole translator, we got 100% of the source and we can add the features to it that we need. I guess you can say that it takes a txt file as input and one as output. We can then add encoders and decoders to it as needed.
I hope this explains how it works. Basically it need some definitions for encoding and decoding so it get the strings as plain text to work with.

[Patchman]

I think it would be nice if one grf could translate several other files, or all common ones, so that you don't need to load 50 translation files. For this we would need a new action 13, but that'd be the cleanest way to implement this anyway, so maybe that's the way to go.

Something like action 4, except specifying the GRFID it's translating instead of the languages the texts are for. Easier to implement would probably be if there's a matching action 13 for each action 4 in the original grf, instead of going by text IDs. However that means that if the original grf changes, the translations will break.

I suggest the following then:

Code: Select all

<spritenum> * <len>  13 <feature> <grfid> <num-ent> <id> <strings...>

If <feature> is less than 4, the <id> is a byte indicating the vehicle type ID, otherwise it's a word indicating the first text ID to translate. This is not entirely congruent with how the feature is used in action 4 however, where the byte/wordness of the ID is indicated by bit 7 of the language byte instead. We can of course add another byte like the language byte, where bit 7 does the same. <grfid> is the GRFID of the set to translate. Another alternative would be setting grfid=FFFFFFFF when translating vehicle names, since those are not grf-specific, instead of using the feature byte or a separate byte for it.

[DaleStan]

There are definitely times that a D0XX text is required for features 00..03, so I'd prefer using something other than the feature byte for determining byte/wordness. Using the GRFID sounds fine to me.

And Bjarni, you mostly answered my question, but I guess I asked the wrong one. I was wondering how the translator would deal with the fact that you can only have one plural form in GRF texts. Or would it just not deal with that, and leave it up to the encoder (some strgen-like program) to drop extra plural forms?

[belugas]

There are definitely times that a D0XX text is required for features 00..03, so I'd prefer using something other than the feature byte for determining byte/wordness. Using the GRFID sounds fine to me.

I agree with that. Otherwise, one day or the other, we will hit a limitation wall.

...I was wondering how the translator would deal with the fact that you can only have one plural form in GRF texts.

Good question. It could be handled by means of a default plural format values, included in the lang file of OTTD, within some PRAGMA section (kind of). This way, translation grf would not need to be touched by that problem, since it is really an OTTD "feature".

[Bjarni]

And Bjarni, you mostly answered my question, but I guess I asked the wrong one. I was wondering how the translator would deal with the fact that you can only have one plural form in GRF texts. Or would it just not deal with that, and leave it up to the encoder (some strgen-like program) to drop extra plural forms?

if there is a problem, then we can add conditions to the string validation, say some conditions are only valid for OpenTTD and some are only for grf files, if needed. This should not be a showstopper.