Fileformat for object descrepion data: tab-files, XML, ...

Archived discussions related to Transport Empire. Read-only access only.

Moderator: Transport Empire Moderators

Locked
User avatar
Zuu
OpenTTD Developer
OpenTTD Developer
Posts: 4553
Joined: 09 Jun 2003 18:21
Location: /home/sweden

Fileformat for object descrepion data: tab-files, XML, ...

Post by Zuu »

Topic: Fileformat for object descrepion data: tab-files, XML, ...


Recently aarona brought this topic up again as a one of many subjects in this thread.

This thread is an atempt from my side to collect what have been said to form a good ground for futher discussion. Also the thread aarona created wasn't aimed for discussing the topics in that he/she brought up.



Old thread from 2004-07-14
oliver wrote:Here are my fileformat suggestions and reasoning. This list is not yet complete.

[ ... ]

Other file's should be normal unix style text files (Linefeeds only not carriage returns)
Map files could be XML if really required.

All these points are open for discussion naturally but I think we got a descent list with open standards here that should pose a problem.
jfs wrote:
oliver wrote:Other file's should be normal unix style text files (Linefeeds only not carriage returns)
It shouldn't be too hard to make support for both DOS, UNIX and Mac style line endings, and if all three kinds were supported it would be easier for the developers and hackers, I think. Eventually the files could be converted at load time, so all text internally is stored with UNIX line endings.
ChrisCF wrote:TBH, the style of line-ending shouldn't really come into it. what's the problem with using UNIX style endings (\n only) on Mac OS, or DOS style (\r\n) on *nix? Sensible text editors support all three, and a myriad of others. Oh, and the encoding will be UTF-8, without question.

As has been discussed before, we should ideally try and write as little code for these things as possible, making use of libraries instead. Somewhere in much earlier discussion, it was raised that with some libraries we can have an XML parser out-of-the-box, exactly as we can have the insides of a Vorbis player - the joys of well-defined standards :)
oliver wrote:
ChrisCF wrote:TBH, the style of line-ending shouldn't really come into it. what's the problem with using UNIX style endings (\n only) on Mac OS, or DOS style (\r\n) on *nix? Sensible text editors support all three, and a myriad of others. Oh, and the encoding will be UTF-8, without question.
Personal I prefer ASCII encoding cos it's the most basic you can get, But i think UTF-8 is quite compiatble with ASCII isn't it. I have to read into that.

Fileformats internally should be \n is what I ment and should be the preferred format. We should try to keep 'our' archive clean this way and not mix and match. same goes for C files etc. (On a side note, does gcc even take UTF-8 chars as input?) that the engine works with both formats I guess shouldn't be a problem, eventually anyway, but we gotta keep things clean and concistent in our code stash.
ChrisCF wrote:As has been discussed before, we should ideally try and write as little code for these things as possible, making use of libraries instead. Somewhere in much earlier discussion, it was raised that with some libraries we can have an XML parser out-of-the-box, exactly as we can have the insides of a Vorbis player - the joys of well-defined standards :)
I agree that we should use as many _standard_ widley availble libraries as possible, but still keep the amount to a minium. Like not implementing 1000x features and use 10000x libs and go through library hell. Just because we write lesser code doesn't mean the game will then be less efficient. But I think I mentioned XML in my first post and I was reffering to using the XML(2) library for that yes.
ChrisCF wrote:
oliver wrote:Personal I prefer ASCII encoding cos it's the most basic you can get, But i think UTF-8 is quite compiatble with ASCII isn't it. I have to read into that.
It's compatible with ASCII, since ASCII is a 7-bit encoding. It's only when you introduce other things like the ISO-8859 range, which have no less than 15 (possibly more) different uses of the "extended ASCII" (8-bit range). It would only come into play in any files which contain strings (and we should use strings, not references).
but we gotta keep things clean and concistent in our code stash.
Fair enough, in which case we need to decide on \n or \r\n now (though TBH I don't see the savings being that much using \n).

As for the issue of XML, my opinion is that we'd be better off with something like libxml2, rather than trying to implement everything from scratch, and finding too many bugs.
I'm not going to quote more stuff about \n vs \r\n vs \r. If you are interested to read the rest of these posts check the orginal thread starting from here.
oliver wrote:If we go with xml, then Yeah i'd say deffinatly use libxml2 and for other things goes the same.


Old thread from 2005-04-14
jfs wrote:What's most interesting though, IMO, is object description formats. (Eg. how to describe a new vehicle to be available in the game.)
Many people immediately scream XML when it comes to this, but I think the most flexible solution would really be to use "scripted data files" for object descriptions! This is also one of the reasons I think a hybrid compiled/scripted language solution is good.
I think most people here have an idea of what can be done with GRF files in the TTDPatch alpha versions, and I'm sure everybody would want TE to be able to do the same (and more?), but in a much easier way Smile So an "object description script" would really consist of a number of functions which are called by the game engine whenever something specific happens. Eg. there is a function that fills a vehicle data structure when a new vehicle of this type is built, one that's called whenever a vehicle of this type is (attempted) added to a "vehicle chain" (eg. a train or a trailer to a lorry) and likewise one for removal, one when loading-status changes etc. etc. That would basically give ultimate freedom for vehicle designers. Also, for basic needs, a "template" could be supplied, so even non-programmers could easily create vehicles (though not as sophisticated ones) for the game.
Zuu wrote:
jfs wrote:What's most interesting though, IMO, is object description formats. (Eg. how to describe a new vehicle to be available in the game.)
Many people immediately scream XML when it comes to this, but I think the most flexible solution would really be to use "scripted data files" for object descriptions! This is also one of the reasons I think a hybrid compiled/scripted language solution is good.
I think most people here have an idea of what can be done with GRF files in the TTDPatch alpha versions, and I'm sure everybody would want TE to be able to do the same (and more?), but in a much easier way :) So an "object description script" would really consist of a number of functions which are called by the game engine whenever something specific happens. Eg. there is a function that fills a vehicle data structure when a new vehicle of this type is built, one that's called whenever a vehicle of this type is (attempted) added to a "vehicle chain" (eg. a train or a trailer to a lorry) and likewise one for removal, one when loading-status changes etc. etc. That would basically give ultimate freedom for vehicle designers. Also, for basic needs, a "template" could be supplied, so even non-programmers could easily create vehicles (though not as sophisticated ones) for the game.
As a refference I have uploaded init_object.lua which defines all objects (which aren't that many ATM) in Junctioneer and LuaC.cpp which contains the C++ functions that is called from lua. The whole package can be obtained at http://www.sf.net/projects/junctioneer/ .
Se the orginal post for the attachments
jfs wrote:On the scripting-question, I know Lua can be cumbersome to work in, but the API is still rather clean. I think there are some nicer C++ wrappers for it as well. (I wonder if a C++ wrapper can somehow be made with templates?)

That's why I also suggested looking at Python. I'm looking at its C/C++ API right now, and it seems to be nicer than Lua's at a first glance. I'll post some more after I've read most of the stuff there is in the docs :)


Edit:
Another thing, I think we should use a virtual filesystem approach to data files. So you can either have all files for eg. a vehicle inside a single packed file, or as separate files on the host filesystem. The engine shouldn't care about that. The only problem is what to do in case of a name clash (several files with the same path and name.)

Also, I've looked a bit more on the Python C API, and it does indeed seem rather nice. You don't need to do all that messing around with a stack like in Lua, but on the other hand, I'm not sure I have discovered how to properly call a function yet :P

Edit again:
Found it now. There's several ways of calling a function/method, but on of them is:

Code: Select all

PyObject* PyObject_CallFunction(	PyObject *callable, char *format, ...)
Now, the really good thing here is the "format" and "..." arguments, you simply specify a kind of "format string" for the arguments the function takes, and then gives the arguments. It can automatically do conversion from basic C types and such. Functions can more or less return multiply values, in the sense that they can return a single touple that contains multiple values. Touples are also rather easy to deconstruct, I think.

Right now, I say we go for Python rather than Lua. Also, I think the "tables" and "meta-tables" concepts in Lua are confusing; Python seems to be more "normal" in most aspects.


New thread form Thursday this week ( 2006-06-08 )
aarona wrote:2. XML config files
--- Although the Design Documentation (ver 0.1.03 - Section 1.3) states we should use "Spreadsheets" or "Tab delimited files", I would strongly suggest useage of XML as our config file standard.
Here is some sample code which shows how easy it is to use XML provided you know all the things inside <> and ""'s, which would be provided in examples or documentation.

Code: Select all

<vehicle="Train">Choo-Choo
	<type>Steam</type>
	<speed>100</speed>
</vehicle>
Suggestion: New thread (or continue the one from years ago...), poll, demonstration files
eis_os wrote:
aarona wrote: 2. XML config files

Code: Select all

<vehicle="Train">Choo-Choo
	<type>Steam</type>
	<speed>100</speed>
</vehicle>
That really depends on how the engine works and how much different logic a designer will be able to do. XML can get quite complicate if you want to have things like variational cargo display, ages or other stuff selecting different sprites. And you should create a tag for the name of the object and not mix it.

Code: Select all

<name>Choo-Choo Patched</name>
<description>A patched Choo Choo Engine</description>
:wink:
aarona wrote:
eis_os wrote:XML can get quite complicate if you want to have things like variational cargo display, ages or other stuff selecting different sprites.
I agree, the point really is that it has to be done some way, so I believe
(from a developers point of view) XML would be *much* easier to implement over making your own. (Unless someone more motivated to spend time on a paser is willing to put their hand up :!: ) It seems to me with regards to TTD-patch that people are either coders or graphics artists (exceptions, of course, do exist :wink:) so I'm hoping that with TE we can have graphics artists who can do their own coding without having to learn too much.
XeryusTC wrote:2. XML works great if you want more programs to parse it, but most parsers are really slow. It would be better to just have your own format if only your application has to read it because it will probably be A LOT faster.


So, what to do from here?
aarona wrote:Suggestion: New thread (or continue the one from years ago...), poll, demonstration files
So more discussion and elaboration if needed, and then a poll have to be set up.
My OpenTTD contributions (AIs, Game Scripts, patches, OpenTTD Auto Updater, and some sprites)
Junctioneer (a traffic intersection simulator)
User avatar
aarona
Traffic Manager
Traffic Manager
Posts: 221
Joined: 26 May 2006 15:54
Location: Perth, Australia
Contact:

Post by aarona »

Does anybody have anything to add?

Zuu has kindly compiled a list of previous comments on this subject, so perhaps we all have enough information now to make an informed decision?

If this is the case perhaps we should have a vote on this matter at the next meeting.
User avatar
Hyronymus
Tycoon
Tycoon
Posts: 13233
Joined: 03 Dec 2002 10:36
Location: The Netherlands
Contact:

Post by Hyronymus »

Voting on it sounds like a good thing to do. If noone opposes I'll add this to the meeting agenda.
User avatar
Purno
Tycoon
Tycoon
Posts: 16659
Joined: 30 Mar 2004 12:30
Location: Almere, The Netherlands

Post by Purno »

Hmm... I still got no good overview on how different 'codes' would look like. I only saw the XML thingy which looks quite well, but I can't compare it with other thingies.

I perhaps could suggest something like the Red Alert rules.ini file, which kinda would look like this;

Code: Select all

[ENGINES]
1 = TRAIN01
2 = TRAIN02

;Name - Name which appears at the depot list (default: Same as tag)
;Year - Year the engine will become available (default: 1900)
;Image - Refers to graphic to use
;DualHeaded - Wether the same engine will be placed at the end of train (default: yes)
;CapacityType - Type of cargo being transported (default: none)
;CapacityAmount - Amount of units of the cargo being transported (default: 0)

[TRAIN01]
Name = Choo Choo
Year = 1943
Image = GRAPH1
DualHeaded = no

[TRAIN02]
Name = Zoof Zoof
Year = 2005
Image = GRAPH2
CapacityType = Passengers
CapacityAmount = 50

[GRAPHICS]
1 = GRAPH1
2 = GRAPH2

[GRAPH1]
Filename = Filename.extension

[GRAPH2]
Filename = Filename.extension

Or sth. I personnaly like this way of handling such thingies. It's just easy :P

What other possibilities are there? (Please show me pieces of code, if possible)
Contributor to the The 2cc Set and Dutch Trainset. Inventor of the Metro concept. Retired Graphics Artist.
Image Image
Download TT | Latest TTDPatch | OpenTTD | OpenTTDCoop | BaNaNaS: OpenTTD content system | 2048² OTTD scenario of the Netherlands
GRF Codec | GRF Crawler | GRF Maker | Usefull graphics & tools sites | NML Documentation Wiki | NFO Documentation Wiki
All my graphics are licensed under GPL. "Always remember you're unique, just like everyone else."
User avatar
aarona
Traffic Manager
Traffic Manager
Posts: 221
Joined: 26 May 2006 15:54
Location: Perth, Australia
Contact:

Post by aarona »

Other possibilities:

XML [XML]
Tab Delimited Files [TDF] (similar to tracking tables, perhaps?)
Text INI files [INI] (such as the one you presented)
Lua, Python "scripting" [LUA] (dunno if it could work)

Possibly a program (parser) could be created to...
[XML] to TECode
[TDF] to TECode
[INI] to TECode
[LUA] to TECode

(or vice versa.)

Where TECode is some type of GRF-style (either textual or binary) format which is easily loadable into TE.

Disadvantages: Someone would have to create this program!
Advantages: No complaints! Once this program has been created, the coding of TE is only dependant on TECode, and not on the other stuff.
User avatar
Hyronymus
Tycoon
Tycoon
Posts: 13233
Joined: 03 Dec 2002 10:36
Location: The Netherlands
Contact:

Post by Hyronymus »

And what about the Civ3 approach and possibly the Civ4 approach? Civ4 is a combination of XML and Python, Civ3 were formatted txt-files.
User avatar
Purno
Tycoon
Tycoon
Posts: 16659
Joined: 30 Mar 2004 12:30
Location: Almere, The Netherlands

Post by Purno »

aarona wrote:Tab Delimited Files [TDF] (similar to tracking tables, perhaps?)
Lua, Python "scripting" [LUA] (dunno if it could work)
Could you show me examples of how this would look like?
Possibly a program (parser) could be created to...
[XML] to TECode
[TDF] to TECode
[INI] to TECode
[LUA] to TECode
And how would this look like?
Hyro wrote:And what about the Civ3 approach and possibly the Civ4 approach? Civ4 is a combination of XML and Python, Civ3 were formatted txt-files.
And how does that look like?
Contributor to the The 2cc Set and Dutch Trainset. Inventor of the Metro concept. Retired Graphics Artist.
Image Image
Download TT | Latest TTDPatch | OpenTTD | OpenTTDCoop | BaNaNaS: OpenTTD content system | 2048² OTTD scenario of the Netherlands
GRF Codec | GRF Crawler | GRF Maker | Usefull graphics & tools sites | NML Documentation Wiki | NFO Documentation Wiki
All my graphics are licensed under GPL. "Always remember you're unique, just like everyone else."
User avatar
aarona
Traffic Manager
Traffic Manager
Posts: 221
Joined: 26 May 2006 15:54
Location: Perth, Australia
Contact:

Post by aarona »

Purno wrote:
aarona wrote:Tab Delimited Files [TDF] (similar to tracking tables, perhaps?)
Lua, Python "scripting" [LUA] (dunno if it could work)
Could you show me examples of how this would look like?
Not really cause I know such things exist but wouldn't have a clue how they would be used. The tab delimited files were a feature listed in the DD, but I wouldn't have a clue how it was intended to work. (Maybe one column for name, one for speed, etc??)

As Hyronymus mentioned, Python is used in Civ 4, and Lua is just a different type of language. Again I wouldn't know how it could be used.
Purno wrote:
Possibly a program (parser) could be created to...
[XML] to TECode
[TDF] to TECode
[INI] to TECode
[LUA] to TECode
And how would this look like?
Well this may be a program where you open it up, choose the input file, chose a name for the output file and press "Parse".
Any errors could be exported to a file, or printed on-screen.
The compiled file would then be loaded pre-game to define which sets to use.

It could go one step further and just have an IDE style, enter certain fields and out pops a XML, TDF, INI, LUA or TECode file. So for example if you wanted to create a train, you would simply click on the New Train button, up would come some options, name, year, etc and viola, you have a new train.

It all comes down to the question of who is willing/able to implement anything like this and would anyone want to use it (I could imagine an IDE would be popular).
User avatar
XeryusTC
Tycoon
Tycoon
Posts: 15415
Joined: 02 May 2005 11:05
Skype: XeryusTC
Location: localhost

Post by XeryusTC »

I've made a mockup of a scripted file, you can find it at http://xeryustc.cjb.net/mockup.lua.

Topic locked, lua made it to the DD (09112006).
Don't panic - My YouTube channel - Follow me on twitter (@XeryusTC) - Play Tribes: Ascend - Tired of Dropbox? Try SpiderOak (use this link and we both get 1GB extra space)
Image
OpenTTD: manual #openttdcoop: blog | wiki | public server | NewGRF pack | DevZone
Image Image Image Image Image Image Image
Locked

Return to “Transport Empire Development Archive”

Who is online

Users browsing this forum: No registered users and 4 guests