A new discussion on languages, file formats etc

[jfs]

Picking up from this old topic on programming considerations and the current discussion of tasks to do in parallel with the DD, I believe it'll be a good idea to discuss programming language choice(s), graphics engine, perhaps also sound engine, data file formats and perhaps some other technical standards we should agree on.

Edit a bit after posting: Hmm I hope this isn't something people had imagined would go in the DD, by the way. IMO this stuff doesn't belong there.


On programming language, I think a hybrid compiled/scripted solution would be the most flexible.
I think C or C++ is the best choice for a compiled language, mostly because they are proven, and lots of libraries exist for them already. The best argument for C would be simplicity. I think it's easy to learn (unless you don't have any programming knowledge beforehand) and there's a limit to how much "logical spaghetti" you can create. C++ allows for some very powerful/complex constructions especially using it's templating feature, which is IMO both a power and a weakness, because it can create exatcly that "logical spaghetti". Object oriented programming is almost a must though, so I think C++ will be the way to go. But as far as I'm concerned, almost any language will do, except VB, assembler and machine code

Now, on the scripted stuff I was talking about. I think it will be a great solution to put much of the actual game logic into some kind of dynamically compiled files. It can allow for faster development by simplifying the coding/testing cycle, because it removes the usual compilation step. Theoretically, it would even be possible to edit the code while the game is running!
This of course isn't a new idea, and I belive the currently most used solutions (apart from proprietary solutions used in some places) are Python and Lua. Of those, I've only tried Lua myself, integrating it into a Delphi application, which went very smoothly. Another possibility might be TinyCC, but I don't know how good a choice C would be for "scripting". (Also see my discussion about file formats below.)

Another topic that has previously been discussed was compiler choice. I think that's a moot discussion, since the obvious answer is: Every single compiler you can imagine. We'll just #define us out of it.


Now, on data file formats.
I believe we should aim for using open file formats and compression algorithms as far as possible, and attempt to avoid know-patented technologies, even if the patents aren't valid in the countries where the developers live. For sound, that would mean Ogg Vorbis, which has been proven as an excellent format numerous times before, and FLAC can be used in case we need "lossless" sound.
For images, PNG would most likely be the format of choice for most graphics, but we would probably also want to support some kind of lossy graphics compression. I believe JPEG is an option for that?
Also, if we need vector graphics for something, SVG would be a choice, I believe. I haven't had a real look at it myself, though, but I know it's used a bit in recent GNOME distributions.

The format for 3D models should probably be dictated largely by the choice of graphics engine, and what's easiest to generate models for. (Of course the choice of engine should also be affected by that.) I don't know enough about this to comment on any formats.

What's most interesting though, IMO, is object description formats. (Eg. how to describe a new vehicle to be available in the game.)
Many people immediately scream XML when it comes to this, but I think the most flexible solution would really be to use "scripted data files" for object descriptions! This is also one of the reasons I think a hybrid compiled/scripted language solution is good.
I think most people here have an idea of what can be done with GRF files in the TTDPatch alpha versions, and I'm sure everybody would want TE to be able to do the same (and more?), but in a much easier way So an "object description script" would really consist of a number of functions which are called by the game engine whenever something specific happens. Eg. there is a function that fills a vehicle data structure when a new vehicle of this type is built, one that's called whenever a vehicle of this type is (attempted) added to a "vehicle chain" (eg. a train or a trailer to a lorry) and likewise one for removal, one when loading-status changes etc. etc. That would basically give ultimate freedom for vehicle designers. Also, for basic needs, a "template" could be supplied, so even non-programmers could easily create vehicles (though not as sophisticated ones) for the game.


The final thing I want to discuss is the choice of a graphics (and perhapd also sound) engine. People have mentioned SDL and CrystalSpace. I think someone also said something about CS being best suited for FPS-style uses, and not so much for isometric-like almost-top-down views, like we'll need much of the time.
With SDL, we would (correct me if I'm wrong) still have to write most of the actual 3D stuff ourselves, since it only provides an operating system neutral interface to OpenGL. This is much more work. On the other hand, the developers would have to learn how to use another pre-written 3D engine we we choose such a one, and that could perhaps turn into just as much work.
I think a graphics engine would need to support not having actual displayable 3D data generated the entire landcape at all time. Also it's an obvious requirement that it must support PiP (picture in picture) for smaller views in eg. vehicle windows. Finally, I think it would be really great if we could have multimonitor support, perhaps with a separate viewport on each monitor. (So you could basically work in full screen on two different areas of the map at one time, given your hardware can support it.)


Hmm, this turned into a rather long essay
RFD
And if someone thinks this thread covers too many topics (it probably does) feel free to split it

[Arathorn]

I don't think we need lossless audio, so Ogg Vorbis should be the way to go.
On the the 3D models format: How do we want to do it? Do we want to make some sort of "container" format that contains the model itself and some sort of config file for every object? I believe Blender can export to a lot of formats, and there are many convertors between formats, but I would prefer to have native .blend support, along with maybe .obj support.

[Zuu]

On programming language, I think a hybrid compiled/scripted solution would be the most flexible.
I think C or C++ is the best choice for a compiled language, mostly because they are proven, and lots of libraries exist for them already. The best argument for C would be simplicity. I think it's easy to learn (unless you don't have any programming knowledge beforehand) and there's a limit to how much "logical spaghetti" you can create. C++ allows for some very powerful/complex constructions especially using it's templating feature, which is IMO both a power and a weakness, because it can create exatcly that "logical spaghetti". Object oriented programming is almost a must though, so I think C++ will be the way to go. But as far as I'm concerned, almost any language will do, except VB, assembler and machine code

Alltroght C compiles faster than C++, I agree with you that C++ is probably the way to go.

Now, on the scripted stuff I was talking about. I think it will be a great solution to put much of the actual game logic into some kind of dynamically compiled files. It can allow for faster development by simplifying the coding/testing cycle, because it removes the usual compilation step. Theoretically, it would even be possible to edit the code while the game is running!
This of course isn't a new idea, and I belive the currently most used solutions (apart from proprietary solutions used in some places) are Python and Lua. Of those, I've only tried Lua myself, integrating it into a Delphi application, which went very smoothly. Another possibility might be TinyCC, but I don't know how good a choice C would be for "scripting". (Also see my discussion about file formats below.)

My experince of Lua (5) is that it requires a layer between C++ and Lua, which is quite complicated, especially the way of reading arguments, and sendig back data. Ie you have to read all arguments from a stack and then put the reurn values (yes you can use more than one) on the stack.

But, there might exist a better way today, where the arguments are provided in a more C/C++ way.

But gennerally I am positive to moving the "content" and some "logic" out of the kernel/core. If it can be done in a neat way.

Another topic that has previously been discussed was compiler choice. I think that's a moot discussion, since the obvious answer is: Every single compiler you can imagine. We'll just #define us out of it.

We'll while it would be neat to support every compiler in the world, I think we have to dra a line somewhere on how buggy and non-standarized compilers we want to support.

I've heard that the OTTD developers have gone cracy on Visual C++ becuse all exceptions for Visual C++ in the code. But I don't know how much lays in the way Visual C++ don't follow the standards, and how much lays in how OTTD is organized.

---

What's most interesting though, IMO, is object description formats. (Eg. how to describe a new vehicle to be available in the game.)
Many people immediately scream XML when it comes to this, but I think the most flexible solution would really be to use "scripted data files" for object descriptions! This is also one of the reasons I think a hybrid compiled/scripted language solution is good.
I think most people here have an idea of what can be done with GRF files in the TTDPatch alpha versions, and I'm sure everybody would want TE to be able to do the same (and more?), but in a much easier way So an "object description script" would really consist of a number of functions which are called by the game engine whenever something specific happens. Eg. there is a function that fills a vehicle data structure when a new vehicle of this type is built, one that's called whenever a vehicle of this type is (attempted) added to a "vehicle chain" (eg. a train or a trailer to a lorry) and likewise one for removal, one when loading-status changes etc. etc. That would basically give ultimate freedom for vehicle designers. Also, for basic needs, a "template" could be supplied, so even non-programmers could easily create vehicles (though not as sophisticated ones) for the game.

As a refference I have uploaded init_object.lua which defines all objects (which aren't that many ATM) in Junctioneer and LuaC.cpp which contains the C++ functions that is called from lua. The whole package can be obtained at http://www.sf.net/projects/junctioneer/ .

[jfs]

We'll while it would be neat to support every compiler in the world, I think we have to dra a line somewhere on how buggy and non-standarized compilers we want to support.

I've heard that the OTTD developers have gone cracy on Visual C++ becuse all exceptions for Visual C++ in the code. But I don't know how much lays in the way Visual C++ don't follow the standards, and how much lays in how OTTD is organized.

Unless I'm badly mistaken, there's a huge difference between VC++ 6 and 7, in that v6 has a large number of problems following the standards, while v7 is one of the most standards-compliant C++ compilers.
And yes, we should set a limit on what compilers we want to support. I think a reasonable choice would be GCC 3+ (and derivates, such as MingW), MSVC++ 7+ and possibly OpenWatcom. I honestly don't know how good Watcom is, though; if it's too broken atm. we shouldn't support it. But basically: officially, only support compilers that aren't majorly broken. If someone wants to compile it with his Fooware Crap++ compiler, then that's his own problem.


On the scripting-question, I know Lua can be cumbersome to work in, but the API is still rather clean. I think there are some nicer C++ wrappers for it as well. (I wonder if a C++ wrapper can somehow be made with templates?)

That's why I also suggested looking at Python. I'm looking at its C/C++ API right now, and it seems to be nicer than Lua's at a first glance. I'll post some more after I've read most of the stuff there is in the docs


Edit:
Another thing, I think we should use a virtual filesystem approach to data files. So you can either have all files for eg. a vehicle inside a single packed file, or as separate files on the host filesystem. The engine shouldn't care about that. The only problem is what to do in case of a name clash (several files with the same path and name.)

Also, I've looked a bit more on the Python C API, and it does indeed seem rather nice. You don't need to do all that messing around with a stack like in Lua, but on the other hand, I'm not sure I have discovered how to properly call a function yet

Edit again:
Found it now. There's several ways of calling a function/method, but on of them is:

Code: Select all

PyObject* PyObject_CallFunction(	PyObject *callable, char *format, ...)

Now, the really good thing here is the "format" and "..." arguments, you simply specify a kind of "format string" for the arguments the function takes, and then gives the arguments. It can automatically do conversion from basic C types and such. Functions can more or less return multiply values, in the sense that they can return a single touple that contains multiple values. Touples are also rather easy to deconstruct, I think.

Right now, I say we go for Python rather than Lua. Also, I think the "tables" and "meta-tables" concepts in Lua are confusing; Python seems to be more "normal" in most aspects.

[Hellfire]

As far as scripting languages are concerned: note that most (all?) of them are not included on out-of-the-box Windows pc's. The only things on them are Visual Basic Scripting, some Java scripting and Batch files... :S

My point here: if we include scripting support, we need to supply the script interpreter. The larger the interpreter, the larger our game. Also, we need to include a different interpreter for each platform, which might give compatibility issues. Now there's something to think about.

[Darkvater]

I've heard that the OTTD developers have gone cracy on Visual C++ becuse all exceptions for Visual C++ in the code. But I don't know how much lays in the way Visual C++ don't follow the standards, and how much lays in how OTTD is organized.

Visual Studio 6 is a pain in the ass, so you'd do best to avoid that. VS7 is a much much better compiler. We have some (minor) problems with it, but that is because MS choose its C compiler not to behave exactly to the standards. From what I have used, VS7's C++ compiler is really great, so no problems with that.

You also have to take care when using some internal functions. For example there is a nice internal qsort() function but that is of course slightly differently implemented in the MS and GCC camps; which required us to write our own qsort implementation. Also VS7's optimizing compiler is at times too eager to optimize and that coupled with some magic code, can result in unwanted bugs only surfacing in Release Builds.
It's the small things that create headaches

[Zuu]

Unless I'm badly mistaken, there's a huge difference between VC++ 6 and 7, in that v6 has a large number of problems following the standards, while v7 is one of the most standards-compliant C++ compilers.

I've heard that the OTTD developers have gone cracy on Visual C++ becuse all exceptions for Visual C++ in the code. But I don't know how much lays in the way Visual C++ don't follow the standards, and how much lays in how OTTD is organized.

Visual Studio 6 is a pain in the ass, so you'd do best to avoid that. VS7 is a much much better compiler. We have some (minor) problems with it, but that is because MS choose its C compiler not to behave exactly to the standards. From what I have used, VS7's C++ compiler is really great, so no problems with that.

I didn't knew that VC++ 7 was much better than VC++ 6, but that sounds nice.

There is always something new to learn.

[Darkvater]

I didn't knew that VC++ 7 was much better than VC++ 6, but that sounds nice.

There is always something new to learn.

When you "buy" Visual Studio don't forget that VC7 is Visual Studio.NET 2003 . Don't even think about buying VS 2002, that is utter crap.