New graphics format

[pasky]

I've put something together and posted on #openttd, so I figured out that it could well live on the forums as well. Feel free to comment on the proposal, it's only a very fresh (still hot from the mind-oven) heap of ideas. I'll keep updating this as the proposal evolves, so the text below is the latest proposal agreed upon.

***

So, how do I imagine this PNG graphics stuff?

Note that this is a very early sketch. I did not really look deeply inside to
this and to the implementation side as I'm busy with making TTDPatch GRF codec
now. Also note that the original idea of using user chunks in PNG files is not
mine - IIRC Vurlix suggested that but I'm not really sure. Please tell me if it
was you.

***

So, as I imagine this, we need to divide the problem a little. First, please
raise above the current way of how things work a little and clean your mind.

So, what are we going to talk about? It is not solely graphics, although it's
one of the main parts. But I want these files to keep all the other relevant
gameplay data - ranging from vehicles info through info about all the buildings
(industries as well as town houses) to user-defined airports. So let's call it
"CGD files" for now, CGD standing for "combined game data".

***

First let us concern with how the CGD is distributed amongst users and used by
OTTD. In order to have all the data easily distributable and compact, we should
keep it all in a single file, much like GRF. And to make it easy to build tools
for working with these files, the underlying format will be pure PNG. This PNG
file will contain graphics chunks containing the sprites and user chunks
containing the relevant game data (usually but not necessarily associated with
a particular sprite).

***

However, such a CGD file would be a PITA to create or modify. Therefore, there
must be a tool like grfcodec, which will uncompress and compress a CGD file
to/from a more friendly form. This form should consist from a set of PNG files
(coming from the graphics chunks) and a set of GPD files (GPD standing for
"gameplay data") (coming from the user chunks). Such GPD files should be
directly modifiable of a text format. Maybe XML, but I would personally favour
something less controversial; simple INI-style format could do.

PNG files on their own would be meaningless - each PNG file would be referened
from one or more of these GPD files. Therefore, imagine ie. this:

Small Arctic Set.cgd :

Code: Select all

Dm3.png
A40.png
Maple tree.png
Paper mill.png
HAirport.png

Dm3.gpd        defines a new rail engine Dm3, sprites in Dm3.png
Dm3-alt.gpd    defines a new rail engine Dm3 with slightly different attributes,
               sprites in Dm3.png as well
A40.gpd        defines a new rail engine A40, sprites in A40.png
Maple tree.gpd defines a new tree, sprite in Maple tree.png
Paper mill.gpd defines a new industry, sprites in Paper mill.png
HAirport.gpd   defines a new type of airport, sprites in HAirport.png

Of course a PNG file could contain multiple logically connected sprites just
like in GRF files, ie. a sprite superset for a coal wagon - 8 sprites per
direction make a spriteset, 4 spritesets per loading state make a sprite
superset (the numbers are just examples).

While in the PNG file we could put each single sprite to a separate chunk and
then group those chunks somehow, in the extracted PNG files we have to use a
different technique. Either we go the way of GRF files and record coordinates
and size of each of the sprites, or we could number the sprites 1,2,3,4,5,...
(by the position of the upper left corner of the sprite box, counting from the
upper left corner of the image) and then bound each sprite by a box from some
special color, say a purple with non-zero (but low) alpha value. This could be
more friendly to the artists.

Then the other question is of course how to encode "special" colors from the
GRF files - transparency, company livery, flashing lights... Regarding
transparency, we can make use of PNG's alpha channel. Regarding the other
stuff, we could again take advantage of some special RGBA combination - I would
personally suggest a certain shade of purple with varying alpha value (keep
alpha value 0, max/2, max for the artists, though). It should be possible to
redefine certain special colors to use some other RGBA combination for a given
sprite, though.

[Darkvater]

I just looked up PNG specifications and the bad news is that user-made data chunks have a size limit
The good news is that that limit is 2GB . So I think we are safe until Michael creates the World Set with all possible liveries ranging from 1800-2100.

[Wolf01]

RPGMaker uses PNG files to create animations, example the water, for terrains and other... all is put together in a large PNG file and the program seems to "know" the position of a single chip of terrain... for animated chips is a sequence of images (now is only 4, but in a future version maybe there will be more)... we can use this method to put all sprites of a thing, example: train car, in one PNG and then configure the game to read the paints at specific coordinates... this would be simple with standard sprites, waggons are 32*32 if i'm not wrong, if someone make a custom sprite with 16*32 the game went weird

a good solution could be a separation line of a "unused color" (in RPGMaker is a sort of pink), when the game encounter that color, it understand that the spite is finished and then pass to the next sprite

[minime]

One note on the structure of the GPD files. I believe that they should have a little more structure than a plain NFO file, but I agree that XML is an overkill (and too much of a hype these days).

Something that would have some similarity to a simple "programming" language would be great. For example you could write a thing like:

Code: Select all

define_vehicle(train, "Class 151", 0x5E) {
   assign_sprites(4,8) {
       sprite("c151.png", 10, 10, 40, 40, ... );
        ......
   }
   set_properties {
       speed = 128;
       weight = 300;
    }
    ...... and so on
}

That would be much more readable than what the NFO files look like at the moment (which is one step away from actually hex-editing the GRF). It should not be terribly hard writing a parser for this and the syntax would be relatively simple.

It would definitely make working with it easier. There would probably be a possibility to make some optimizations in the back end to for example reduce redundancy and produce most efficient output.

It would also allow more flexibility in designing the graphics files, since you could possibly extend the syntax to even allow variables, simple computation, etc.

The only problem may be in re-generating the text back from the binary data. However this should hardly be necessary, since all the source files should be publicly available anyway.

[Villem]

I would vote for implementing LUA scripting system..

[MYOB]

I would vote for implementing LUA scripting system..

From my memory of Lua, its horribly over complex and painful to use

BeOS 5.1 uses compiled Lua script and PNG images for the window decors, and they're incredibly incredibly complex to make.

[minime]

A whole scripting language would definitely be an overkill. Remember, the point of my idea is to make the graphics definitions
a) easier to work with
b) more structured (which will support point a)

Don't let yourself be confused as to what the example I gave looks like. It is just for illustration.

[Braino]

a good solution could be a separation line of a "unused color" (in RPGMaker is a sort of pink), when the game encounter that color, it understand that the spite is finished and then pass to the next sprite

I think it uses the ALPHA channel option to make the color transparent, the sprite is still at the size of 16*16 px (hm, I am quite sure, cause I have had much fun with the RPG-Maker, plus I am the webmaster of one of the biggest German RM sites, I think ). The use of the ALPHA channel could be an alternative for OpenTTD as well - or am I wrong? Then designing new tiles should be much easier! =D

[Wolf01]

a good solution could be a separation line of a "unused color" (in RPGMaker is a sort of pink), when the game encounter that color, it understand that the spite is finished and then pass to the next sprite

I think it uses the ALPHA channel option to make the color transparent, the sprite is still at the size of 16*16 px (hm, I am quite sure, cause I have had much fun with the RPG-Maker, plus I am the webmaster of one of the biggest German RM sites, I think ). The use of the ALPHA channel could be an alternative for OpenTTD as well - or am I wrong? Then designing new tiles should be much easier! =D

yes, also the alpha channel is good, i used it to make transparent zones of a picture for another game, it works very well

[sign]

We could use the Console Scripting Language for the CGD Files.... its easy extendable and we can use custom variable types for GRF Files. The only thing we need is that we should hide these commands from the commandlist and we should hook their access to be only accessable from CGD File parser... but that meight not be a problem

[Cephalaspis]

Could CGD file be a zip or rar (or cab) archive with PNG and code file in it? O.o. Then there is no need for coding/encoding tools.

[iridium]

[foreword] The aim of my post is to suggest ways which would make it easy for people to develop new 'gameplay data/graphics' sets. Some of my ideas may be stupid/impractical, please tell your honest opinions. I don't intend for this post to be 'definative' in any way, I'm just trying to provoke some thought/imagination. [/foreword]

In this gameplay data file, what information needs to be specified?

Obviously, some kind of sprite identifier, the position of a particular sprite in a PNG.

And, say, for vehicles - properties such as its cost?
Speed, horsepower, etc etc?

What about stuff like animations? I quite like minime's idea. I sorta envisage:

Code: Select all

// File name, identifier
spritedefinitions( "c151.png", 1 )
{
  sprite( 0x0001, 5, 5, 10, 10 );
  sprite( 0x0002, 20, 5, 10, 10 );
  ...
};

// File name, identifier
spritedefinitions( "cursors.png", 2 )
{
  sprite( 0x0001, 5, 5, 10, 10 );
  sprite( 0x0002, 20, 5, 10, 10 );
  ...
};

Then later..

// Vehicle class, name, identifier
vehicle( train, "Class 151", 0x0001 ) 
{
  spritefile( 0x0001 );
  sprites(0x0001, 0x0002, ...);
  properties 
  {
    cost = 1000;
    speed = 128;
    weight = 300;
    carries = { CARGO( passengers, 100 ) };
  };
};

// Animation Identifier
animation( 0x0001 )
{
  spritefile( 0x0002 );
  sprite( 0x0001 );
  delay( 0x0005 ); // Some kind of time?
  sprite( 0x0002 );
  delay( 0x0010 );
  ....
}

// Cursor identifier
cursor( "dynamite" )
{
  animation( 0x0001 );
}

This format could be compiled/decompiled in to a simple, pure binary format so that (relatively) slow parsing doesn't have to take place on game load.

All of the identifiers in a particular file only need be unique within the file. The engine can assign 'globally-unique' identifiers when the files are loaded . This would make things simple for anyone making more/new gameplay data.

When OTTD starts, it can scour the data directory for *.gpd files and load them. This means that adding extra trains/graphics/whatever would just be a matter of downloading a gameplay data pack from whatever site, and dumping it in the data directory - No need to edit any files and add things in.

I also propose that the gameplay data file have room for comments, e-mails, websites, names, etc.

Basically my idea is to allow easy modification of any of the graphics and 'gameplay data' (such as vehicle information, town house cost, etc).

Each of the .pngs could have their own palette and the game overall could use a 32-bit palette. This could allow for more colour flexability (if it is wanted or needed?)

As an aside from my post, I suggest that if/when we choose a new graphics format, the format is made first using the current(ly available?)
graphics so that development may start. I might be able to help make some weird and wonderful tools to allow conversion between grf/the new format. (And ways of getting the current gameplay data from the OTTD include files to the new gpd format).

Also, as a way of not cluttering the filesystem, a single gameplay data file could contain multiple gameplay data files. One could potentially be extracted from the file and be made seperate at any time.

I like the idea of using any kind of open source archiving tool.

Pete.

[Joker]

I think this idea is very good.
As for the vehicle information, you say that XML is an overkill. But I think XML was created exactly for tasks like this one. If you need "more than plain .NFO file", then XML will fit just right.

[Darkvater]

I am not saying anything about the proposed format, just a comment about XML.

XML is a huge overkill for this stuff. A compact binary format is a lot better. Of course this is very close to the .grf format which is uneditable. That's why it needs a decoder/encoder; and there isnothing wrong with doing that into XML. Just like what patchman did with Locotool (or close to).

[traskjd]

A compact binary might be small but I don't see what advantage you gain?

- XML can be easily modified
- Being text based it is highly compressible (no 56k issues)
- The data can be quickly read in with an xml text reader and put into the same data structures as what the binaries would be.
- Much easier to add custom sections for patches etc

I don't really see any issue and would like to see XML used.

- JD

[DaleStan]

- Being text based it is highly compressible (no 56k issues)

Correction: Being text based, it is full of redundant data, hence highly compressible. It takes up to 11 bytes to store a signed 32-bit int in text (decimal); it takes 4 to store it in binary. (6 and 2 for a 16-bit, 4 and 1 for 8-bit. Remove one byte from each text-format-size for unsigned ints.)

Making format decisions on the basis of compressibility is NOT a good way to go. One-shot systems for a certain format are almost always better compressors (when properly designed, at least) than generic systems like ZIP, RAR, GZ, &c.

[traskjd]

- Being text based it is highly compressible (no 56k issues)

Making format decisions on the basis of compressibility is NOT a good way to go.

Hence why I didn't suggest it based on this one item alone. Even with a few bytes here and there it's still going to be fine for 56k users. One thing that is changing in IT in general is that with increasing power and performance we can finally afford to use technologies that are slightly more usable by human beings.

Why require that people have tools to edit files in order to save a one off increase of say 50KB (which would be a crap load of compressed xml)? Surely you see that having a tool to edit custom files is redundant in that they still need to download more data to do that.

The performance hit to the game would be negligible. One thing that may hold this project back is the perpetual notion that it should run on decade old hardware - sure, I love how TT plays but that doesn't mean we can't make it more usable.

The type of argument that we should save tiny amount of data is about as useful as claiming we should abandon OpenTTD in its current state and rewrite it in assembler.

We can afford some extra usability now. It will only help make things simpler over time and I see no reason why we shouldn't start with something as simple as this.

- JD

P.S. These are just my views, I enjoy a healthy debate and in no way do I speak on behalf of the developers etc of OpenTTD

[Alltaken]

i also agree, i think that things need to move into the future, a 75Mhz computer uses more energy than is efficient compared to a 2.8Ghz computer.

they litterally cost more to run than is even worth it. why design for them.

Alltaken

[mdhowe]

i also agree, i think that things need to move into the future, a 75Mhz computer uses more energy than is efficient compared to a 2.8Ghz computer.


they litterally cost more to run than is even worth it. why design for them.

Alltaken

That's got to be the stupidest argument I've ever heard. I don't care how energy efficient a 2.8Ghz computer is, the simple fact is if you can't afford one you make do with what you've got.


Listen up people!

If your computer has anything less than 2.8Ghz processor then throw it away, it litterally [sic] costs more to run than is worth it! Jesus, how much money have I wasted! I feel so stupid.

[DaleStan]

I'll admit that I'm a bit biased here, in that I understand the NFO format.

Just to make sure we aren't talking past each other: if NFO were augmented with XML, that would be a Good Thing, but the binary version should be (1) canon (ie the format distributed by graphics coders and read by OTTD) (2) human parsable and (3) human editable. The XML version should just be an ease-of-use extension, and back-form-able from the binary version. If we are in agreement here, then disregard the rest of this message.

The type of argument that we should save tiny amount of data is about as useful as claiming we should abandon OpenTTD in its current state and rewrite it in assembler.

I didn't say that. You did.

At least, it seems to me that you said "We should use text so we can compress it, to reduce the size of the downloads," to which I said "OK, I'll agree that download size should be reduced, at least for now. Therefore, why not use a format that is already compressed, and probably smaller?" (OK, so I didn't actually say the first sentence, but the thought was definitely there.)

I feel that the compressibility of the format should have little, if anything, to do with which format is selected. The conditions I see are, in this order:
1) Speed of run-time parsing (runtime of the graphics loader, including any built-in decompression)
2) Size of parsable version (total data read from disk, to be passed to the graphics loader)
3) Speed of one-time compiling (runtime of GRFCodec)
Numbers 2 and 3 are very close to each other, but this is the order as I see it.

All else being equal, smaller is better, but all else is rarely equal.
Should we really require that everyone who wants to use OTTD build it themselves, just to make the download(s) smaller?

Why require that people have tools to edit files

What do you mean by "tools"? If you're referring to GRFCodec (and/or its successor), sure, it takes something to compile PCX+NFO into GRF, but it'll still take something to compile PNG+XML into GPD.

By the same argument, why require people have tools (built into OTTD or otherwise) to interpret files they are never going to change, or even look at? Basic GRF support is already built into OTTD. I doubt XML parsing is.
The size of the compiler shouldn't matter to anyone except the developers; similarly, the size of GRFCodec shouldn't matter to the average user.

One thing that may hold this project back is the perpetual notion that it should run on decade old hardware

I don't know about you folks, but TTDPatch regularly consumes 80+% of my 1.8GHz P4, which, although no longer exactly cutting edge (OK, so it was rather dull when I got it), isn't "decades old" either.

We can afford some extra usability now. It will only help make things simpler over time and I see no reason why we shouldn't start with something as simple as this.

The NFO format is incredibly powerful. Within the limits of TTD, there is nothing it cannot do, or be extended to do. Granted, a properly designed XML format could duplicate the powers of NFO, but it wouldn't really be that much easier[0]. Writing NFO files isn't called "coding" for no reason; it is a programming language. One thing that non-programmers often fail to understand is that it really doesn't matter how cryptic the language is, most of your energy will be spent figuring out how to do what you want to do, and relatively little will be spent actually coding it.

The only problem I see with the binary format (this is not insurmountable) is endian-ness, and it's easier to read big-endian number on little-endian hardware than it is to read a decimal number on either hardware type.

[0] Try designing a XML format to duplicate the powers of the 80+x variables of Variational Action 2, without actually using the hexadecimal numbers 80-FF.