Why is the savegame file the way it is?

[spl]

(first time poster so be gentle =)

Evening all,

I've been looking into the format of the savegame file looking to do something with it outside the game (wouldn't it be nice to print a decent map?) and after spending a few hours looking at the file itself and pouring over the sourcecode I put it aside. Why is this file, by today's standards, so cryptic? Maybe cryptic is a bad word but it would just seem that any game play mechanic or resource (RAM, DISK, CPU) limitation that would have required it to be so disappeared a while ago.

-S

[Brianetta]

Nobody has rewritten it. Does it need to be rewritten? Perhaps. I'm sure the devs won't even consider rewriting it until they decide upon a new map format, which is when they will have to look at the saved game format anyway.

[Rubidium]

You say it is cryptic, but what would your suggestion be to improve it? As it has done the job pretty good so far.

Keep into consideration that the more time you need to actually write the savegame and read it has a direct influence on the playability of OpenTTD over the internet; a client has to be in-sync with the game (as in: the player is in the same game 'frame' as the server is) within a specified amount of ticks from the moment that saving of the savegame that is sent to the other player is begun. The client has to perform all commands/actions etc that the server has performed during that time, which includes pathfinding and such. Also a larger savegame would mean that it takes longer to transfer it to the client, which also affects the time needed to get in-sync with the game.

Every extra tick that is used for saving/transferring/loading the savegame does take cannot directly be seen as: it only takes X ticks more. You have to account for the fact that those X ticks have to be processed by the internal game engine not mean that the time needed to get in-sync is that same time. So you have to multiply X by some Y, where Y > 1 and dependant on the size of the map and the speed of the computer.

Another option is to pause the server (and clients) when somebody tries to join, but the extra time needed to save/transfer/load will influence the playability for the already playing people on that server.

Furthermore: not everybody using/playing OpenTTD has a state-of-the art computer; I know somebody who plays OpenTTD on one of the earlier Pentiums, as in one from before the time that the Pentium II came out.

[Darkvater]

I don't know what you call cryptic, but OpenTTD's savegame is most certainly not that.

The format starts with a four letter identifier of the compression used (eg OTTZ for zlib), then the savegame version.
After that the whole savegame is organized into so-called chunks, identified by again a four-letter identifier (eg VEHS for vehicles), its length and the actual contents.

The savegame might seem cryptic to you because the data inside a chunk is just binary data; what it is and how to read it is handled by structures inside OpenTTD. There is nothing cryptic about it.

[DaleStan]

If you want cryptic, look at TTD's original save-game format.
Up to 47 characters, followed optionally by a null and enough garbage to fill the 47-byte space, and then a checksum.
Then an RLE-compressed memory dump, and another checksum.

[hertogjan]

I always wondered why savegames are incompatible between versions of which patch settings differ, while the openttd.cfg file remains compatible.
I mean, when writing a patch and adding new patch options, this breaks compatibility of savegames ― i.e. it's not posssible to load a savegame from the patched version into the unpatched version. However, the configuration file from any version loads into any version; unknown patch settings are ignored, and missing settings are reset to default. Why is this not possible with savegames too?

[Darkvater]

It is because, as I said earlier, the whole savegame system is code-driven. The data inside a chunk is just binary "junk" with no description whatsoever. It is the code in openttd that tells it that for example the first 2 bytes are vehicleID, the next 4 are a v->next pointer, the next 1 is v->direction if the savegame version is between 10-21, in versions 22-40 the next 2 bytes are v->direction and between 41-and HEAD the next 4 bytes are v->image (because we dropped v->direction). Just an example.

So if a patch drops some un-trunk data into a chunk the code will not know what to do with it, and bail out.

The config file is different because the file itself has descriptors built into it. You know when you read the value 10 for example that it belongs to resolution. If the current codebase doesn't have it, that value can simply be ignored. This is something the savegame data doesn't have.

[spl]

Cool... this is all good stuff....

I guess the real question is why is the same game that's written to disk so closely tied to the internal representation of the game? Does it really need to be? It's not a striaght forward, easy-to-read file.

I looked through the save game/source-code and noticed early on that it was compressed, so the "cryptic" I spoke of was just the chunk and its internals in an OTTN file. I tried to approach it and started writing a chunk parser in PERL. All looked pretty ok until I got to the VEHS chunk.

Why can't the save file be self describing (not necessarily the same format as the config file though)? Why couldn't save routine generate a readable file and the load rebuild the internal structures? What ill-effects would this have on the game-play?

-S

BTW, how are the scenerio files carried from verion to version?

[Rubidium]

See my previous post in this topic. It might be possible to make it a little more self-describing, but adding too much overhead is bad for network gameplay and require a new saveload system and format. While you need to maintain compatability with the current version.

Scenarios are basically normal savegames without players.

[athanasios]

Cryptic or not, we cannot edit it in the editor without the players getting lost.

[DaleStan]

gunzip it and then apply a hex editor.

And there are perfectly good reasons for creating scenarios with pre-existing players -- see the MegaRail scenario for one.

[hertogjan]

It is because, as I said earlier, the whole savegame system is code-driven.
(...)
The config file is different because the file itself has descriptors built into it. You know when you read the value 10 for example that it belongs to resolution. If the current codebase doesn't have it, that value can simply be ignored. This is something the savegame data doesn't have.

Yes, I know the difference in how the data is stored.
The real question is: Why is the patch settings data in the savegames not stored in the same way as in the config file (i.e., "varname=value") for cross-patch compatibility?
I mean, if you would just attach a config file to a savegame, then you wouldn't have the problems with incompatibilities (supposed that the program knows how to handle it; i.e. to read and write it the same way as it does with openttd.cfg).

[Quark]

I want to see savegame format as ‹chunk branch id›‹chunk branch version›‹chunk data› and all chunks, that have different ‹chunk branch id› or ‹chunk branch version› should be simply ignored when loading saved game.

[Darkvater]

Yes, I know the difference in how the data is stored.
The real question is: Why is the patch settings data in the savegames not stored in the same way as in the config file (i.e., "varname=value") for cross-patch compatibility?
I mean, if you would just attach a config file to a savegame, then you wouldn't have the problems with incompatibilities (supposed that the program knows how to handle it; i.e. to read and write it the same way as it does with openttd.cfg).

I just explained to you why it is the way it is. This produces the smallest savegames and is how all the other chunks are handled.

I think you guys are on a wrong track here. If your real problem is with the intercompatibility of branches, or the difficulty of hacking savegames then you should address, that, not via this detour. Branch incompatibility won't be solved by a descriptive system. If bits 0..3 in m4 mean something totally different in branches/cbh for example it will not make your branch game load any better in trunk/.

[spl]

Darkvater has explained many times that any change in the chunk date will result in version incompatibility and I don't think that's misunderstood. Though from what I been hearing it sounds like the method used to save games to disk is the same way games are "saved" (sync'd) to other players during online play. I'm not sure this was understood (am I correct/0. Ok, can't this be separated? Is it absolutely required or is it just convenient?

From my point of view a save-to-disk happens very infrequently compared to the online transactions of a multiplayer game that you can sacrifice a bit of performance for a few seconds to write/read a savefile to/from disk then return to the "cryptic" method for online play.

Interop between versions would be very very kool plus the sides effect having more complex scenarios (w/ players & verhicles) and perhaps external world generation (or conversion say from real map data) could add another dimension to the game. To approach any of these w/o straight forward format just multiplies the complexity of the problem and people are less likely to make the attempt.

This game has a fair bit of history but does this extent to the save file? There was an idea proposed above about having a cfg style file to map out the bits&bytes, This is step in the right direction but I think we should take it farther. Look at what's happening with other document formats (online and/or offline and that really what what we got here is a document)... the "binary" form is falling away and XML is replacing it. (I'm sure that going to touch a few nerves) It's less intimidating to look at, reasonably self documenting, and there are several strong, stable libraries available (and any worth their salt support unicode.) Why can't OTTD benefit from this?

more of my $0.02 (ok may a bit more)

-S

[hertogjan]

I was talking only of patch settings here, not about all other things like map array and vehicle data, etc.

My issue is the following: When I write a patch (one which stores new patch settings in savegames), then it will break savegames when the trunk is updated with other new patch settings. As far as I know, there is no way to get it right (i.e., a way that allows you to maintain savegame compatibility when new settings are added either from your side or from the svn trunk side). If you have any tips how to do it right with the current savegame structure, then I want to know.*

And is it that hard to add a new chunk with just the openttd.cfg-style configuration options? Old versions may ignore it (for compatibility; I hope that the program currently does this with unknown chunks) when it is present. The new version of openttd that supports the new chunk, uses it when it is present in the savegame, and use the existing patch settings chunk for old savegames where the new chunk is absent. It doesn't add much to the size of the savegame. The file openttd.cfg is only a few kB, and not all variables in there need to be stored, and moreover, it is compressed like the rest of the savegame.


*I already know how it should look like when it would be directly integrated to trunk (i.e., do a savegame version increment). The problem is about savegames for unmerged patches.

[DaleStan]

Darkvater has explained many times that any change in the chunk date will result in version incompatibility and I don't think that's misunderstood. Though from what I been hearing it sounds like the method used to save games to disk is the same way games are "saved" (sync'd) to other players during online play. I'm not sure this was understood (am I correct/0. Ok, can't this be separated? Is it absolutely required or is it just convenient?

Required. Well, almost. All clients must have a state that is absolutely identical to the state on the server.
If the savegames and the network-transfer are different, then either
1) The savegames contain extraneous information,
2) The savegames are in a different format, or
3) The savegames have a different compression scheme (eg bz2, not gz)

#1 just strikes me as stupid, and #2 is fraught with immense quantities of extra work, and hence extra bug-potential. #3 is the only option that is really reasonable.

Look at what's happening with other document formats (online and/or offline and that really what what we got here is a document)... the "binary" form is falling away and XML is replacing it.

Are there any other games where the saves can exceed 5MB post-compression[0], period? If so, are those saves routinely transfered over the internet?

[0] Pre-compression, there is up to 24 MB of map data alone.

[Brianetta]

Are there any other games where the saves can exceed 5MB post-compression[0], period? If so, are those saves routinely transfered over the internet?

Yes, and yes. I beta tested and coded missions for Egosoft's X2: The Threat and X3: Reunion (you can find my name, both as Brian Ronald and as Brianetta, several times in the credits for both games).

For the initial release of X2, the saved games were over 50MB, and weren't compressed. An update introduced compression, getting the size down to 5MB on average. Saved games are routinely transferred over the internet.

The game's configuration (and mod interface) are entirely XML based. Even the Turing-complete scripts, where there are literally things analogous to <line number=10>print "Hello World"</line> (with a nice XML stylesheet to syntax-highlight all that). That sounds pretty lame, but there were reasons for this (such as translating function names and constants to names that are meaningful in other languages in the in-game menu-driven editor). Anyway, I'm diverging from my point.

The saved game could have been XML based, too. Goodness knows, the rest of the game is. It was considered not optimal, though, and the chosen saved game format is close to (but not quite) a simple memory dump, compressed.

[Darkvater]

I was talking only of patch settings here, not about all other things like map array and vehicle data, etc.

And is it that hard to add a new chunk with just the openttd.cfg-style configuration options? Old versions may ignore it (for compatibility; I hope that the program currently does this with unknown chunks) when it is present. The new version of openttd that supports the new chunk, uses it when it is present in the savegame, and use the existing patch settings chunk for old savegames where the new chunk is absent.

The current code aborts the load code if an unknown chunk is found (SlLoadChunks() function).

To spl: the current savegame format has absolutely nothing to do with TTD's original savegame format, besides the fact that both are, as Brianetta said, some elegant memory-dumps. OpenTTD saveload code can intelligently store and restore pointer relations, compress arrays and handle ALL savegames up to the very first version released. Look at Warcraft III for example. Every time a new patch comes out all your old replays, become unusable.

The fact that the code is not written for loading games from totally different branches cannot be faulted to OpenTTD, mainly as I explained before (different variable usage, etc.). An exception to this is the PATS section as it indeed only stores a certain part of the configuration file and could be more lenient. But no amount of hacks/changes to the current code will make that work.

There are plans to change the saveload system to make it data-driven (eg have some XML-style (since everybody seems to like XML I'll call it that) data descriptors) and be more tolerant to unknown values, but it will take a little while, and is certainly not top priority.

[spl]

Darkvater has explained many times that any change in the chunk date will result in version incompatibility and I don't think that's misunderstood. Though from what I been hearing it sounds like the method used to save games to disk is the same way games are "saved" (sync'd) to other players during online play. I'm not sure this was understood (am I correct/0. Ok, can't this be separated? Is it absolutely required or is it just convenient?

Required. Well, almost. All clients must have a state that is absolutely identical to the state on the server.

almost? Then it's not required.

If the savegames and the network-transfer are different, then either
1) The savegames contain extraneous information,
2) The savegames are in a different format, or
3) The savegames have a different compression scheme (eg bz2, not gz)

#1 just strikes me as stupid, and #2 is fraught with immense quantities of extra work, and hence extra bug-potential. #3 is the only option that is really reasonable.

#1 extraneous information... It might, but so what, for versions older or modified/patched versions that's don't understand it would simply be ignored and never makes it into the internal reps. Currently, every version (even a micro version) has the potential to obsolete the previous savegame. This has got to wreak havoc on regression testing. (OpenTTD is regression tested, isn't it?)

If the savegame was more transparent I could imagine setting up a test map where all the features of the game (the bells and whistles as it were) would be laid out in artificial test scenarios. Worked up to ensure things are doing what there suppose to do. The file would need to be re-engineered for every single release.

#2 ok, sure. They're in a different format. Again, so what. I'm not asking for 31 flavors of the OpenTDD (binary) save (that's what we got now). One of the beauties of XML is that you can forget the bit/byte counting. It's not parsed that way. Once the the main structures are properly defined they will remain stable for years. Small changes won't cause the parser to go bananas (it's all about the tags) and if the grammar changes significantly, It's easy to whip up a XSLT stylesheet and convert/correct the savegame (extremely easy if compared to doing the same in say C/C++ with the current savegame). At best this discussion could result in a spec for OpenTTD save game format v2.0.

Immense? isn't that going a bit far? I know it's a bit cliche, but OpenTTD is large and complex piece of software, I'll bet it took immense quantities of extra work to complete. Was it worth it? (I think the ensure bug hunt would we worth the returns.)

#3 Compression doesn't address any of the transparentness of a save file... It can be applied to any save file. Take a look at OpenOffices save(filetype:swx)... It's a bunch files (XML, bitmaps, etc) all rapped up in some InfoZIP goodness. In OpenTTD's case XML doesn't imply uncompressed.

Look at what's happening with other document formats (online and/or offline and that really what what we got here is a document)... the "binary" form is falling away and XML is replacing it.

Are there any other games where the saves can exceed 5MB post-compression[0], period? If so, are those saves routinely transfered over the internet?

[0] Pre-compression, there is up to 24 MB of map data alone.

I'll bet ya that an XML version of the complete same game will be 2-3x (atleast) that size but what harm does it do if at the beginning of the game you got the same map in front of you and the same internal rep?

again... I do not want to change the way that OpenTTD manages to keep a multiplayer game in sync (you guys got that licked). I really don't (database syncing isn't my thing) but I do want some transparency into the savegame.

-S