SERIOUS Suggestion - Please Read !!!

[SoofMan]

I think i know how to help you fight with the synchronization errors, wchich are very inconvinient for players.

1. The First, and MOST important thing is for the client to check MD5 sum, of a Map file every time it connects to a game, so it won't need to download the map every time ! I'm getting mad because i get desync errors VERY often, and every time i need to download the map !.

Of course the server must remember all MD5 sums of all the map files, so it can give it to a client if he asks for it. MD5 sum checking should be default and locked setting for all servers.
I think it shouldn't be a lot of work for such a good programmers.

2. Second thing is to FORCE every server to pause when a client enters the game so there will be much less synchronization errors. This setting should be default and locked, and should not be changed at all at any time.

2a). Of course when desync error occurs, the client and/or server should wait/try until the connection can be synchronized again instead of disconnecting, but you are probably working on it, so this is minor thing.

PS.
(Sorry for my english - i could have made some mistakes )

[XeryusTC]

I think i know how to help you fight with the synchronization errors, wchich are very inconvinient for players.

1. The First, and MOST important thing is for the client to check MD5 sum, of a Map file every time it connects to a game, so it won't need to download the map every time ! I'm getting mad because i get desync errors VERY often, and every time i need to download the map !.

Of course the server must remember all MD5 sums of all the map files, so it can give it to a client if he asks for it. MD5 sum checking should be default and locked setting for all servers.
I think it shouldn't be a lot of work for such a good programmers.

And rehash every time something in the map changes, that includes city expansion, terraforming, economy changes, building etc. That would make OpenTTD really slow

2. Second thing is to FORCE every server to pause when a client enters the game so there will be much less synchronization errors. This setting should be default and locked, and should not be changed at all at any time.

I agree partially with you here, you should be able to turn the pausing off, people could abuse this feature by connecting/disconnecting to a server continously.

2a). Of course when desync error occurs, the client and/or server should wait/try until the connection can be synchronized again instead of disconnecting, but you are probably working on it, so this is minor thing.

Desyncing occurs when the server and client have different data, this can be from a calculating difference in goods production to different grf sets. If you desync you just have the wrong data, you could try to restore it but that won't always help.

[Brianetta]

1. The First, and MOST important thing is for the client to check MD5 sum, of a Map file every time it connects to a game...

There's a misunderstanding of how the game deals with the map.

The map is in a constant state of change. It's not like other maps where the basic "level" never changes, but positions of objects might. No, the whole map is capable of change (and frequently does change a lot) so the MD5 checksum of the original map is useless after the game has begun.

When you connect to the server, what you get isn't the original map. What you get is basically the current saved game, as if it were saved over the wire into your client.

[SoofMan]

And rehash every time something in the map changes, that includes city expansion, terraforming, economy changes, building etc. That would make OpenTTD really slow

Then maybe the server should just hold memory of the diffrence between the beginning map state, and the current map state - this could save a lot of memory (or at least i think so - am I wrong ?).

BTW. The map data sent to client is ZIP/gzip-compressed, isnt it ?

I agree partially with you here, you should be able to turn the pausing off, people could abuse this feature by connecting/disconnecting to a server continously.

Why do you think the people would abuse this feature ? I though they connect to play, not to make other players upset. A 2 or 3-sec pause EVEN every 2-5 minutes isn't so bad at all - it gives you some time to think . At least it is A LOT better than desync error and disconnection, ISN'T IT ??

[SoofMan]

1. The First, and MOST important thing is for the client to check MD5 sum, of a Map file every time it connects to a game...

There's a misunderstanding of how the game deals with the map.

The map is in a constant state of change. It's not like other maps where the basic "level" never changes, but positions of objects might. No, the whole map is capable of change (and frequently does change a lot) so the MD5 checksum of the original map is useless after the game has begun.

When you connect to the server, what you get isn't the original map. What you get is basically the current saved game, as if it were saved over the wire into your client.

Oh yeah - now i understand. Then if nothing can be done about the map itself, then some way to remove the desync errors should be found.

Surely there is something wrong with the connection protocol. I know this maybe a silly suggestion, but maybe the programmers could learn something from creators of popular Open Source P2P software - like Emule, Azureus wchich IMO create very stable connections and handle reconnecting and synchronizing easily. Generally some study of popular open protocols could be useful.

Good luck for you people anyway - i hope i helped you just a little to improve this wonderful game.


EDIT :

Desyncing occurs when the server and client have different data, this can be from a calculating difference in goods production to different grf sets. If you desync you just have the wrong data, you could try to restore it but that won't always help.

Then if it is so, the server should calculate all cargo transport and generally all events on the map and simply post it to clients. And when a client has wrong data, it should change it's data to to the server data state. Server rulez the game. Am i wrong ?

EDIT2:
I don't know what did you mean by "different grf sets". If the client has diffrent data files from the server, it shouldn't be alowwed to play, as it can make the game unstable. So, the server should check exactly what data files does the client have (or simplier what version of the game does the client have). Correct me if I'm wrong.

[Prof]

I agree partially with you here, you should be able to turn the pausing off, people could abuse this feature by connecting/disconnecting to a server continously.

That's easy to fix - once any player has disconnected, don't allow connections from that IP address for a fixed length of time (say, 1 or 2 minutes)

[Brianetta]

Pause on join is an option anyway.

[SoofMan]

Pause on join is an option anyway

This is what i'm saying - it shouldn't be AN OPTION - it should be a DEFAULT and LOCKED setting - so the server owners cannot change it.

On servers where option "pause on join" is ON i do not suffer almost ANY connection problems, and on servers where the option is OFF, i sometimes get disconnected about every 5-10 minutes.

All :

What i'm saying is that solving the connection problems is absolutly THE MOST IMPORTANT thing - this is what makes the gameplay a terrible experience . Most of the servers is not playable at all because of frequent disconnections.

The programmers should leave everything else and make the connections be stable at any cost.

[Syranide]

Surely there is something wrong with the connection protocol. I know this maybe a silly suggestion, but maybe the programmers could learn something from creators of popular Open Source P2P software - like Emule, Azureus wchich IMO create very stable connections and handle reconnecting and synchronizing easily. Generally some study of popular open protocols could be useful.

P2P is nothing like a game protocol, not even close, they are pretty much as far apart two protocols can get without going to extremes. Surely desyncs could be fixed, but I'm guessing the time for doing such changes would be enormous, and generally it shouldn't be a problem unless your connection is really bad.

EDIT: SoofMan said about the server just updating the client with whatever is wrong, if one forgets the complications of perhaps just moving trains and states around (conflicting), you still have the fact... what is different? The server cannot possibly know without the client sending its entire state. So, not a possibility.

[CharlyHRO]

I think he means the server has to know everything and the client in a multiplayer game should just be a "dumb terminal" wich only tells the server what the user changes for himself and the server everytime knows whats going on in the game. This results in -> only the server knows the whole game. Thats the way a shooter works. every client tells the server what it is doing. and the server tells every client, what the other clients are doing. so if a client has a different state than the server, the client is WRONG and the server is RIGHT.

[DaleStan]

I think he means the server has to know everything and the client in a multiplayer game should only tell the server what the user changes for himself and the server everytime knows whats going on in the game.

Yes, so far, so good.

This results in -> only the server knows the whole game.

Now we have a problem.
The client has to know where every single vehicle is, so it can properly draw and route the vehicles (since some vehicles can interact).
The client also has to know the full map ("map" being the tile height and current tile contents (eg rail, road, station, &c.)), for the same reason.

There isn't much that doesn't fall into those two classes, so there's basically no point in reworking massive portions of the MP handling to reduce the map-transmission by maybe 5%.

every client tells the server what it is doing. and the server tells every client, what the other clients are doing. so if a client has a different state as the server, the client is WRONG and the server is RIGHT.

And again, this is the way it already works.

[SoofMan]

CharlyHRO wrote:
every client tells the server what it is doing. and the server tells every client, what the other clients are doing. so if a client has a different state as the server, the client is WRONG and the server is RIGHT.

And again, this is the way it already works.

OK. Then where do the desync errors come from ?? I suppose that they come from "desynchronization" as the name says. So if the client is not synchronized with the server properly, it should be forced to wait until he can be synchronized (and show "please wait - synchronizing with server") or just do something to get synchronized, instead of disconnecting.

Disconnect should ONLY occur, when the connection is lost totally, not when some data between client and server doesn't match. The server should fix all the incorrect data states of the clients - this is how the other game servers works - it also protects them from attempts of hacking (server knows everything and rules everything, so its more difficult to change its data).

[Prof]

My guess is that the server throws a desync error when the client indicates that it's stored map state is significantly different from the server's (i.e. reconnecting is the most sensible way to rectify the issue, to ensure that the connection is completely re-synced, rather than re-send lots of map data and other data).

[DaleStan]

Indeed.
As already discussed, resynching requires a complete transmission of the entire game state (often several megs). If a slightly broken client connects (wrong newgrfs, for example), being kicked is a FAR greater incentive to fix your client than just having it download the game over and over and over. Implementing such auto-resynching is an excellent way to make it impressively easy to DoS any server anywhere. Especially if pause_on_join is on.

Obviously, anyone could fix^H^H^Hbreak their client to auto-reconnect on desynch, but that requires work.

[SoofMan]

Great.

I have downloaded latest version from your page, everything seems to be OK with my client, i didn't change any settings or data files, and still on some servers I get disconnected INCREDIBLY OFTEN. Also i have noticed that other people get desync errors as often as I do on that servers.

It's simply TOO OFTEN - there is no other game that disconnects clients as frequently as OpenTTD does. So ther must be something wrong with the protocol.

Hey i have an idea. What about if the server checks the map's CRC/MD5 sum every period of time (let's say 1 or 2 minutes) and the client has to do the same ?.

EXACTLY :
(In the middle of game)
1. The server sends a "synchronization signal" to the client saying that the client has to create MD5 sum of the current map state in EXACTLY 10 seconds from receiving the signal.

2. The server and the client calculate the CRC/MD5 sum of the map data AT EXACT THE SAME TIME, to avoid data diffrences.
2a) There could be 0,25 or 0,5 sec break in data transfers to ensure the data won't be changed in the time the checksum is created). A break this small shouldn't be even very noticeable to players.

3. Now the server checks both MD5 sums to ensure that the client is properly synchronized with the server.

4. If the client's data state is diffrent from the server, then the client is forced to re-download current data state - the client does it and automatically reconnects.

EDIT2 :
I forgot to add, that (of course) the client's local time and the server's local time must be synchronized (meaning the server can calculate the exact time of the synchro) - but you probably already know that.

IN GENERAL, the client should automatically reconnect after he loses synchronization with the server. Manual reconnecting game with the server 5 times every 10 minutes is really annoing.

I hope this idea will help you in development.

EDIT : Another idea.

The server could store let's say 5 data states (and MD5/CRC sums) from last 10 minutes (when the CRC is created every 2 minutes). So when a client gets desynchronized, the server sends to client only diffrence between previous saved data state and the current data state - this should save a lot of data, and the clients wouldn't have to download whole data state every time they get desynched.

[MeusH]

I think MD5 checks for data files would fix some of the problems especially with other grf sets and scenarios (for diffs, see below)


Also, I think diff between the current map and seed map could have been sent, since it would be usually smaller file. Later in the game, when things get changed too much, whole map would be sent. But a diff would be awesome way to play faster.


It would be also a nice way to reconnect faster, if server makes a diff on client's disconnect, and sends that diff on client's connect.


Number of diffs would be not larger than amount of clients disconnected - if player gets disconnected the second time, previous diff would be overwritten.
Client should store the map when disconnecting, so one diff may be applied to continue playing.


Welcome, please post your suggestions and comments.

[SoofMan]

Ohh this game is so great...

If i was an ACTIVE C++ programmer, i would help you with the work. But for now, i programm only in DHTML, PHP, JavaScript and MySQL. I programmed a little in C++ once, but it was just "Hello world"- like programs and it was long ago... eh...

And my situation doesn't allow such pleasures... (no free time). I hope that some day I will have more time to help you.

[XeryusTC]

First of all, OpenTTD is coded in C not C++, there is a difference.

I don't think that md5 sums will help you solve the desync problems. I also think you don't know when it is possible to desync, you can even desync for very small things like a different amount of cargo generated by a factory or a difference in town growt.
The only thing that would help desyncing problems is that the client is more like "I am here to serve you".

You could also lower the amount of desyncs by playing on a server that is less laggy, this will improve your connection and by this the update frequency.

[MeusH]

Please take a look at http://wiki.openttd.org/index.php/User:MeusH/Graphs showing my ideas in more graphical way. The only requirement would be to generate map on seed and creating and applying diff files on maps

[SoofMan]

First of all, OpenTTD is coded in C not C++, there is a difference.

Well, i never wrote anything in C . And yes, i know there is a diffrence. Can i ask why did they choose C instead of more popular C++ ??

You could also lower the amount of desyncs by playing on a server that is less laggy, this will improve your connection and by this the update frequency.

I dont think so, because i played games on servers that seemed laggy (low FramePerSec, screen freezed for few seconds often) , and i had no desyncs at all. On the other hand i played on servers that didn't seem to be laggy (small map, smooth game, fast connection, and high FPS) , and i got desynced very frequently.

The only thing that would help desyncing problems is that the client is more like "I am here to serve you".

I don't understand what exactly do you mean by that. Explain it in more detail please.

I don't think that md5 sums will help you solve the desync problems. I also think you don't know when it is possible to desync, you can even desync for very small things like a different amount of cargo generated by a factory or a difference in town growt.

Thinking is one thing, trying it in practice is another. I think that MD5 or CRC suming everything can be a way to avoid data corruption. And some kind of data corruption is what causes the desync errors. Logical conclusion.