Cargo Distribution

[fonso]

I don't know how far you'll get with your laptop, but there is a nice manual about desync debugging in the Wiki: http://wiki.openttd.org/OpenTTDDevBlack ... _debugging

If you'd like to help me you could try some of the things listed there. I'd be very happy about that. I'll probably find some time for cargodist on the weekend. Then I can do some debugging myself.

Maybe you'll want to change LinkGraph::SpawnThread in linkgraph.cpp to not actually spawn a thread, but use the current one. That can easily be done by removing the line "if (!ThreadObject::New(&(RunLinkGraphJob), this, &thread)) {" and the corresponding closing bracket. If that fixes the desync, we know the problem is about the threading.

[Creat]

I've let the game run for a while now on two machines with the own thread for the link graph disabled and it seems fine so far (even though there hasn't been much player-activity, it's just running in the background). So there does seems to be a race condition hidden in there somewhere. It also makes sense, since my laptop is a (non-hyperthreading) single core, the other PC a dual core, so there's plenty of room for it to appear.

I had a look at the wiki-article you linked for debugging desyncs, I'm afraid it would take way too much time for me to become acquainted with all that. Even starting with trivial things like the fact that I can't run configure commands. I'm on windows and shell scripts don't work, not even from a bash shell.

I'm gonna have a closer look a little later though and try again, but if it truly is a threading problem I don't think there is much I can do about it anyway...

[fonso]

I'm gonna have a closer look a little later though and try again, but if it truly is a threading problem I don't think there is much I can do about it anyway...

You could do the same test again with threading enabled. Reloading on the server usually changes something and I guess the desyncs happened before you reloaded..

[Creat]

Yea I just tried that, and couldn't even get in the game with the second client... OpenTTD Crashes, and the crash happens on joining the game from the second PC (my laptop), without the crash handler stepping in and the game continues to run for a short time (while displaying the crash message).
This all sounds a lot like the linkgraph thread (or another non-main thread, if there are any besides the linkgraph) is causing the crash.
The problem was, that I could only get the other PC (without debugging capabilities) to crash, not the version on my laptop (maybe because it's a single core?).
I finally managed to get remote debugging to work (wasn't easy as the crash wouldn't happen with my user, but only the main user on that pc), but as soon as the debugger is attached the crash no longer occured either. I had to attach it post mortem to finally get a call stack, and it really isn't of much help (see attached file).

If there is anything more I can do to help clear this up, let me know...

As I could now join the game again with the second PC (multithreading enabled) we started to continue playing the game. At first it only ran in the background, then we actively played. It took over 30 minutes to get a desync, and it happend suspiciously close after I saved the game (might still be a coincidence). Since the other (non-multithreaded) attempt was without this much player interaction (any only ever at my laptop since I was alone) that doesn't say much about the desync to be honest... We'll continue this game in the eavening with the single threaded build to see if that also produces a desync after some time...

[fonso]

Thanks for all the feedback. Obviously there are several problems. For debugging the desync you could set the desync debug level to 2. You don't need to recompile for that. Just open the game console with "^" and type "debug_level desync 2". Then you get a lot of output on stdout from both games. I'd be interested in that. You can also set it to 3, then you get additional "autosaves" named dmp_cmd* from both games. Maybe you can compare them and give me the last pair that matches and the first one that doesn't.

[fonso]

about the crash: Can you remember anything about the error message? Any words you might have spotted? Was it a short one or was it a monster like this: http://thedailywtf.com/Articles/Avoidin ... aspx#PPic3

[Creat]

The crash message was a regular windows type "do you want to report the error to microsoft" kind of thing. It wasn't caught by the OpenTTD crash handler. I'm now sure that this goes away when using a single threaded build, btw (as we've been playing with such a build for a few hours now). Error details are as follows:
AppName: openttd.exe AppVer: 1.0.0.19561 ModName: openttd.exe
ModVer: 1.0.0.19561 Offset: 0022f1c4

Additionally, OpenTTD just disappeared a couple of times. No error message, no windows notification that the program has crashed and is about to be closed, it was just gone. I'm 98% sure that also only happened with the multithreaded version...

I can also tell you now that desync issues are practically gone with a single threaded build, and the remaining desync is reproducible: If the server (my laptop) saves the game, the other client (pc) shortly thereafter desyncs. Always.

I'd love to give you more detailed desync debugging outputs, I just don't know how to get the stdout written to a file. I can't redirect it like with a console application (for example "openttd.exe > logfile.txt"), because it's a gui apllication. Does OpenTTD automatically write this to some file? Or can I enable it somewhere?

[Zuu]

glx has written a small CLI tool to convert openttd.exe to a console application: http://devs.openttd.org/~glx/convert.zip

After using that, you can run OpenTTD from the terminal and redirect stdout (and stderr if you want) to file.

[Creat]

The only entries I got in the log were a bunch of these:
dbg: [misc] String too long for destination buffer

and yes, I made sure debuglevel for desync was set to two.

With the single threaded version I can now confirm that the only desyncs happening are those after saving the game (they don't happen on autosaves for some reason). We also didn't encounter even a single crash (as described above), so there is definitely some sort of bug in the synchronization...

[fonso]

"String too long for destination buffer" doesn't happen anymore with the latest version ... Can you recheck the version strings on both computers? I see ... you're playing with the patch from that post above, so your version string might be longer than 15 characters. I need the exact version string to load that Multiplayer.sav, though.

Edit: ... ...

Code: Select all

alve@schaf:~/projekte/openttd$ git apply ~/Downloads/patch\ from\ trunk\ r19561\ to\ r19571.diff
/home/alve/Downloads/patch from trunk r19561 to r19571.diff:632: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

*clears throat* ... However, this is the chinese language file, so it's probably not the problem. In general, files with messed up EOF are *evil* and lead to strange bugs.

Edit: That Multiplayer.sav is seriously corrupted. If you are able to load that, basically anything can happen. Do you have an earlier save? Could it be that the save is from an earlier version of cargodist, for example the one corresponding with trunk r19533? The savegame format has changed between that and the current one, but the save/load version hasn't. So the current version will try to load a game like that and interesting things will happen, for example crashes.

Edit: It could also be that cargodist's save/load code is broken. I'm investigating that.

Edit: Yes, the save/load code is horribly broken. Sorry for that. You can stop debugging now.

[fonso]

Creat's desyncs and crashes should be fixed now. Actually the savegame wasn't really corrupted, but the loading code was broken. You should be able to load it with the latest version and it should not desync (however, it might give you some strange link graphs at first).

The desyncs were caused by two effects of my refactoring:

• Link graph components were cleared when saving. This of course has interesting and nondeterministic effects on the results of currently running link graph calculations.
• Link graph handlers were not stateless and different invocations of the handlers were overwriting each others states when running concurrently. Actually they have never been stateless but before the refactoring a new handler was created for each run.

Both problems are fixed now. Thanks for the report.

[Creat]

*clears throat* ... However, this is the chinese language file, so it's probably not the problem. In general, files with messed up EOF are *evil* and lead to strange bugs.

I didn't get that warning when I applied it to my repository, the patch was simply generated on a git clone of the trunk repository, so no idea how that happened...

Thank you for the prompt fixing of the crash/desync, I'll probably won't get to play until Monday evening or so though, I'll let you know how it goes

I've also finally take a couple of minutes to document the following bug (at least I think it is one...), which has been bugging me for a while now:
If you play a game with ECS vectors (in particular the agricultural vector), you get fishing grounds. These are basically boats that produce fish and have a built-in dock and heli-port. In order to increase production to a meaningful level, they have to be supplied with passengers (meaning: workers).
With CargoDist you can't connect the ship route to your network, as there will never be any passengers sent to the fishing grounds, even though the linkgraph sees the link capacity to the fishing grounds correctly. There is just never any planned/scheduled flow.

It works fine if you create an isolated route (meaning the dock or heliport of the player isn't a joint station, which is part of a train/bus route), but the link graph also doesn't show any planned/sent flow and the passengers always go to 'any station'.

I'm guessing the problem is probably connected to the fact that these "fishing grounds"-docks don't belong to my company. I'd still very much appreciate it if this could somehow be fixed

Savegame with an example is attached, the used version is just a plain build from the git repository, no patches or anything (version in file description).

[fonso]

The problem with the fishing grounds is that it doesn't produce any passengers on its own. You have to supply it with passengers first. If you're playing with symmetric demands for passengers no passengers will be brought to the fishing grounds because no passengers can be collected there - otherwise it' wouldn't be a symmetric distribution.

This is a hen-and-egg problem. You can switch passengers to asymmetric until the first planned flow to the fishing ground is created. Then you can switch back to symmetric and it will mostly work. However, the "bursty" nature of ECS' cargo generation doesn't play well with cargodist's method of measuring production. The route might stall again at some later point in time.

You can also try to play with asymmetric demands for passengers. With the new supply and acceptance measurements introduced by supply scaling the distribution shouldn't be that different from the symmetric one.

[Creat]

In a short test (using the savegame from above) switching the demand from symmetric to asymmetric for a short time to jump start the connection worked, and it did seem reasonably stable - occasionally the passengers at the fishing grounds had 'any station' as their destination though. So with more aggressive linkgraph settings (shorter recalculation intervals), at a longer distance from the shore or with a smaller ship this will probably not work.

I think it is just a workaround though. After all the ECS vectors aren't doing anything 'illegal' or that they aren't supposed to do. So it should still be possible to use that set to the full extent in CargoDist.

I can think of two solutions:
1) Slightly alter the symmetric function to always create a non-zero demand for any station that does accept the cargo, an offset if you will. This doesn't have to be huge, just enough to allow industries like this to start up
2) use the demand-function that is already built into these industries. The amount of passengers that can be delivered to fishing grounds (for example) is limited anyways. It is also displayed in its infobox, but I'm not sure if it can be queried somehow. As this is the actual demand of the industry there is no need to calculate any virtual demand based on it's production. It may need to be scaled a bit to avoid sending almost every passenger there when the network is still small and other demands are significantly smaller.

[fonso]

I have changed the symmetric demand calculation so that any leftover supply after the regular demand calculation is assigned to stations without supply. This fixes your problem and might lead to funny results in other situations: If you have one very big station, otherwise only very small ones and one fishing ground you'll get a LOT of passengers travelling from the big station to the fishing grounds ... well, whatever, it doesn't need to be realistic and covering all strange corner cases would make the code horribly complicated. In the case of fishing grounds the problem fixes itself at the next link graph recalculation and if you have a lot of stations accepting things but not supplying them you better chose asymmetric distribution anyway.

[Logital82]

Question: In Station a are waiting 1000 people. 100 of them wants to C 900 to D. There are two line running with trains capaciy of 100 passangers. One line goes from A via B to C, the other from A via B to D. Now the train that runs to C enters the station. Will now the passengers randomly enter the train or only those one that wants to go to C?

[fonso]

Question: In Station a are waiting 1000 people. 100 of them wants to C 900 to D. There are two line running with trains capaciy of 100 passangers. One line goes from A via B to C, the other from A via B to D. Now the train that runs to C enters the station. Will now the passengers randomly enter the train or only those one that wants to go to C?

They will randomly enter. In fact at station A we don't know yet which passengers will go to which station. The numbers you see in the station GUI are more or less educated guesses. We only know they are all going via B and at B they'll decide to either stay there or where to go next, depending on the planned and real flows at B at the moment they arrive there. When the train to C arrives at B with some of those people, the ones deciding to go to D will get off the train and wait for the one to D. The ones deciding to go to C will stay on the train.

[nighthawk_c_m]

Either I am blind or there is no explicit documentation?

I seek some *.txt file or so that explains the single options in the menues of Openttd which were added by the CargoDist patch.

right now I am more or less successfully trying to figure their working ways out.

If anyone knows of a documentation or has one - please kindly point me towards it or slap me if it is in a more then obvious place.

[fonso]

There is no short .txt file with those options explained, but you can read up on them on the wiki page: http://wiki.openttd.org/Cargodist

OK, the page is long and complicated, but most of the options can't be explained in one sentence anyway. If anything is missing I'll gladly add it.

[Arie-]

Hey fonso, I was wondering: what's your idea of the development as it is currently. Are you planning another beta test on a COOP server, are devs reviewing the code, or are the functionality in general or the advanced configuration settings still subject to change, e.g..