Questions for: Godot + OpenTTD = 3D Viewer project

[HayabusaShinkansen]

As described in this thread (please read to see the scope) => viewtopic.php?p=1258268#p1258268 with @zephrys' blessing, I'm starting work on a 3D Viewer (running as a separate program) using zBase in 3D, and in future, potentially others.

I've been doing some research. It turns out doing saved games is easy, but that deprives you of the delight of watching your vehicles move around. To do that, I need to attach to a game in progress. I'm thinking of doing that with the communications protocol. I've found there are many libraries out there which do that, but I am thinking the ideal approach would be to use OpenTTD's networking module directly, to convert that into a Godot native C++ plugin, and then have it communicate with Godot through GDScript (a variant of Python modified for real-time performance). That way the 3d Viewer could get the map and vehicle data directly from the horse's mouth. It would save rewriting yet another OpenTTD admin library, and protect it against future changes.

Can someone with experience of the OpenTTD code base tell me is this possible? Is there are better way?

One thing that might be a problem for the above is it might demand a network game?, which would deprive you from looking at a game you started standalone. Is there any other way to monitor an OpenTTD game in progress?

Also reason I decided on a separate program is, using Godot really demands that, and I'd hate to add 3D hardware prerequisites which would prevent OpenTTD from running on a lot of older PCs on which it now runs beautifully.

[jfs]

Keep in mind that the way OpenTTD network games function is that clients only send commands about user actions to the server, and the server only shares those commands around to the other clients. The actual movement of vehicles, cities growing, trees and water expanding etc. is all simulated on every client separately. You would need to code a new protocol if you wanted to share all details about a game in a way an external program could read.

I think the best bet would be to hook into the running OpenTTD process' memory and read data out directly, from your 3D viewer program. You can extract the data structures you're interested in from the OpenTTD source and make sure you're building with a compatible compiler, so you get the exact same memory layout. Then somehow find the locations of all the global variables pointing at the data you want, and you're basically ready to pull out your data.
The main disadvantage of this approach is that your viewer would only work with one specific version (in fact, one specific build) of OpenTTD at a time.

This idea is similar to what e.g. Stonesense for Dwarf Fortress does.

[Eddi]

there are basically two main options for extracting game data from a live game.

the first option, like jfs describes, is a patched client that accesses the data structures directly, which is fast, but must be recompiled for each new version. if this is done carefully (i.e. you don't interfere with the game state) this patched client can connect to normal servers, e.g. as a spectator, or run as a server (or single player) by itself.

the second option, is to have a GameScript which exports data through the admin interface. this is probably rather slow, but more flexible in the sense that the script interface is pretty stable between releases (and a backwards compatibility layer exists)

[jfs]

For the GS with Adminport option, keep in mind that the Adminport is only available during multiplayer games.

[HayabusaShinkansen]

Thanks @jfs and @Eddi. You've saved me a lot of time!

What I'm thinking now: I'll write the program modularly, so you have the option of using any method, and I can roll it out a stage at a time. I'll do (1) the 3D module, (2) a game state module - to represent the map and current vehicle states, and (3) the various interfacing methods as their own module which update the game state, which will in turn push the changes through to the 3D module.

I'll do the 3D module and saved game method first (because I expect it's the easiest), the GameScript method next (because it's more work), and the Patch last (because I suspect that'll be the most work).

Godot's 3D is very easy to use. You create an object, load it with a GLB mesh, give it's orientation in 3D space, and that's it. But I'll probably multimeshing(instancing) esp for the map since that let's you draw thousands of 3D objects in not much more time than it takes to draw one, so the size of the map shouldn't be an issue. If it is, I'll add some tiling.

I'll check out aidansean's OpenTTD Surveyor https://github.com/aidansean/OpenTTD_surveyor as a basis for the save game loader. I'll begin exploring the OpenTTD source code. First thing to do would be to export the map, vehicles later. Please keep an eye on this thread and as I hit issues I'll ask for help here.

Thanks for your advice!

[HayabusaShinkansen]

@zephrys, could you please point me in the right direction? I've been through your blend files and understand most of it, but can't see where you add the textures to certain structures (the brickwork on copper_ore_mine, the worn paint on the oil_refinery, etc.) Where are those textures coming from? When I render in Blender I get plain shading, and the material definitions don't specify any textures. (I'm using Andy's clone at https://github.com/gravitystorm/zbase since openttdcoop.org has gone. He says made some minor changes, though dropping the textures seems unlikely.)

@jfs, @Eddi: Thanks for your advice. Shared memory turned out to be the easiest. The OpenTTD source is well documented, so it was easy to find my way around. I'll add the other methods eventually, but for now, this is enough. Only done the map at this stage. Know the vehicles will be much harder, but one step at a time.

[jfs]

Last night I got an idea about maybe making a general interface to let external applications query OpenTTD for its memory layout of various game objects, such that you could e.g. send a query for "what is the memory address of the Vehicles array" and "what are the offsets of a vehicle's X,Y,Z coordinates in the vehicle structure". I have no idea how feasible that would be, but it might be a fun project too. It could definitely support more use cases for people wanting to make applications to draw advanced statistics for their company finances, or make transit maps of their networks.

[JGR]

Last night I got an idea about maybe making a general interface to let external applications query OpenTTD for its memory layout of various game objects, such that you could e.g. send a query for "what is the memory address of the Vehicles array" and "what are the offsets of a vehicle's X,Y,Z coordinates in the vehicle structure". I have no idea how feasible that would be, but it might be a fun project too. It could definitely support more use cases for people wanting to make applications to draw advanced statistics for their company finances, or make transit maps of their networks.

This sort of approach adds whole categories of technical and platform-specific difficulties.

Trying to extract statistics from a remote process using debugger mechanisms is much more difficult and unreliable than just asking it to create a savegame (e.g. using the admin port), and then analysing the save file.

[HayabusaShinkansen]

I will eventually add support for a savegame state over the admin port, but I'll also see how I can connect the 3D Viewer to an OpenTTD process in as unobtrusive a way as possible so I can show real-time trains etc. Windows and Linux both have a similar memory sharing API which is quite nice. The terrain map is straight forward, but the vehicles are harder because they're a pool. Synchronization is an issue. I'll explore these and post what I discover.

[smurfix]

Well, what I would do if I was trying to do that (which I totally want to but have less than zero time for) would be to run an embedded Python interpreter as a thread in a "headless" OpenTTD process. You can then easily pass a bunch of accessor functions to this interpreter that let you figure out which sprites in which orientation are where, add a couple of callback hooks to tell the interpreter about major events / movement it's currently interested in, etc.

The idea of reconstructing an ongoing game using a network connection is … um … daunting, You basically need to recreate all of OpenTTD's logic faithfully. While rewriting the game in Python would be a superb achievement, it's also a ton of work which somebody already did, to the tune of ~200'000 lines of C++ code.

[JGR]

You'd be vastly better off forgetting about using Python.

Inserting event logging or query functions into the OpenTTD source is doable but trying to make it work with Python would just be a huge headache for no benefit, especially if your target system is already in C++.

[HayabusaShinkansen]

I've coded a lot in C++, Python, and have integrated Python into C++ programs, but I don't think that's necessary here since it adds bloat and compatibility problems. Infamously Python dropped support for Windows 7 IMHO without good reason.

In this case though an easier solution is to use a DLL dynamic link interface. That would let third-party authors write their own plugins/addons/extensions. On startup, OpenTTD would look for such plugins (they are just DLL files with an agreed API), and when game execution hits a part in the code you want to inform the plugins, do the callback by calling the nominated DLL function. The code footprint is small and it's just a few CPU cycles to check. If the user isn't using any DLLs the cost is negligible. With a clean interface, you could conceivably integrate this into OpenTTD's main branch. I've used this technique before in commercial software and it worked well.

And if you really, really want a Python interface, you can implement that as a DLL, without breaking anyone else's compatibility.

You basically need to recreate all of OpenTTD's logic faithfully. While rewriting the game in Python would be a superb achievement, it's also a ton of work which somebody already did, to the tune of ~200'000 lines of C++ code.

Yes, I don't think you could do that. Just not practical. Callbacks are far easier, but appreciate they need to be low overhead and not overcomplicate the rest of OpenTTD.

General report: Found shared memory with OpenTTD's map was easy to implement. Looked briefly at doing callbacks, but that harder to figure out what information needs to be passed so needs more reasearch. Found interfacing to Godot is possible but painful as they have just migrated to a new version, and their addon interface is a PITA.

@zephrys, please let me know where I can find your wear and tear textures (described above). Also a re-upload of zBase dev would be appreciated, since openttdcoop.org is down.

[HayabusaShinkansen]

Update: Haven't done any more on this because I haven't been playing OpenTTD, because I've grown incredibly bored with the chore of creating and scheduling new trains every time I create a new station. viewtopic.php?p=1264154 I know how to do it manually, but I must have done it 10,000 times manually. If someone can suggest a way to do this automatically, that'd be great...

[WenSim]

Update: Haven't done any more on this because I haven't been playing OpenTTD, because I've grown incredibly bored with the chore of creating and scheduling new trains every time I create a new station. viewtopic.php?p=1264154 I know how to do it manually, but I must have done it 10,000 times manually. If someone can suggest a way to do this automatically, that'd be great...

Sounds like you're looking for the 'Shared orders' feature.

When you clone a vehicle, hold the Ctrl key, and the orders of the existing will be automatically shared to the new vehicle.
Similarly, when in the 'Orders' screen, you can Ctrl+click on another vehicle (after clicking 'Go to') and it will share the orders from the clicked vehicle into the one you're editing.

Shared orders already done half of the stuff. You might be looking for the "transfer orders" feature, which only exists in JGRPP.
This feature allows you to transfer train orders from one station to another. It also comes with a filter, so that you can filter out specific trains you want don't want.

[HayabusaShinkansen]

Thanks, @WenSim. Found JGRPP here: https://github.com/JGRennison/OpenTTD-patches Hope it rekindles my love for OpenTTD!