Questions for: Godot + OpenTTD = 3D Viewer project

Forum for technical discussions regarding development. If you have a general suggestion, problem or comment, please use one of the other forums.

Moderator: OpenTTD Developers

Post Reply
User avatar
HayabusaShinkansen
Engineer
Engineer
Posts: 10
Joined: 10 Feb 2023 01:04

Questions for: Godot + OpenTTD = 3D Viewer project

Post by HayabusaShinkansen »

As described in this thread (please read to see the scope) => viewtopic.php?p=1258268#p1258268 with @zephrys' blessing, I'm starting work on a 3D Viewer (running as a separate program) using zBase in 3D, and in future, potentially others.

I've been doing some research. It turns out doing saved games is easy, but that deprives you of the delight of watching your vehicles move around. To do that, I need to attach to a game in progress. I'm thinking of doing that with the communications protocol. I've found there are many libraries out there which do that, but I am thinking the ideal approach would be to use OpenTTD's networking module directly, to convert that into a Godot native C++ plugin, and then have it communicate with Godot through GDScript (a variant of Python modified for real-time performance). That way the 3d Viewer could get the map and vehicle data directly from the horse's mouth. It would save rewriting yet another OpenTTD admin library, and protect it against future changes.

Can someone with experience of the OpenTTD code base tell me is this possible? Is there are better way?

One thing that might be a problem for the above is it might demand a network game?, which would deprive you from looking at a game you started standalone. Is there any other way to monitor an OpenTTD game in progress?

Also reason I decided on a separate program is, using Godot really demands that, and I'd hate to add 3D hardware prerequisites which would prevent OpenTTD from running on a lot of older PCs on which it now runs beautifully.
User avatar
jfs
Tycoon
Tycoon
Posts: 1750
Joined: 08 Jan 2003 23:09
Location: Denmark

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by jfs »

Keep in mind that the way OpenTTD network games function is that clients only send commands about user actions to the server, and the server only shares those commands around to the other clients. The actual movement of vehicles, cities growing, trees and water expanding etc. is all simulated on every client separately. You would need to code a new protocol if you wanted to share all details about a game in a way an external program could read.

I think the best bet would be to hook into the running OpenTTD process' memory and read data out directly, from your 3D viewer program. You can extract the data structures you're interested in from the OpenTTD source and make sure you're building with a compatible compiler, so you get the exact same memory layout. Then somehow find the locations of all the global variables pointing at the data you want, and you're basically ready to pull out your data.
The main disadvantage of this approach is that your viewer would only work with one specific version (in fact, one specific build) of OpenTTD at a time.

This idea is similar to what e.g. Stonesense for Dwarf Fortress does.
Eddi
Tycoon
Tycoon
Posts: 8258
Joined: 17 Jan 2007 00:14

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by Eddi »

there are basically two main options for extracting game data from a live game.

the first option, like jfs describes, is a patched client that accesses the data structures directly, which is fast, but must be recompiled for each new version. if this is done carefully (i.e. you don't interfere with the game state) this patched client can connect to normal servers, e.g. as a spectator, or run as a server (or single player) by itself.

the second option, is to have a GameScript which exports data through the admin interface. this is probably rather slow, but more flexible in the sense that the script interface is pretty stable between releases (and a backwards compatibility layer exists)
User avatar
jfs
Tycoon
Tycoon
Posts: 1750
Joined: 08 Jan 2003 23:09
Location: Denmark

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by jfs »

For the GS with Adminport option, keep in mind that the Adminport is only available during multiplayer games.
User avatar
HayabusaShinkansen
Engineer
Engineer
Posts: 10
Joined: 10 Feb 2023 01:04

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by HayabusaShinkansen »

Thanks @jfs and @Eddi. You've saved me a lot of time!

What I'm thinking now: I'll write the program modularly, so you have the option of using any method, and I can roll it out a stage at a time. I'll do (1) the 3D module, (2) a game state module - to represent the map and current vehicle states, and (3) the various interfacing methods as their own module which update the game state, which will in turn push the changes through to the 3D module.

I'll do the 3D module and saved game method first (because I expect it's the easiest), the GameScript method next (because it's more work), and the Patch last (because I suspect that'll be the most work).

Godot's 3D is very easy to use. You create an object, load it with a GLB mesh, give it's orientation in 3D space, and that's it. But I'll probably multimeshing(instancing) esp for the map since that let's you draw thousands of 3D objects in not much more time than it takes to draw one, so the size of the map shouldn't be an issue. If it is, I'll add some tiling.

I'll check out aidansean's OpenTTD Surveyor https://github.com/aidansean/OpenTTD_surveyor as a basis for the save game loader. I'll begin exploring the OpenTTD source code. First thing to do would be to export the map, vehicles later. Please keep an eye on this thread and as I hit issues I'll ask for help here.

Thanks for your advice!
Last edited by HayabusaShinkansen on 17 Feb 2023 01:05, edited 2 times in total.
User avatar
HayabusaShinkansen
Engineer
Engineer
Posts: 10
Joined: 10 Feb 2023 01:04

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by HayabusaShinkansen »

@zephrys, could you please point me in the right direction? I've been through your blend files and understand most of it, but can't see where you add the textures to certain structures (the brickwork on copper_ore_mine, the worn paint on the oil_refinery, etc.) Where are those textures coming from? When I render in Blender I get plain shading, and the material definitions don't specify any textures. (I'm using Andy's clone at https://github.com/gravitystorm/zbase since openttdcoop.org has gone. He says made some minor changes, though dropping the textures seems unlikely.)

@jfs, @Eddi: Thanks for your advice. Shared memory turned out to be the easiest. The OpenTTD source is well documented, so it was easy to find my way around. I'll add the other methods eventually, but for now, this is enough. Only done the map at this stage. Know the vehicles will be much harder, but one step at a time.
Attachments
256_0003.png
256_0003.png (63.62 KiB) Viewed 3157 times
Last edited by HayabusaShinkansen on 24 Feb 2023 00:17, edited 2 times in total.
User avatar
jfs
Tycoon
Tycoon
Posts: 1750
Joined: 08 Jan 2003 23:09
Location: Denmark

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by jfs »

Last night I got an idea about maybe making a general interface to let external applications query OpenTTD for its memory layout of various game objects, such that you could e.g. send a query for "what is the memory address of the Vehicles array" and "what are the offsets of a vehicle's X,Y,Z coordinates in the vehicle structure". I have no idea how feasible that would be, but it might be a fun project too. It could definitely support more use cases for people wanting to make applications to draw advanced statistics for their company finances, or make transit maps of their networks.
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by JGR »

jfs wrote: 24 Feb 2023 09:41 Last night I got an idea about maybe making a general interface to let external applications query OpenTTD for its memory layout of various game objects, such that you could e.g. send a query for "what is the memory address of the Vehicles array" and "what are the offsets of a vehicle's X,Y,Z coordinates in the vehicle structure". I have no idea how feasible that would be, but it might be a fun project too. It could definitely support more use cases for people wanting to make applications to draw advanced statistics for their company finances, or make transit maps of their networks.
This sort of approach adds whole categories of technical and platform-specific difficulties.

Trying to extract statistics from a remote process using debugger mechanisms is much more difficult and unreliable than just asking it to create a savegame (e.g. using the admin port), and then analysing the save file.
Ex TTDPatch Coder
Patch Pack, Github
User avatar
HayabusaShinkansen
Engineer
Engineer
Posts: 10
Joined: 10 Feb 2023 01:04

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by HayabusaShinkansen »

I will eventually add support for a savegame state over the admin port, but I'll also see how I can connect the 3D Viewer to an OpenTTD process in as unobtrusive a way as possible so I can show real-time trains etc. Windows and Linux both have a similar memory sharing API which is quite nice. The terrain map is straight forward, but the vehicles are harder because they're a pool. Synchronization is an issue. I'll explore these and post what I discover.
Last edited by HayabusaShinkansen on 27 Feb 2023 08:38, edited 1 time in total.
smurfix
Engineer
Engineer
Posts: 11
Joined: 29 Sep 2013 13:26

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by smurfix »

Well, what I would do if I was trying to do that (which I totally want to but have less than zero time for) would be to run an embedded Python interpreter as a thread in a "headless" OpenTTD process. You can then easily pass a bunch of accessor functions to this interpreter that let you figure out which sprites in which orientation are where, add a couple of callback hooks to tell the interpreter about major events / movement it's currently interested in, etc.

The idea of reconstructing an ongoing game using a network connection is … um … daunting, You basically need to recreate all of OpenTTD's logic faithfully. While rewriting the game in Python would be a superb achievement, it's also a ton of work which somebody already did, to the tune of ~200'000 lines of C++ code.
User avatar
JGR
Tycoon
Tycoon
Posts: 2557
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by JGR »

You'd be vastly better off forgetting about using Python.

Inserting event logging or query functions into the OpenTTD source is doable but trying to make it work with Python would just be a huge headache for no benefit, especially if your target system is already in C++.
Ex TTDPatch Coder
Patch Pack, Github
User avatar
HayabusaShinkansen
Engineer
Engineer
Posts: 10
Joined: 10 Feb 2023 01:04

Re: Questions for: Godot + OpenTTD = 3D Viewer project

Post by HayabusaShinkansen »

I've coded a lot in C++, Python, and have integrated Python into C++ programs, but I don't think that's necessary here since it adds bloat and compatibility problems. Infamously Python dropped support for Windows 7 IMHO without good reason.

In this case though an easier solution is to use a DLL dynamic link interface. That would let third-party authors write their own plugins/addons/extensions. On startup, OpenTTD would look for such plugins (they are just DLL files with an agreed API), and when game execution hits a part in the code you want to inform the plugins, do the callback by calling the nominated DLL function. The code footprint is small and it's just a few CPU cycles to check. If the user isn't using any DLLs the cost is negligible. With a clean interface, you could conceivably integrate this into OpenTTD's main branch. I've used this technique before in commercial software and it worked well.

And if you really, really want a Python interface, you can implement that as a DLL, without breaking anyone else's compatibility.
You basically need to recreate all of OpenTTD's logic faithfully. While rewriting the game in Python would be a superb achievement, it's also a ton of work which somebody already did, to the tune of ~200'000 lines of C++ code.
Yes, I don't think you could do that. Just not practical. Callbacks are far easier, but appreciate they need to be low overhead and not overcomplicate the rest of OpenTTD.

General report: Found shared memory with OpenTTD's map was easy to implement. Looked briefly at doing callbacks, but that harder to figure out what information needs to be passed so needs more reasearch. Found interfacing to Godot is possible but painful as they have just migrated to a new version, and their addon interface is a PITA.

@zephrys, please let me know where I can find your wear and tear textures (described above). Also a re-upload of zBase dev would be appreciated, since openttdcoop.org is down.
Last edited by HayabusaShinkansen on 07 Apr 2023 05:36, edited 2 times in total.
Post Reply

Return to “OpenTTD Development”

Who is online

Users browsing this forum: No registered users and 17 guests