JGR's Patch Pack

[romazoon]

hello JGR

i think i managed to break the custom bridgehead...i wasn t very cautious, and i managed to replace a bridge wich had a custom bridge head while a tram already started to enter the custom tile. now it s stuck, can t do anything, i can t redo the custom bridge head because it says vehicle is on the bridge, i can t demolish it cause vehicle on the bridge...

good luck smashing that bug

p.s : for the newgrf you will miss, send me a pm, but probably you won t need them as they are not involved in that situation

[KeldorKatarn]

@JGR I already fixed this in my version. Bridge upgrades preserve custom bridgehead for road and signal for rail data. Check my fork for that. Unfortunately, that won't help this player out It can'T be fixed AFTER the fact.

[KeldorKatarn]

@JGR: Btw, is there ANY way to introduce your way of handing savegame compatibility to a version that doesn't have it yet WITHOUT breaking savegame compatibility with a previous version?
Seeing how we might soon all try to be more compatible with OpenTTD's push to GitHub and a compiler farm being prepared for all versions including patch packs... I guess that's something we should talk about?
What worries me a bit is that even the features we share might not be 100% identically implemented and all that?
Thoughts?

[KeldorKatarn]

@JGR: Also... I've been racking my brains over this:

static Industry *PlaceIndustry(IndustryType type, IndustryAvailabilityCallType creation_type, bool try_hard)
{
uint tries = try_hard ? 10000u : 2000u;
for (; tries > 0; tries--) {
Industry *ind = CreateNewIndustry(RandomTile(), type, creation_type);
if (ind != NULL) return ind;
}
return NULL;
}

in IndustryCmd.cpp


That entire thing seems to very inefficient. I noticed that some ECS industries have various complicated layouts and the game is sometimes not capable of finding locations for them (especially the bauxite mines since they need a specially shaped sloped edge on a hill), depending on the map. Also most tourist attraction castles can not be auto-placed at all. They require such specific locations you have to do it by hand.


But even normal industry locations really takes a s*** of time. I once increased these try counts to about double that to make sure on one of my plays the game was able to place bauxite mines during the game better but sometimes the game freezes because of this whenever an industry completely fails to build so I'm going to reduce that in my next version again. I don't like those freezes.

I can't see how this could be done in parallel. Maybe that would work during game creation because form what I know that does never happen in the network but BEFORE clients connect, but during gameplay for new industries being founded that wouldn't work because parallel would be non-deterministic in its outcome (any of the threads might find a good location first) so that would cause desyncs.

Maybe it could be done in a background thread but again.. that might finish at different times...

Caching locations also seems possibly not ideal since expanding cities, other industies being founed and of course the player transforming and building things would quickly invalidate such a cached industry location lookup.

Any ideas? It REALLY bugs me to have "let's just try this a million types and hope for the best" approach in the gode. it's incredibly brute force and not even guaranteed to work and potentially slows a tick down by a lot, and the number of tries already seem to be at the limit of what is even possible to do in a tick without completely freezing everything.

Maybe make it in a background thread, schedule it for a certain future tick and force join in then, doing a pause if necessary like you did to the linkgraph thing? Might that work to synchronize it? The game SHOULD still have enough CPU idle time left to be able to benefit from another thread doing this kind of stuff in the background.

(This is why I hate C++ for developing stuff like this these days btw. It's SUCH an outdated language for everything parallel or async... yuck)

[JGR]

Also btw JGR, I'm about to steal 2 years worth of your patchpack into my version I'd like to thank you for the work you're putting into making small usability changes and importing all kinds of stuff. I would never have the time for half these things. You're one of the few C++ devs out there that I trust enough to just copy & paste most of the code and trust it'll work fine.

That's no problem, I'm glad that it's well received.

Now, I just realised this feature was added in trunk, and I am unable to figure out what it is so please JGR enlighten me. What is "through-load" ?

Really this needs an explanatory GIF. I will look into making one next week perhaps.
In short it's a different way of handling load/unload of long freight trains in short through platforms.
In my opinion it's more realistic and useful than the default behaviour.

You can try it out by turning on the advanced loading mode features setting. It adds an extra platform stop location/mode for freight trains.

hello JGR

i think i managed to break the custom bridgehead...i wasn t very cautious, and i managed to replace a bridge wich had a custom bridge head while a tram already started to enter the custom tile. now it s stuck, can t do anything, i can t redo the custom bridge head because it says vehicle is on the bridge, i can t demolish it cause vehicle on the bridge...

good luck smashing that bug

p.s : for the newgrf you will miss, send me a pm, but probably you won t need them as they are not involved in that situation

@JGR I already fixed this in my version. Bridge upgrades preserve custom bridgehead for road and signal for rail data. Check my fork for that. Unfortunately, that won't help this player out It can'T be fixed AFTER the fact.

I'll look into this shortly.
It is often possible to repair savegames with problems like this, but as it does take time I probably would not bother if you have a recent backup which doesn't have the problem.

@JGR: Btw, is there ANY way to introduce your way of handing savegame compatibility to a version that doesn't have it yet WITHOUT breaking savegame compatibility with a previous version?

It should be possible, though I'll admit that I haven't looked through your savegame code.
Backwards savegame compatibility is something that I was keen on implementing from day 1 so it has influenced some of the design decisions which I've made.

Seeing how we might soon all try to be more compatible with OpenTTD's push to GitHub and a compiler farm being prepared for all versions including patch packs... I guess that's something we should talk about?
What worries me a bit is that even the features we share might not be 100% identically implemented and all that?
Thoughts?

Things like differing bit allocations and enum values can be smoothed over in afterload.cpp.
The bigger problem is that not all features will exist at all or in compatible forms in different patchpacks.

@JGR: Also... I've been racking my brains over this:

static Industry *PlaceIndustry(IndustryType type, IndustryAvailabilityCallType creation_type, bool try_hard)
{
uint tries = try_hard ? 10000u : 2000u;
for (; tries > 0; tries--) {
Industry *ind = CreateNewIndustry(RandomTile(), type, creation_type);
if (ind != NULL) return ind;
}
return NULL;
}

in IndustryCmd.cpp


That entire thing seems to very inefficient. I noticed that some ECS industries have various complicated layouts and the game is sometimes not capable of finding locations for them (especially the bauxite mines since they need a specially shaped sloped edge on a hill), depending on the map. Also most tourist attraction castles can not be auto-placed at all. They require such specific locations you have to do it by hand.


But even normal industry locations really takes a s*** of time. I once increased these try counts to about double that to make sure on one of my plays the game was able to place bauxite mines during the game better but sometimes the game freezes because of this whenever an industry completely fails to build so I'm going to reduce that in my next version again. I don't like those freezes.

I can't see how this could be done in parallel. Maybe that would work during game creation because form what I know that does never happen in the network but BEFORE clients connect, but during gameplay for new industries being founded that wouldn't work because parallel would be non-deterministic in its outcome (any of the threads might find a good location first) so that would cause desyncs.

Maybe it could be done in a background thread but again.. that might finish at different times...

Caching locations also seems possibly not ideal since expanding cities, other industies being founed and of course the player transforming and building things would quickly invalidate such a cached industry location lookup.

Any ideas? It REALLY bugs me to have "let's just try this a million types and hope for the best" approach in the gode. it's incredibly brute force and not even guaranteed to work and potentially slows a tick down by a lot, and the number of tries already seem to be at the limit of what is even possible to do in a tick without completely freezing everything.

Maybe make it in a background thread, schedule it for a certain future tick and force join in then, doing a pause if necessary like you did to the linkgraph thing? Might that work to synchronize it? The game SHOULD still have enough CPU idle time left to be able to benefit from another thread doing this kind of stuff in the background.

(This is why I hate C++ for developing stuff like this these days btw. It's SUCH an outdated language for everything parallel or async... yuck)

I personally wouldn't bother with threads, I doubt that you would get much benefit. Mainly because the the map array and any connected auxiliary bits which might get tested would need to be immutable during any industry placement search, which effectively means that the main thread can't be running. The NewGRF layer is also likely to resist any attempt to make it thread-safe.
I would imagine that there could be room to optimise the test function to reject failures more quickly.
Without reading any more of the code, if the NewGRF layer is called on every test that is probably the main reason why it is slow. It's difficult to see how that could be avoided short of using a JIT to turn the NewGRF code into something which can be executed quickly though.

[romazoon]

no need to repair my savegame ! i was just doing a bug report, i did have a backup and went straight back to it knowing that tram was going to be a problem

[ISA]

In short it's a different way of handling load/unload of long freight trains in short through platforms.

If that's so, thank You very much!

[ino]

Nice. Tried it in many configuration and I still haven't managed to brake this looks-fragile feature yet.

A question though: why not also for passenger? If the reason is realistic, there's actually a train line in Tokyo that does that... (Granted, not exactly the same. Basically a 8-car train pulled into 4-car platform. The first four car is boarded/unboared, then the first 4 cars is detached from the rear, goes to different destination, then the rear 4 cars pull into the station)

And I think I manage one bug:



In other configuration it works as expected though (without conditional, or with next order conditional).

[KeldorKatarn]

Hehehe, yeah stuff like this is why I didn't steal this yet This needs more testing still but I feel it could be really cool. There's quite a few station graphics now which would totally fit for such a through station.

[KeldorKatarn]

I personally wouldn't bother with threads, I doubt that you would get much benefit. Mainly because the the map array and any connected auxiliary bits which might get tested would need to be immutable during any industry placement search, which effectively means that the main thread can't be running. The NewGRF layer is also likely to resist any attempt to make it thread-safe.
I would imagine that there could be room to optimise the test function to reject failures more quickly.
Without reading any more of the code, if the NewGRF layer is called on every test that is probably the main reason why it is slow. It's difficult to see how that could be avoided short of using a JIT to turn the NewGRF code into something which can be executed quickly though.

Yeah you're probably right about the threads. I don't see how to optimize the checks more though. They seem to be already sorted so the one that fails the most often is checked first. Then again, another one might also often fail but the one that fails the most of all is asking the NewGRF if the industry can be placed on a tile.

The way NewGRFs are implemented is something I'm not a huge fan of anyway... that stuff should be done with a standard scripting language that is
a) easy to write (there's a ton of people who have no real grasp over how it all works at all)
b) easy to read
c) possible to JIT compiler for high performance.

That would be a huge undertaking though.

[Alberth]

I don't see how to optimize the checks more though. They seem to be already sorted so the one that fails the most often is checked first. Then again, another one might also often fail but the one that fails the most of all is asking the NewGRF if the industry can be placed on a tile.

I believe the solution to this problem is not by speeding up trying random placements.
The reason for the code to perform random placements is that a NewGRF has a callback "build", which either succeeds (and then the industry exists), or fails ( in which case the industry is not created). There is no "please check if you can built here" call, nor a "why does it fail?" function.

What I think is needed is en extension to the NewGRF spec to allow a NewGRF to express its wishes wrt placement. For example, currently, a large industry is often difficult to place, as the surface needs to be a specific shape over a large number of tiles. The GRF tests and rejects the surface, but it cannot tell what is wrong with it. If it could, OpenTTD could do the check, or even modify the surface to the needs before trying to place the industry.
Such a required shape could also aid manual placement, like with the shore-based industries of FIRS that are quite non-intuitive to place manually. Likely openttd should not change the surface in such a case, but it could highlight a problematic tile, or show "all green" to the user if it's positioned correctly.

The way NewGRFs are implemented is something I'm not a huge fan of anyway... that stuff should be done with a standard scripting language that is
a) easy to write (there's a ton of people who have no real grasp over how it all works at all)
b) easy to read
c) possible to JIT compiler for high performance.

That would be a huge undertaking though.

The simplest solution here is to see the current NFO as compiled output that you don't ever read (most people at least).
grfcodec is basically an 'assembler', and code generators like nml or m4nfo or grfmaker are then 'compilers'. While GRF is slow relative to native C++ code, it's still fast, as it runs as part of the animation loop deciding which graphics to display etc. Likely that limits the number of feasible scripting languages quite dramatically.

[KeldorKatarn]

Every single AAA game out there uses scripting languages for their logic so I beg to differ. Nobody writes game logic in C++ anymore unless that happens to be used as a scripting language like Unreal does it. but that's just the syntax, it's still a scripting language.

I don't claim to know exactly what NewGRFs do or rather how they do it, but all profiling shows they're very slow and a major bottleneck and looking through the forums it's also pretty clear that they're anything but easy to write outside of a group of people that worked with them for decades since their inception.

I'm not bashing anything, I'm just saying this could probably be improved. And the fact that constantly "specs" have to be designed when changing them is also not ideal. Any scripting language would simply define a new function or change one and be done with it. This seems such an elaborate process that any change takes months or years to be finalized. That's just a horrible workflow in my book.

But as I said.. ripping all that out and replacing it would be a huge undertaking. Adding a proper scripting environment isn't exactly easy. But it's the way to go. Every modern engine does it like that. In some engines the scripts are not even precompiled and just lying around as source files, and then compiled right before game start, then cashed for later game runs with checksum and recompiled if necessary.

But again.. future music. For now, NewGRF is indeed the bottleneck and I wouldn't spend too much time on respecing everything, because that'll take forever anyway. It's not like this is the only call that's slow.

[michael blunck]

[...] NewGRFs are implemented is something I'm not a huge fan of anyway... that stuff should be done with a standard scripting language that is
a) easy to write (there's a ton of people who have no real grasp over how it all works at all)
b) easy to read
c) possible to JIT compiler for high performance.

What Josef wrote 10 years ago:

I mean, what gain do you buy with the complicated nfo-language?

Versatility and performance. Because of the tight integration with the game's state engine, the only limit is the imagination of the grf coder, and performance is good because it's byte code that's heavily optimized for the usage patterns that you will need in displaying sprites. In principle, the byte code can even be compiled into machine code, at least for TTDPatch where we work with CPU instructions directly. It might be possible to do that for OTTD as well by adding the required assembly code for given platforms, though.

Saying, in essence, that "in two years the scripted approach will be fast enough" is not really a good argument...

Really, the only problem you seem to have with NFO code is that it's hard to write and even harder to read. There have been several proposals to solve this by writing a compiler from a high-level language to NFO, which is probably a more reachable goal than having to essentially rewrite all the features NFO provides but with a scripted backend instead of the NFO bytecode. Because even if you use LUA, you still have to code all the hooks it offers into the game, after all.

Don't reinvent the wheel just because you want to put a pretty hubcap on it.

regards
Michael

[JGR]

I would doubt that the available manpower exists for development of a replacement scripting environment for GRFs.
Even if it did occur, the existing NewGRFs will still need to be supported indefinitely.

Adding more efficient callback interfaces to the NewGRF spec could work, but the existing (less efficient) ones also need to be supported indefinitely, and existing GRFs are highly unlikely to be upgraded.
In my observation, GRF authors and devs/patchpack authors tend to optimise for different things. Marginal run-time performance improvements at the expense of additional NewGRF development effort/complexity doesn't sound like a vote-winner on the GRF side to me, (especially as the base development difficulty is already high).

That said, OpenTTD does already have a squirrel interpreter environment for AIs and gamescripts.

(Getting the crayons out, if I were to replace NFO, something along the lines of eBPF would seem ideal.)

[Alberth]

Every single AAA game out there uses scripting languages for their logic so I beg to differ.

For high level logic, sure. Now name a few that use scripting to decide the graphics to draw every single frame in the animation, for all things visible. I am pretty sure it's very close to 0, if not actually 0. (Pointing to an animation as a whole does not count!)

Whether doing a callback for an image every single frame is sane is of course also an issue, but openttd does it in this way currently. For fair comparison the scripting language of your choice should also do that.

Nobody writes game logic in C++ anymore unless that happens to be used as a scripting language like Unreal does it. but that's just the syntax, it's still a scripting language.

Even OpenTTD moves away from that, new things are often opened up as gamescript interface, so you can write a game script for your game logic.

I don't claim to know exactly what NewGRFs do or rather how they do it, but all profiling shows they're very slow and a major bottleneck and looking through the forums it's also pretty clear that they're anything but easy to write outside of a group of people that worked with them for decades since their inception.

Profiling is showing you the performance gap between native code and interpreted byte-code, you'd also see that gap with any scripting language, unless it gets translates to real native code in a smart way.

As Patchman said, NewGRF is the byte-code of the graphics engine of OpenTTD, the assembly-language of graphics, so to say.

The big problem of NewGRF byte-code is that it's quite complicated to get into. It's all "action X", without high level overview of what X-es exist, what the purpose is of each of them, and how they work together. It's not unlike the problem of studying assembly language by means of opcodes without a processor model in your head.
However, about 95% of more of the code is very simple, a few "if" statements, and that's it. Mostly a one-liner in a typical scripting language. Th trick is to avoid understanding NFO as a set of seperate actions, and see it as functions.
Almost all NewGRF authors do pretty much just copy/paste of existing code from other projects, which doesn't need deep understanding at first.


Seeing NewGRF as the one and only true way to write graphics code would be sub-optimal imho, much like we don't write assembly language anymore, we write C++, and let the compiler do the conversion. In NewGRF, we have nml and other compilers that give a high-level view of the functions. No need to hack nfo, unless you like NFO bytes, write openttd interface code, or extend one of the compilers.


Before attempting any form of replacement of newgrf by a scripting language, I'd start with an experiment of a scripting language returning the image to draw for a vehicle, for every frame, for all the vehicles running in the game. If that runs smooth, try adding some simple logic based on game year or so. If the game hasn't exploded by then, try a few extreme openttdcoop games for speed comparison. If that works, start thinking about replacement.
I quite doubt however you can be as fast as newgrf is today. Simple straightforward scripting languages using byte-code won't get the job done, as something general purpose is highly unlikely competitive with a language dedicated for the job at hand.

I'm not bashing anything, I'm just saying this could probably be improved. And the fact that constantly "specs" have to be designed when changing them is also not ideal.

Specs are the public interface between the program and the newgrf, ie the set of callbacks, provided properties by openttd, etc. You have to define them to have a clear interface definition between the grf byte-code and the openttd program.

Any scripting language would simply define a new function or change one and be done with it. This seems such an elaborate process that any change takes months or years to be finalized. That's just a horrible workflow in my book.

So how is your newly invented callback function called by openttd at the proper time then? How can you differentiate between grfs that have the new callback and those that don't?

NewGRF is not a main function with a lot of logic where you have to partition work into some functions. It's a large set of callback functions, each function being very short, a few statements only in general.

But as I said.. ripping all that out and replacing it would be a huge undertaking. Adding a proper scripting environment isn't exactly easy. But it's the way to go. Every modern engine does it like that. In some engines the scripts are not even precompiled and just lying around as source files, and then compiled right before game start, then cashed for later game runs with checksum and recompiled if necessary.

Do the experiment, I think you'll be surprised how slow a general purpose scripting language is.

Adding more efficient callback interfaces to the NewGRF spec could work, but the existing (less efficient) ones also need to be supported indefinitely, and existing GRFs are highly unlikely to be upgraded.
In my observation, GRF authors and devs/patchpack authors tend to optimise for different things. Marginal run-time performance improvements at the expense of additional NewGRF development effort/complexity doesn't sound like a vote-winner on the GRF side to me, (especially as the base development difficulty is already high).

It's not more work for a NewGRF author, he already has to specify the exact conditions when an industry can be built, for example. To them, it's just shifting code to a new callback, or transforming it into a new form.

If it also improves user experience, I think many authors would update. Some existing grfs would not, so yep, you can't kill the "repeat random placement attempt" code, unless you drop backwards compability, but it becomes legacy much like all the legacy that already exists.
Many users however eventually play with newer grfs, and get the benefits of them.

[acs121]

I would doubt that the available manpower exists for development of a replacement scripting environment for GRFs.
Even if it did occur, the existing NewGRFs will still need to be supported indefinitely.

Adding more efficient callback interfaces to the NewGRF spec could work, but the existing (less efficient) ones also need to be supported indefinitely, and existing GRFs are highly unlikely to be upgraded.
In my observation, GRF authors and devs/patchpack authors tend to optimise for different things. Marginal run-time performance improvements at the expense of additional NewGRF development effort/complexity doesn't sound like a vote-winner on the GRF side to me, (especially as the base development difficulty is already high).

That said, OpenTTD does already have a squirrel interpreter environment for AIs and gamescripts.

(Getting the crayons out, if I were to replace NFO, something along the lines of eBPF would seem ideal.)

NML was an attempt to replace NFO, however most features of NFO still can't be done yet with NML.

[andythenorth]

NML was an attempt to replace NFO, however most features of NFO still can't be done yet with NML.

It would be super helpful if you could make a list of these (in another thread). Thanks.

[KeldorKatarn]

I appreciate the feedback by some developers here, but as someone who has worked professionally in the gaming industry for several years I think I know a little bit more about what I'm talking about.

Comparing OpenTTDs interpreted scripting API which is incredibly slow to a modern engine that runs precompiled scripts is abit weird.

Also calling what games do with their scripting "high level code only" is a bit missleading. It's the entire game code, everything, including per frames calls. No the rendering engine is not written in that, that's what the game is supposed to figure out on its own. It's already not a good idea to enforce the necessity to ask a NewGRF what frame it wants to render.

The example with animations is showing why the current model is bad, not why any other model wouldn't work. Animations should be predefined and triggered at agreed positions, not handled frame by frame by constantly polling data.
Calling the industries for their current animation frame is one of the biggest slowdowns the game has at the moment on big maps and it's done even if the industry is not on screen AND if you deactivate that you run into desync in multiplayer. It's incredibly unoptimized to force animations to be calculated that nobody sees and why every client in MP has to be in sync with their (hopefully) non-game impacting animations is another weird matter.


And I pretty much don't care what people that had no clue what they were talking about were writing 10 years ago.
I don't make demands here, I said what I thought about the entire NewGRF thing from a professional's perspective, what people do with that is their business, I never said I expect someone to do that just because I say so.

Also I will never understand the attitude of "high level" = "slow", "low level assembler/byte code" = fast.
I'm sorry but that is incredibly wrong. First of all a proper compiler that compiles a scripting language will always write WAY more optimized stuff than anybody here will be able to write in their wildest dreams and if low level is so wonderful I truly wonder why OpenTTD was not written in CPU instructions or at least assembly and instead is now pushing for more C++11. I thought high level was slower...

But those are my final thoughts about this. The topic is leading nowhere.

[acs121]

NML was an attempt to replace NFO, however most features of NFO still can't be done yet with NML.

It would be super helpful if you could make a list of these (in another thread). Thanks.

Are you pretending NML can do everything that NFO can do ?

[andythenorth]

Are you pretending NML can do everything that NFO can do ?

I'm not pretending anything. Your reply comes across like a rather odd response to a simple request for a list.

If you're not able to engage in a constructive conversation about this, you can't help move anything forward.

But those are my final thoughts about this. The topic is leading nowhere.

I think it's useful to critique the current newgrf implementation. I'd rather have a discussion based on more knowledge of the OpenTTD/NewGRF implementation, and not so much generalised approaches from the gaming domain, but eh, it's still good to kick the tyres.

I'm not aware of many people who are both (a) well informed and (b) think the newgrf implementation is a perfect solution.

But I do think this old joke applies here http://wiki.c2.com/?WouldntStartFromHere