Openttd Client load

[Gigigonzalez]

Hello all

Don't worry this isn't another 'Please make Openttd Multithreaded' lol


So we've been running a server for quite a while now which is great.
But we are bumping into an issue where at some point some clients are simply too slow, and the client CPU load is maxing out because of some 'functions'.

The problem we're having is mostly with trains, we can run road vehicles, boats, tons of them, but once the train number goes up, client cpu usage goes up significantly.

What I do know is this: Its not related to pathfinding, more with something else which Developer Rubidium once explained in another Thread.
I did profile the application myself to verify Rubidium's story, just out of curiosity, ofcourse he's right! But who knows it mightve been something else what was causing it.

viewtopic.php?f=32&t=37064&start=60

On the bottom, is where he explains what takes by far the most CPU usage.



Now this is as far as my technical expertise goes unfortunately, and the question isn't how to solve this huge bottleneck in the code itself, which ofcourse would've already been optimized if there was a better way to do so.
But my question is, how do I keep this specific cpu load under control or how to surpress it?

Before we used to play with 8 Trainsets, and I've already reduced it to only 4 but at first glance it didn't seem to affect the cpu load much, though its hard to tell because server load is somewhat different each round.



Questions:

1. Will less (Train) vehicle GRF sets reduce load?

2. Does it also affect road vehicles/trams ships and aircraft?

3. Does the amount of train wagons affect it? I could speculate that having A LOT of wagons behind the train all of those wagons constantly had to 'fetch' data perhaps? or when they make a turn?

4. I assume that speed of the trains also negatively affects the cpu load, thats pure speculation though, but if this would to be true, would enabling Wagon speed limit restrictions kind of surpress the load further?

5. Has map size any direct or indirect effect on this load issue I'm talking about? Or is that totally unrelated to this?

6. Could it be NewGRF specific? Like, some sets have sounds, gearing animation etc, on most you can disable it which I did, could there be 'hidden' stuff that is not visible for the end user on some vehicle newgrf's which occupy a lot of CPU? And if so, how would I be able to discover which sets would be problematic for high CPU load.



Sorry for all the questions, but I'd rather explain properly what I'm talking about (Hopefully)
Thanks in advance.

[Eddi]

the "determine what sprite to draw each tick" bottleneck is not really about how many train GRFs you have loaded, but more about how complicated of a decision chain the GRF uses to actually get to the answer (i.e. how many switches it goes through, and which variables it checks). so some GRFs may have bigger impact than other GRFs. but the only way to completely circumvent this issue is to use no NewGRFs at all, because the fact that it has to check the GRF in the first place causes a significant overhead already. (this applies to all GRFs, not just trains)

[Gigigonzalez]

So reducing the amount of trainsets have no effect? Id imagine if more people use the same trains from the same set that also reduces overhead?

A.nother question might be, why the hell does it have to calculate something even if youre looking at empty water/landscape, makes no sense to me. Must be the way this game is programmed.

because the fact that it has to check the GRF in the first place causes a significant overhead already. (this applies to all GRFs, not just trains)

I assume though that GRFs that are just eyecandy dont have to be redrawn each tick, assuming they do not animate etc...

[planetmaker]

So reducing the amount of trainsets have no effect? Id imagine if more people use the same trains from the same set that also reduces overhead?

A.nother question might be, why the hell does it have to calculate something even if youre looking at empty water/landscape, makes no sense to me. Must be the way this game is programmed.

because the fact that it has to check the GRF in the first place causes a significant overhead already. (this applies to all GRFs, not just trains)

I assume though that GRFs that are just eyecandy dont have to be redrawn each tick, assuming they do not animate etc...

Well, your vehicles continue to move, the industrie processes cargo, trees die and grow, tows destroy and build houses - even without you looking at it...

Rest assured, that OpenTTD only redraws those rectangles where actually something changed on the screen. But even then it needs a check *whether* something should change.

[Gigigonzalez]

Well, at least having compiled with profile guided optimization reduced client load a whole bunch

[PikkaBird]

A.nother question might be, why the hell does it have to calculate something even if youre looking at empty water/landscape, makes no sense to me.

The game definitely gets faster if you zoom in on an empty corner of the map.

NewGRFs aren't "just eye candy": they also include vehicle stats, and those stats can change on the fly through callbacks. A NewGRF could include a vehicle which has three times the running costs on Wednesdays, or goes faster when it's 100% loaded. The game needs to test those conditions and apply the callback results even if the vehicle isn't on the screen.

More complex NewGRFs are going to have a higher overhead. I would definitely avoid UKRS2.

[Gigigonzalez]

A.nother question might be, why the hell does it have to calculate something even if youre looking at empty water/landscape, makes no sense to me.

The game definitely gets faster if you zoom in on an empty corner of the map.

NewGRFs aren't "just eye candy": they also include vehicle stats, and those stats can change on the fly through callbacks. A NewGRF could include a vehicle which has three times the running costs on Wednesdays, or goes faster when it's 100% loaded. The game needs to test those conditions and apply the callback results even if the vehicle isn't on the screen.

More complex NewGRFs are going to have a higher overhead. I would definitely avoid UKRS2.

We do use UKRS2 but with all the configurable bells and whistles disabled

atm 550 trains, 1250 road vehs, 330 ships on big 4k x 4k map

average client cpu load 1.43 (out of 25)

Its become A LOT better since we recompiled with icpc

[Eddi]

Id imagine if more people use the same trains from the same set that also reduces overhead?

no. none of the callback results from one wagon may be used for another wagon, because all the surroundings may be completely different (front engine, current load, animation state, wagons before and after it, randomization, ...) and openttd has no way to check this without actually running the callback.

so unless your bottleneck is the spritecache, the situation will not improve.

[Gigigonzalez]

Too bad then the trainset creators never decided to deliver a set versin with all the special stuff removed, just the basic graphics and stats, like a stripped down version. The big sets like nars, ukrs, dutch could definately use that!

Or just a parameter to disable anything but base functionality. Couldnt a global toggle in openttd be able to do that?

[Alberth]

I think it's for the same reason you don't strip the server setup to a default game at 64x64, and nothing else.

You spent a big effort in getting it set up just the way you and your users like it, and it would be a big waste to throw it all in the bin.

[FooBar]

Too bad then the trainset creators never decided to deliver a set versin with all the special stuff removed

Simple. It's too much work. I'm not interested in spending a lot of time on something I'm never going to use.
But luckily in the case of the Dutch set at least, the source code is freely available. So anyone who wants to make a stripped version is welcome to do that.

[Transportman]

Or just a parameter to disable anything but base functionality. Couldnt a global toggle in openttd be able to do that?

And how would OpenTTD know what to do and what to skip? There is no way of knowing. For example, in the Dutch Trainset you can set liveries. Not essential, so OpenTTD could skip the whole livery selection switch. But now it wants to draw the vehicle on screen and set the properties of the vehicle, and that depends on the livery selected. But how would OpenTTD know which livery is set? It skipped that part as it is not essential. Should it default then to the default value or something? Then it basically already made the calls needed, but just discards the result (and the default value might not always be the wanted value or be consistent across multiple switches that depend on the same variables).

Same goes for the set authors, all that basic and special stuff is one package. It is very difficult to code it then to only do the basic stuff, as everything is linked directly and indirectly.

But if you could show that solving switch X or determining variable Y is taking a lot of performance (and with a lot I really mean a lot), then you have something, then you could move people to change things. Nobody is going to do any big changes to their code unless there is a good reason to do that, and changing code because of some performance bottleneck is something that is a big job (as it could affect the whole code and not just some small part) and thus needs some very good reasons.

Too bad then the trainset creators never decided to deliver a set versin with all the special stuff removed

Simple. It's too much work. I'm not interested in spending a lot of time on something I'm never going to use.
But luckily in the case of the Dutch set at least, the source code is freely available. So anyone who wants to make a stripped version is welcome to do that.

And how much would you gain in performance? Is determining the result of every switch-block the same performance-wise? Is figuring out which cargo_subtype is used (for the livery) more demanding than determining position_in_vehid_chain (for graphics depending on location in the vehicle)? And does OpenTTD store the values of variables for re-use in other chains also depending on that same variable? I think the actual performance gain (at least for the Dutch Trainset) would be poor compared to the effort it would take to create such a stripped set.

[Alberth]

TrueBrain recently ran a profiler, and found the NewGRF evaluation problem. It's in the bug tracker somewhere.

Individual switches are not the problem. Just the need to resolve things through switches etc, is.
In other words, the startup costs to evaluate the newgrf switches are bigger than the evaluation itself. Reducing switch evaluation costs can thus only be done in a significant manner by throwing away all switches.

[Transportman]

TrueBrain recently ran a profiler, and found the NewGRF evaluation problem. It's in the bug tracker somewhere.

Do you mean this bug? Or are you talking about something else?

Individual switches are not the problem. Just the need to resolve things through switches etc, is.
In other words, the startup costs to evaluate the newgrf switches are bigger than the evaluation itself. Reducing switch evaluation costs can thus only be done in a significant manner by throwing away all switches.

Clear, so the suggestion from Gigigonzales would not actually help increase performance for most sets, unless they would be able to remove all switch-blocks from the code (which would break all those sets completely)?

[Alberth]

TrueBrain recently ran a profiler, and found the NewGRF evaluation problem. It's in the bug tracker somewhere.

Do you mean this bug? Or are you talking about something else?

Given that the bug you refer to is the only open bug by true brain, it must be

Individual switches are not the problem. Just the need to resolve things through switches etc, is.
In other words, the startup costs to evaluate the newgrf switches are bigger than the evaluation itself. Reducing switch evaluation costs can thus only be done in a significant manner by throwing away all switches.

Clear, so the suggestion from Gigigonzales would not actually help increase performance for most sets, unless they would be able to remove all switch-blocks from the code (which would break all those sets completely)?

You pay startup costs for each evaluation separately. If you have switches that are not run very often, you could consider keeping them.
My main point was that looking at individual switches is not going to give you good results fast. If you have a callback with 2 switches, and you throw away one of them, you will gain much less than 50% which you naively would expect. On the other hand, it will be more than 0% of course. Less code to run is always faster.

To me, Gigigonzales sounded like he was suggesting throwing away all switches. I have no idea how "basic" the result is going to be, if you do that. It won't be very pretty, I guess.

[Gigigonzalez]

Thanks for all the responses.

Are you saying there is a 'bug' or something that could be better optimized?

The patch on the site were the bug is posted on seems interesting, but if it breaks all kinds of stuff then

[Alberth]

Are you saying there is a 'bug' or something that could be better optimized?

Perhaps, and no doubt some smart people have looked into the problem already.
The fact that it's not solved indicates it's at least a non-trivial issue.

I wouldn't call it a bug. We are trying to push a s*** of stuff onto a finite resource. If you look at the original game, and see where we are now, it's amazing how much you can do.
Adding more stuff however takes exponentially more effort. The question is when does it stop being useful.

The patch on the site were the bug is posted on seems interesting, but if it breaks all kinds of stuff then

That is probably why it's in the bug tracker instead of in the game source code