Multi-core support please!

[Eddi]

as i said, im stupid if it comes to programming, and personally: why did OS writers not implement a generic multicore support for programmes?

about the same reason why they didn't invent a way so you can disassemble your car, get on the train, and when you get off the train, you can reassemble your car again to go on with your journey.

[mastamage22]

as i said, im stupid if it comes to programming, and personally: why did OS writers not implement a generic multicore support for programmes?

about the same reason why they didn't invent a way so you can disassemble your car, get on the train, and when you get off the train, you can reassemble your car again to go on with your journey.

i think of it a bit different: you design a car, but everyone needs to bring along their own stick for the gearbox, without the stick you cant shift the gear up, making everyone without a stick driving 10 km/h at max

[Eddi]

ok, let's rephrase it this way: "why do car-producers not build in support so that the driver has additional hands so he can control the steering wheel, the gear-stick, the radio and the cell phone at the same time?"

[Rubidium]

Why don't car manufacturers make cars with 8 engines? That would make the car 8 times as fast!

Good luck with the 8 clutches, 8 accelerators and 8 gear sticks though.

For what it's worth, posts in this thread are worthless unless you have proof that a certain suggestion does work and has no adverse effect, i.e. you have a working patch.

[Eddi]

errr... that example is bad. (some) cars have 8 engines, but they call them cylinders...

[Rubidium]

errr... that example is bad. (some) cars have 8 engines, but they call them cylinders...

Just like a single core CPU has multiple ALUs (arithmetic logic units) that are used by OpenTTD. So I'd equate your cylinders to these ALUs.

Anyhow, an aircraft can easily have 4 or even 6 engines, so why can't a car?

[andythenorth]

For what it's worth, posts in this thread are worthless unless you have proof that a certain suggestion does work and has no adverse effect, i.e. you have a working patch.

Err....no? All we have to is find the right metaphor, and this problem is solved.

[mastamage22]

Why don't car manufacturers make cars with 8 engines? That would make the car 8 times as fast!

Good luck with the 8 clutches, 8 accelerators and 8 gear sticks though.

For what it's worth, posts in this thread are worthless unless you have proof that a certain suggestion does work and has no adverse effect, i.e. you have a working patch.

first of all, a car with 8 V8 engines would have 1 clutch, 1 gearbox and 1 gearstick, if their where 8 gearsticks.... you simply transform your car into a projectile launcher.


back too the basics: the fact that microsoft is a band of douchebags, which did not put in multicore support for programmes in, causes the REAL programmers, like the devs of openttd/ supreme commander and so on( also devs of programes with multicore support), waste otherwise useful time on researching something that should have been implemented in the operating system, and NOT into programmes running on the OS!

why should it be OS based?
- every programmer programs different, so there are loads of different multicore support codes, some of them are even dangerous for the system, gta4 is one of the dangerous ones for overriding the overheat failsafe
- waste of diskspace if more than 1 program has multicore support
- i want microsoft to prove themselves, none of their programes have proper multicore support, they probably are incompetent.

google "STARS! game" for proof, please note that my laptop is a weird system and it might run stuff in a different way than any other system

supreme commander 1 and FA have a third party app that makes it multicore supporting, this program is called core optimizer

and on the other hand, if there is spare time offcourse, recompile to make it multicore supporting?
- when i make in-game a long train-line, the first one is always a mess, so i break it down and rebuilt it from scratch. a certain developer did that with all of his sequel games.


back to the beginning of the post, a car can have only 1 control system on 1 line, having multiple will always make the car unstable, 1 control system(the gearbox and stick) will force the multiple engines into a synchronous state, even if they stand parallel. translate this to the cpu cores: 1 cpu core is acts as the controller and compiles all the data while the other calculate everything. the controller spreads the work evenly over the other cores. in the near future, when ASML is finished testing their new machines, we will have chips with 6 effective sides opposed to the 1 effective side the chips have now. this gives more bandwidth, but also more heat generation, so there might be more cores coming, but they will have less speed per core.

[giddorah2]

Uhm... I'm sorry... But Windows does have multicore support.

[Hyronymus]

Please, maybe (I'm not making promises) multicore support for OpenTTD actually reaches a viable point when the discussion on what it is compares to stops? It's obvious advantages can be named, shouldn't the focus be on how to achieve those advantages then?

[Alberth]

Actually the game runs multi-core, game save and a few other things are done in a separate thread.
People do not perceive it as such since there is still one big job, namely "move the game forward in time" that dominates all other things. This job cannot be split in a beneficial way. Several very smart people have tried, and it gets nowhere.

People posting here only see the outside, many seemingly independent vehicles moving around, and a single core in use. They want to run a bigger game (more vehicles etc), and there you have the obvious solution: use more cores.
What they don't realize is that the game shows a fake world; vehicles are not independently moving around. It is all connected through the common world map.

Unfortunately, it is hard to explain that technical reality does not match the visual perception. They refuse to believe it, up to the point that we must be idiots, and my doctoral degree on computer science has no meaning at all.


I don't believe OpenTTD can be made multi-core in a beneficial way in its current form.
If you are prepared to drop functionality like multi-player, there is room. It is debatable whether such a variant still qualifies as being OpenTTD.
Another option is to rewrite the game from scratch and take multi-core into account from the start. If you do that, I see no advantage in rebuilding OpenTTD, you might just as well make a different transport/build game.

[JacobD88]

Well explained Alberth

With regards to those wanting to push the advantages of a multicore system as much as possible (though for negligible benefit) by moving OTTD to run on a single dedicated core i wrote a tutorial some time ago on this here:

Creating a dedicated core for OTTD

And that's about as good as it will get; bear in mind however, that forcing the executable to run on only one core will also force the multi-threaded parts of OTTD to also be on that core; which is why it's dubious as to whether or not there is any benefit to doing this.

That said, on some systems, particularly Laptop CPUs and Early Multi-Cores i did notice some improvement by moving everything to a tertiary core that wasn't being abused by windows or other processes.

Modern Multi-Cores such as the i7 or new AMD chips, there's no noticeable difference as to how things run (Unless you are using a patchpack with super large maps or 255 height levels).
Also Windows 7 handles everything a lot better than XP, so probably no point for Windows 7 users either

If you know your HEX then you can also set OTTD to use a range of cores by adding all the "0xn" mask values of each core up and inputting the total as a new "0xn" value for the batch file listed on the thread...
I've not tested this yet... So haven't a clue if it makes any difference to users with 8,12 or 16 core PC's...

For example:

Code: Select all

    0x01 = 1 = CPU0
    0x02 = 2 = CPU1
    0x04 = 4 = CPU2
    0x08 = 8 = CPU3
    0x10 = 16 = CPU4
    0x20 = 32 = CPU5
    0x40 = 64 = CPU6
    0x80 = 128 = CPU7

Therefore you would use, to set OTTD, for the first four cores only in an 8-Core system:

Code: Select all

0x0f

= 15 = CPU0, CPU1, CPU2, and CPU3 (1+2+4+8 = 15)


OR MORE SENSIBLY TO USE THE LAST FOUR CORES OF AN 8-CORE SYSTEM

Code: Select all

0x96

= 96 = CPU4, CPU5, CPU6, and CPU7 (10+20+40+80 = 96)

[Arie-]

Also for the person ranting on MS for "not supporting" multiple cores. A "simple" problem often used in programming courses to illustrate problems with multiple processes accessing the same resources: The Dining Philosophers Problem. I'm not familiar with the OpenTTD internals, but I can imagine loading different vehicles with the same resources will definitely give problems when multiple threads access the same resources at the same time. Another example would be two trains accessing a block at the same time through different signals. Managing the data being used in different processes is just something what has to be orchestrated within the specific application, not by an OS.

[Samu]

... and to know that my FX-8150 single thread performance is worse than that of a Phenom II... :/

[planetmaker]

Also for the person ranting on MS for "not supporting" multiple cores. A "simple" problem often used in programming courses to illustrate problems with multiple processes accessing the same resources: The Dining Philosophers Problem. I'm not familiar with the OpenTTD internals, but I can imagine loading different vehicles with the same resources will definitely give problems when multiple threads access the same resources at the same time. Another example would be two trains accessing a block at the same time through different signals. Managing the data being used in different processes is just something what has to be orchestrated within the specific application, not by an OS.

you hit the nail(s) pretty much on the spot. And in multiplayer the orchestration has to be synchronized on all clients or we get - you might guess it - a desync. Thus any improvement to the issue must involve that all clients do exactly the same things in the same order, independent of hardware and OS being used.

[Bazhosh]

I actually don't see a problem with multiplayer desyncs, if you organize your work with a proper pipeline.

For example:
Lets say we have four cores and they do pathfinding for four vehicles at the same time.
[1][2][3][4]
------------->

Now, [1] can finish without problems.
If there is a dependency between [1] and [2], [2] must look if [1] used some of [2]'s path. If so, recompute all after [1].
Simple and deterministic behavior, but with a lot of speed potential (imho).

[Lemike]

Imagine four trains which are about to enter a block with shared tracks at the same time. In a 1-thread pathfinding you could simply generate a random number in between zero and one. If the number turns out to be lower then 0.25, the first train gets a green signal, if it's in between 0.25 and 0.5, the second one has green light and so on. You can not have this operation split into multiple threads (or cores). All you could do is having another "path decicion to take" beeing calculated in another thread _but_ you have to make sure those two incidents are not at all related to each others. It might turn out that finding out whether or not two incidents interfere with each others takes longer then actually doing the pathing in the first hand.

please correct me if I'm wrong

[YNM]

For example:
Lets say we have four cores and they do pathfinding for four vehicles at the same time.
[1][2][3][4]
------------->

Now, [1] can finish without problems.
If there is a dependency between [1] and [2], [2] must look if [1] used some of [2]'s path. If so, recompute all after [1].
Simple and deterministic behavior, but with a lot of speed potential (imho).

I think that'd instead increase the use of cores, wthout significantly reduce time ? I mean, Isn't that's just the same as if core 1 works alone, by means that core 2,3,4 wait for core 1 results ? Even that could echo to core 3 and 4 waiting for core 2, core 4 waiting for core 3. Also, the complexity increase when you have another train to fill just the trains left (look at #openttdcoop ProZone games).

What I'm thinking is easy to apply is to make graph calculation (cargodist) be calculated on a separate core. So at max OpenTTD will utilize 3 cores for the movement, graphics, then graphs.

[Eddi]

cargodist already uses a separate thread for the graph calculation.

[Alberth]

I actually don't see a problem with multiplayer desyncs, if you organize your work with a proper pipeline.

For example:
Lets say we have four cores and they do pathfinding for four vehicles at the same time.
[1][2][3][4]
------------->

Now, [1] can finish without problems.
If there is a dependency between [1] and [2], [2] must look if [1] used some of [2]'s path. If so, recompute all after [1].
Simple and deterministic behavior, but with a lot of speed potential (imho).

It looks so simple and elegant at first sight.

Now please find out how to detect "If there is a dependency between [1] and [2]" cheaply. Doing a walk over all elements of [1] (and testing if they are in [2]), and vice versa is too costly. In that time, you can roughly do 2 more path findings (ok, pessimistically, say 1 other pathfinding).

Your problem is bigger. [1] and [3], [1] and [4], [2] and [3], [2] and [4], and [3] and [4] may also have dependencies. Ie you need to do 5 comparisons afterwards to check whether your 4 paths are valid, which is roughly equivalent to 5 more path findings.

The second problem that you have is writing of the result of [1]. The map structure is shared, so how do you write a result while other path findings are running?
Locking the map is the generally accepted solution, but since path finding is map-bound, adding a lock means you slow down to using a single core (namely the core that has access to the map). If you also add the overhead of getting and releasing the lock, you are slower than the sequential case.

Writing the data without locking has the problem that other path finders may read partially written data, leading to problems like crashes because values of the map are wildly out of range.