Big AI test game

[Steffl]

I started a AI test game today to see myself which AI is the best.

The AIs are:

Acela Express v1
Admiral AI v25
AIAI delta 3 (v15)
CluelessPlus v22
Convoy v11
Denver & Rio Grande v1
Mogul AI v5898
NoCab v495
OtviAI v312
PAXLink v14
RoadRunner v4
Rondje om de Kerk v313
SimpleAI v4
Trans v100307

The AI's where set to start 1 day after each other. In game it takes longer, but all where started within the first year.
The settings are hard, map size is 512*512 in temperate climate. The advanced settings are my custom settings. If someone thinks an AI has a problem with something he can watch it, I will add it to the attachment.

openttd.cfg

(9.2 KiB) Downloaded 229 times

After 2 years:

Human, 1st Jan 1952.png

(210.49 KiB) Downloaded 2 times

AITest Year 2.sav

(318 KiB) Downloaded 216 times

Rondje, NoCab, AdmiralAI and AIAI had the best start, the others are still thinking what they should do

[Steffl]

After 5 years:

Human, 1st Jan 1955.png

(200.82 KiB) Downloaded 2 times

Nocab and Rondje are fighting for the first place, AIAI and Admiral fell back a little bit. The slow starting RoadRunner AI is now on place 5. The first victim in this hard world is AcelaExpress. It got already a profitable bus connection, but then it runs out of money. I think it would need a loan management to work better.



After 10 years:

Human, 1st Jan 1960.png

(210.64 KiB) Downloaded 2 times

AITest Year 10.sav

The save game after 10 years

(466.79 KiB) Downloaded 208 times

Rondje now has the best company value, although Nocab has a higher income. Admiral AI seems to have problems, so RoadRunner now is on 4. place.
PAXLink and Acela Express had gone bankrupt, Denver & Rio Grande had built some 90° rail curves, but this isn't allowed in this game.

[Steffl]

After 20 years:

Human, 1st Jan 1970.png

(218.62 KiB) Downloaded 2 times

graphs without the first 4 companies:

Human, 1st Jan 1970#1.png

(221.65 KiB) Downloaded 2 times

AITest Year 20.sav

(636.49 KiB) Downloaded 238 times

Now AIAI almost overtook Rondje in company value. Rondje sold many vehicles before, so its income decreased. I made a second screenshot to show the companies in the second row. OTVI, AdmiralAI, SimpleAI, Trans and Convoy are making a good profit. Clueless and Mogul AI are fighting to survive but made a little profit too and Acela Express and Denver & Rio Grande got bancrupt short before screenshot.

[Zuu]

Nice work there. I think the profit graph would be more useful than the income graph. In the income graph a company with high costs and high incomes would show up good while it might not make any profit.

[Attila7]

I would view any test that includes Rondje as a participant as severely skewed if not downright meaningless.

This AI, while cute and a programming accomplishment in its own right, demonstrates some of the things that is wrong with OpenTTD's design and exploits each of these design flaws to the fullest.

Letting this AI loose in a test mostly made up of RV running AIs is like letting a fox loose in the hen-house and see which hen survives the longest. Pointless.

In case you are not aware:

1. Rondje waits for another AI to build a road connection, then uses it to compete with the AI that built the road. This is design flaw #1, as any AI that uses another AI's road should pay a toll of some sort.

2. Rondje builds a vehicle at one end of the route, delivers the cargo and then sells the vehicle at the delivery end. This is repeated for every vehicle on every route. This behavior takes advantage of two design flaws in the game.

2a) Vehicle depreciation is too small, so Rondje gets a lot of money back if the vehicle is sold after one trip. This, of course, saves the time not having to return to pick up cargo. This is a design flaw also because it rewards AIs that overbuy vehicles for a route and just sell them after a year or so instead of properly balancing their needs.

2b) When a vehicle that is less than 3 years old enters a station, the station rating goes much higher than if the vehicle were older than 3 years. The design flaw here is that cargo is divided between competitors based on this rating and 2/3 of the cargo will go to the better rated station even if the difference is just 1%. Rondje always has new vehicles, so it automatically kills its competition by taking most of a route's cargo.

The developers really ought to look at plugging these exploits or all AIs will likely use one form or another, even if not in such a purely exploitative way as Rondje does.

[Lord Aro]

That's not entirely fair...
Rondje was created in an attempt to win the TJIP tournament (is that right?)
The goal of this was to achieve the highest company value, by any means necessary. Rondje won the tournament, and so they released the AI for others to admire
The AI had done its job, and so while it may not be the nicest of AIs, it shouldn't be objected to in that way

[Kogut]

And 2 exploits are fixed
- payment on start of unloading - Rondje was able to sell vehicle that was almost fully loaded with sth and get money for transport
- no time limit for AI initialization - Rondje was designed to preprocess map on loading - what resulted in very long game freezes on ai start.
1) What is killer in fight with train/air ai
2) What is called "one way vehicles" and blocked on part of multiplayer servers

[Steffl]

If you think no other AI has a chance against Rondje then you will be surprised when you look at the next years.

After 30 years:

Human, 1st Jan 1980.png

(241.7 KiB) Downloaded 2 times

After 40 years:

Human, 1st Jan 1990.png

(248.04 KiB) Downloaded 2 times

And after 50 years:

Human, 1st Jan 2000.png

(250.21 KiB) Downloaded 2 times

Now game is very slow. I think there is a reason why it usually allows only 500 vehicles for a company I never stopped the game and there are only 50 years over. AIAI has alone almost 2000 vehicles, Nocab another 1000. I will run the test till tomorrow and look who is best then, but I think the only way to beat AIAI is, when it would crash or something.
AIAI and Nocab have a big distance to the other AI's. Roundje has still a huge company value, but it doesn't get much higher anymore. The next AI is RoadRunner. It has over 350 vehicles, but I think they could make more profit. The following are Simple AI, Trans, Convoy and Clueless.
Simple AI has 15 trains und 70 road vehicles, Trans has 60 ships and 7 planes(and 24 airports), then Convoy with 167 busses. Convoy seems to have problems now, because its income is sinking and the newest bus it has is already 18 years old. Cluesless Plus has 125 road vehicles. AdmiralAI is the next with 65 road vehicles and 3 planes. I think it has problems with the game settings too, because it has many rail stations but no train. This AI's all playes since start in 1950 and don't got bankrupt. The other 5 got bankrupt and restarted. But no AI crashed because of a coding error.

[Steffl]

Graphs without the first 2 companies and with operating profit graph:

Human, 1st Jan 2000#3.png

(186.09 KiB) Downloaded 2 times

Savegames:

AITest Year 30.sav

(748.81 KiB) Downloaded 208 times

AITest Year 50.sav

(895.06 KiB) Downloaded 254 times

And sorry that you have to open all images separately. I tried to show it direct in my post, but it didn't work. I think they are to big. Sorry too for my bad English. I learned it in school, but it's not the same as if you speak English naturally. I hope you understand what I ever wanted to say you.

[Attila7]

The AI had done its job, and so while it may not be the nicest of AIs, it shouldn't be objected to in that way

I don't object to Rondje

I object to it being in a test to evaluate a set of AIs. The one Rondje goes after - usually the one that is best at building RV routes - is the one that will perform poorly. How is this test useful?

[Steffl]

Here is the final result after 75 years:

Human, 1st Jan 2025.png

(229.34 KiB) Downloaded 2 times

here without AIAI and Nocab:

Human, 1st Jan 2025#1.png

(230.69 KiB) Downloaded 2 times

and the savegame:

AITest Year 75.sav

(1.01 MiB) Downloaded 211 times

The Winner of this test game is AIAI, the second place goes to NoCab and on third place there's SimpleAI.

[Brumi]

Again, congratulations to Kogut and AIAI!
It's quite surprising for me that SimpleAI did well in such a strong field... And I wonder what happened to AdmiralAI here
It would be nice to include the real old AI in these games, to see how much progress we've made since then

I took some screenshots from the final savegame:

Human, 20. febr. 2025.png (49 KiB) Viewed 5000 times

TransAI's ships with the original ship pathfinder have trouble getting to their destination, but despite this Trans was doing quite well

Human, 2nd Mar 2025.png (292.75 KiB) Viewed 1279 times

The usual screenshot from these AI games, not the longest queue in the OpenTTD world, but already quite annoying for the drivers...

Human, 9th Mar 2025.png

(254.14 KiB) Downloaded 2 times

Getting a little oldAI-ish, although this was a collaboration of almost all AIs in the game.

And one thing about Rondje:
It's not unbeatable with road vehicles, some time ago I did a test with Rondje against a road vehicle-only NoCAB. They were going head to head in the first years, but later NoCAB took the clear lead and beat Rondje.

[fanioz]

The Winner of this test game is AIAI, the second place goes to NoCab and on third place there's SimpleAI.

Hurray

It's quite surprising for me that SimpleAI did well in such a strong field...

Because it was just 'SimpleAI' ?

TransAI's ships with the original ship pathfinder have trouble getting to their destination, but despite this Trans was doing quite well

I've wrote the version that use buoy, but didn't have enough time to test yet.


Thanks to Steffl for the tournament run.