AI Ranking Site

Discuss the new AI features ("NoAI") introduced into OpenTTD 0.7, allowing you to implement custom AIs, and the new Game Scripts available in OpenTTD 1.2 and higher.

Moderator: OpenTTD Developers

Post Reply
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

AI Ranking Site

Post by mousetail »

Over the years OpenTTD has gained a lot of AI's from various contributes with various strategies. The quality and stability of these AI's varies a lot, some crash after a few years, others just do nothing, others go bankrupt fast. This makes choosing an AI difficult, especially for new players. It is no fun to carefully plan your AI-beating strategy only to have it crash after 10 years.

This could be better.

I wanted to classify AI's to figure out which ones are solid, so that I can figure out which ones are the best. For this purpose, I setup a server to run AI matches 24/7 and analyze the results. Eventually I use this to rank all the AI's so you can see which are the best. Also I aim to help AI authors to debug their AI's.

The site can be found at https://openttd.mousetail.nl/

Image
Screenshot.png
(74.5 KiB) Not downloaded yet
A lot more matches will need to be played before the ranking stabilizes, so don't hold your breath. Also the site is still work in progress, there might still be errors and rough edges and spelling mistakes etc.

Let me know of any bugs you find or suggestions you might have.
my AI: CPU, SnakeAI
LaChupacabra
Route Supervisor
Route Supervisor
Posts: 385
Joined: 08 Nov 2019 23:54

Re: AI Ranking Site

Post by LaChupacabra »

A good idea with this ranking, only the methodology is not entirely accurate. :wink: The problem is that there are so many settings and variants of the game that with each game the AI ​​rating can be completely different. For example, it is enough to add an AI with planes for such OTVI and turn on maintenance costs. AI with planes will go bankrupt every now and then, and OTVI right behind it because it has such a silly habit to buy out every failing company. I was also surprised by the distant position of ShipAI on your list, which in my opinion is one of the most competitive opponents.
Generally, there are several basic elements that significantly affect the AI ​​rating:
1. Maintenance costs: on /off
2. Cargodist for passengers: on / off
3. Additions - for example, many AI cannot use articulated vehicles, while others do not know where to drive when the industrial set is changed.
4. Gameplay / construction method - sometimes AI is very annoying, which steals factory products, build railroad crossings and destroy vehicles to other companies, or, for example, like your CPU, clutters the map with thousands of signs, which completely destroys the game. Then it doesn't matter what income this AI earns.

At one time I played for AI testing. I even made a description to many of them. A little for personal use to know in the future what not to use, but also after finishing I wanted to post the full list on another knowed for me forum of this game that I have known for years. Definitely the profits that AI generated was not the only criterion for its assessment. For example, I will put a description of one of them: AdmiralAI - one of the best AIs. Maybe it will be useful to you in some way.:)

AI: Admiral AI
Version: 25
Overall rating: 4-/5
Activity: 3/5
Stability: 3/5
Competitiveness / Revenue: 3/5
Includes infrastructure costs: No
Compatibility with industrial sets (e.g. FIRS): 4/5
Construction aesthetics: 3/5

Buses: YES (can be turned off)
Trucks: YES (can be turned off)
Trams: YES
Composite vehicles (semi-trailer trucks, articulated buses): YES
Trains: YES (can be turned off)
Ships: NO
Planes: YES (can be turned off)
Helicopters: NO

Description:
Nice AI, using all means of transport. It deals with rail transport quite interestingly. His individual connections do not look very interesting, because the lines in both directions usually run a completely separate route. However, after some time, a rather interesting picture emerges from the chaos of its interconnected routes. As it turns out, this style of construction is unrivaled on very tight maps.

Advantages:
  + generates relatively high revenues
  + transports all loads including machines
  + chaotic construction style that works on smaller maps
  + builds industrial stations from the ISR set
  + many options in the settings allowing you to choose the means of transport with which you will build connections
  + does not freeze, but ... it has other problems

Disadvantages:
  - does not include infrastructure maintenance costs (should not use aircraft when costs are included)
  - creates too short connections for his income to pose any threat
  - almost as many airports as airplanes - thousands of passengers are waiting at most airports, but sooner will build another airport than buy a new plane
  - sometimes has fits of madness, during which he terraforms randomly selected areas spending almost all the accumulated funds for this purpose
  - other times he builds railway stations anywhere he can, which he does not connect or delete - these remain unused on the map
I am sorry for may English. I know is bed.
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

Re: AI Ranking Site

Post by mousetail »

I know profit is not the only possible metric, but I had to choose one. For this reason I included many other statistics about the matches so that you can get a glimpse of the behavior other than just who ranks the best. "Best" is subjective.

Multiple rankings for different variations of game rules are planned, with or without newGRFs, with or without infrastructure maintenance, etc. However, this will massively increase the amount of computing power necessary to create the ranking so I'm waiting until the "default" preset ranking has stabilized.

I like your reviews. Are they all published somewhere? Perhaps, with your permission, they could also be posted in the AI descriptions on the website.
my AI: CPU, SnakeAI
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: AI Ranking Site

Post by planetmaker »

I suggest to create a set of different maps and re-use those in all tests. It will definitely be interesting to share these starting maps / scenarios in order to have comparable results also in the future. Parameters to vary between maps could be (among others):
* island-like map vs nearly-all-land
* very hilly vs. very flat map
* small vs. big map
* choice of selected NewGRF
* choice of starting year and climate
* choice of certain parameters like cargo weight, maintenance cost, cargodist...

Without availability of the test maps being used, the results remain highly subjective and non-repeatable. And AI authors don't even have something to benchmark their AI against
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

Re: AI Ranking Site

Post by mousetail »

@Planetmaker
Yes, those are soom good variants for presets.

You can download the save games from the site so AI authors can debug what went wrong.

I'm not sure if standardizing the maps is a good idea. It might lead to over-fitting where one AI happens to be good on a certain map but is not very strong overall. Generating random maps every time makes sure that there is no overspecialization.
my AI: CPU, SnakeAI
User avatar
planetmaker
OpenTTD Developer
OpenTTD Developer
Posts: 9432
Joined: 07 Nov 2007 22:44
Location: Sol d

Re: AI Ranking Site

Post by planetmaker »

mousetail wrote: 12 Dec 2019 13:40 I'm not sure if standardizing the maps is a good idea. It might lead to over-fitting where one AI happens to be good on a certain map but is not very strong overall. Generating random maps every time makes sure that there is no overspecialization.
Oh, I'm not arguing for having only one map. But having like a dozen maps which cover the above points (and more). But using those identically for all AI and also for challanges to compare AI evolution. The latter, sadly, has been pretty stagnant over the recent years.
LaChupacabra
Route Supervisor
Route Supervisor
Posts: 385
Joined: 08 Nov 2019 23:54

Re: AI Ranking Site

Post by LaChupacabra »

mousetail wrote: 12 Dec 2019 13:40 I'm not sure if standardizing the maps is a good idea. It might lead to over-fitting where one AI happens to be good on a certain map but is not very strong overall. Generating random maps every time makes sure that there is no overspecialization.
In my opinion, developing such template scenarios would be a better solution. This would give you the possibility to more objectively compare AI and see differences in performance. Random maps, each time different, especially different for each AI, do not give such a possibility.
From these conditions, which Planetmaker wrote about in good winds, several dozen or even several hundred scenario variants would emerge, so overspecialization would rather not threaten any AI.:) However, I think it would be better to limit that number to a dozen of suitably diverse scenarios, maybe even from downloading window in game(?), because even if it were several hundred variants, anyway they would not take into account all possibilities. And, of course, any data of income or size of the company will say what the game looks like with a given AI and whether it is interesting, so apart from "dry" tests descriptions containing the most important information would be useful, something like what I mentioned above.
mousetail wrote: 11 Dec 2019 20:19 Are they all published somewhere? Perhaps, with your permission, they could also be posted in the AI descriptions on the website.
For now, I have not published these descriptions anywhere. In total, I described about 20 AI. Not all descriptions are full and above all they are writte in Polish, so they won't help you much. Google Translator seems to explain it, but it's still quite lame for today :wink: If you wanted to publish it 1: 1, or even 1:10, I would have to verify everything again, because even now I see some inaccuracies in my description - it is not known whether AdmiralAI uses ships or doesn't use them. Checking each AI is always a few hours of fun. At present, I would not have much time to deal with it. However, I do not exclude, but neither do I promise, maybe I will come back to it next year. Then I would send you these descriptions. I don't mind if you use these descriptions or even a template for information.

Ps. Although I know this game rather well, the ELO ranking system tells me nothing at all - I could use some explanation
Ps2. These my assessments are quite subjective, but I think it could be more objectified.
I am sorry for may English. I know is bed.
mjm
Engineer
Engineer
Posts: 11
Joined: 23 Oct 2016 21:37

Re: AI Ranking Site

Post by mjm »

Looks awesome, great job, I hope it keeps running for these 468 days and longer.

256x256 random maps with default (fixed) settings sounds like a good choice for me, especially considering how long the simulation of the each game takes. General-purpose AI should do at least acceptable on such setup and the number of towns/industries is limited enough so that 2 AIs will have to compete about them with each other.

About the metric/win-lose condition - I think operating profit or the company value would be better than "balance-loan", as for me a "better" AI is the one which earns more, not the one that has more cash. It doesn't make a big difference for longer games (20+ years), but in the shorter ones, it discriminates the AIs that can have big income but spend all their money on new infrastructure/routes.

Are you ensuring that you use the latest AI version before the simulation starts?
xarick
Transport Coordinator
Transport Coordinator
Posts: 337
Joined: 26 Feb 2015 00:52

Re: AI Ranking Site

Post by xarick »

<Samu> my AI won't showcase its best with default presets. Easy profile is the default :(
<Samu> many others won't too
<Samu> looking for the best AI and then using easy preset, feels like...
<Samu> I dunno
<Samu> maybe I don't like losing
Formerly known as Samu
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

Re: AI Ranking Site

Post by mousetail »

Added the option to download the AI log files for debugging. Since the ranking has nearly stabilized I will be adding more "presets". Currently, I'm thinking of the following 3 presets:

Default > Everything default except cargodist is enabled for passengers (not sure why it is disabled by default)
Ocean > Higher sealevel so ship based AI's perform better
NewGRF > FIRS + OpenGFX+ Road Vehicles + OpenGFX+ Trains + Fish + AV8. Should be different enough to catch overfitting AI's while not having too specific rules for AI's to understand.

@mjm I update the AI's during each match, so the next match should have the latest version. AI's are updated quite rarely so I think this should be fine. I would use profit instead of cash for ranking, except it seems like when I parse the save format the "profit" is always 0. I may be parsing it wrong, but it seems like the profit is not actually stored.

@Samu :( I'm sorry but it seems right now a bit too complex to manage setting different settings dependent on the AI type
my AI: CPU, SnakeAI
Brumi
President
President
Posts: 920
Joined: 18 Jul 2009 17:54

Re: AI Ranking Site

Post by Brumi »

Whoa, this is awesome!

Anyways, what is the git repository of your script? You have a description about running it (https://openttd.mousetail.nl/static/doc/about.html), but the repository link is nowhere to be found :) Not making any promises to run it as I have no random servers lying around at home, but I'm interested in playing around with it for a while.
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

Re: AI Ranking Site v0.0.1

Post by mousetail »

New update [v0.0.1]:
  • Replaced some obscurity based security with actual security, so it's safe to release the source now
  • Added tentative support for "metrics", so you can have a seperate ranking perhaps for most money, or more resource effective, or whatever people think of. Currently, "Money" which behaves like the old system is the only metric.
Source code can be found here: https://gitlab.com/mousetail/openttd-save-parser. Wear sunglasses, the code isn't very pretty. Feel free to submit PRs or issues, see if this can become an actually useful tool.
my AI: CPU, SnakeAI
xarick
Transport Coordinator
Transport Coordinator
Posts: 337
Joined: 26 Feb 2015 00:52

Re: AI Ranking Site

Post by xarick »

mousetail wrote: 28 Dec 2019 18:29 @Samu :( I'm sorry but it seems right now a bit too complex to manage setting different settings dependent on the AI type
I don't mean the available settings of each AI, but this:

Code: Select all

[script]
settings_profile = hard
If you plan for a new profile, have this into consideration, please.
Add it to here, https://openttd.mousetail.nl/static/con ... efault.ini, or for a new preset.
Formerly known as Samu
User avatar
GeekToo
Tycoon
Tycoon
Posts: 961
Joined: 03 Jun 2007 22:22

Re: AI Ranking Site

Post by GeekToo »

Many AIs would score a lot better if they wouldn't crash. Do you also collect statistics about the number of crashes of an AI? Obviously a crash will result in lower ranking, but it would be good to know whether the low ranking stems from a crash, or just being a bad AI
User avatar
jfs
Tycoon
Tycoon
Posts: 1750
Joined: 08 Jan 2003 23:09
Location: Denmark

Re: AI Ranking Site

Post by jfs »

The website seems to be unreliable, it takes a long time to load, and gives errors on various pages/requests. For instance I'm not able to view the rankings for other than the default game settings.
mousetail
Engineer
Engineer
Posts: 20
Joined: 22 Mar 2014 08:45

Re: AI Ranking Site

Post by mousetail »

@jts I was seeing a lot of errors this week but there seemed to be no clear cause. I have restarted the server and it seems stable again for now.

@GeekToo Unfortunately, it seems like the openttd save game doesn't actually store whether or not a AI crashed. You might be able to extract it from the logs but that would be a lot more effort. Feel free to submit a pull request if you have any ideas.

Also, I have updated the configuration to use "hard" difficulty by default. Thanks.
my AI: CPU, SnakeAI
User avatar
GeekToo
Tycoon
Tycoon
Posts: 961
Joined: 03 Jun 2007 22:22

Re: AI Ranking Site

Post by GeekToo »

Well, it shouldn't be that hard since Openttd reports the crash, so you don't need to parse the AI logs, only the ottd output. The number in the message can be used to determine the AI index.
Yexo
Tycoon
Tycoon
Posts: 3663
Joined: 20 Dec 2007 12:49

Re: AI Ranking Site

Post by Yexo »

Nice project!

Can you add a license to your repository?

I'd also like to echo an earlier comment that looking at balance-loan is not a good metric, especially not for the early years. Currently for the 1976 rankings the top 3 is "Orders assistant AI", "IdleMore" and "Idle". AIs that are working well don't yet have enough cash at that moment so they're still capable of spending all their cash on new routes, which is working as intended.

As for reading 0, I think this code may be part of the problem:
https://gitlab.com/mousetail/openttd-sa ... ler.py#L28
This is reading an array of length 32, but NUM_CARGO is 64.
Post Reply

Return to “OpenTTD AIs and Game Scripts”

Who is online

Users browsing this forum: No registered users and 9 guests