Moderator: OpenTTD Developers
This could be better.
I wanted to classify AI's to figure out which ones are solid, so that I can figure out which ones are the best. For this purpose, I setup a server to run AI matches 24/7 and analyze the results. Eventually I use this to rank all the AI's so you can see which are the best. Also I aim to help AI authors to debug their AI's.
The site can be found at https://openttd.mousetail.nl/
A lot more matches will need to be played before the ranking stabilizes, so don't hold your breath. Also the site is still work in progress, there might still be errors and rough edges and spelling mistakes etc.
Let me know of any bugs you find or suggestions you might have.
Generally, there are several basic elements that significantly affect the AI rating:
1. Maintenance costs: on /off
2. Cargodist for passengers: on / off
3. Additions - for example, many AI cannot use articulated vehicles, while others do not know where to drive when the industrial set is changed.
4. Gameplay / construction method - sometimes AI is very annoying, which steals factory products, build railroad crossings and destroy vehicles to other companies, or, for example, like your CPU, clutters the map with thousands of signs, which completely destroys the game. Then it doesn't matter what income this AI earns.
At one time I played for AI testing. I even made a description to many of them. A little for personal use to know in the future what not to use, but also after finishing I wanted to post the full list on another knowed for me forum of this game that I have known for years. Definitely the profits that AI generated was not the only criterion for its assessment. For example, I will put a description of one of them: AdmiralAI - one of the best AIs. Maybe it will be useful to you in some way.
AI: Admiral AI
Overall rating: 4-/5
Competitiveness / Revenue: 3/5
Includes infrastructure costs: No
Compatibility with industrial sets (e.g. FIRS): 4/5
Construction aesthetics: 3/5
Buses: YES (can be turned off)
Trucks: YES (can be turned off)
Composite vehicles (semi-trailer trucks, articulated buses): YES
Trains: YES (can be turned off)
Planes: YES (can be turned off)
Nice AI, using all means of transport. It deals with rail transport quite interestingly. His individual connections do not look very interesting, because the lines in both directions usually run a completely separate route. However, after some time, a rather interesting picture emerges from the chaos of its interconnected routes. As it turns out, this style of construction is unrivaled on very tight maps.
+ generates relatively high revenues
+ transports all loads including machines
+ chaotic construction style that works on smaller maps
+ builds industrial stations from the ISR set
+ many options in the settings allowing you to choose the means of transport with which you will build connections
+ does not freeze, but ... it has other problems
- does not include infrastructure maintenance costs (should not use aircraft when costs are included)
- creates too short connections for his income to pose any threat
- almost as many airports as airplanes - thousands of passengers are waiting at most airports, but sooner will build another airport than buy a new plane
- sometimes has fits of madness, during which he terraforms randomly selected areas spending almost all the accumulated funds for this purpose
- other times he builds railway stations anywhere he can, which he does not connect or delete - these remain unused on the map
Multiple rankings for different variations of game rules are planned, with or without newGRFs, with or without infrastructure maintenance, etc. However, this will massively increase the amount of computing power necessary to create the ranking so I'm waiting until the "default" preset ranking has stabilized.
I like your reviews. Are they all published somewhere? Perhaps, with your permission, they could also be posted in the AI descriptions on the website.
* island-like map vs nearly-all-land
* very hilly vs. very flat map
* small vs. big map
* choice of selected NewGRF
* choice of starting year and climate
* choice of certain parameters like cargo weight, maintenance cost, cargodist...
Without availability of the test maps being used, the results remain highly subjective and non-repeatable. And AI authors don't even have something to benchmark their AI against
Yes, those are soom good variants for presets.
You can download the save games from the site so AI authors can debug what went wrong.
I'm not sure if standardizing the maps is a good idea. It might lead to over-fitting where one AI happens to be good on a certain map but is not very strong overall. Generating random maps every time makes sure that there is no overspecialization.
Oh, I'm not arguing for having only one map. But having like a dozen maps which cover the above points (and more). But using those identically for all AI and also for challanges to compare AI evolution. The latter, sadly, has been pretty stagnant over the recent years.
In my opinion, developing such template scenarios would be a better solution. This would give you the possibility to more objectively compare AI and see differences in performance. Random maps, each time different, especially different for each AI, do not give such a possibility.
From these conditions, which Planetmaker wrote about in good winds, several dozen or even several hundred scenario variants would emerge, so overspecialization would rather not threaten any AI. However, I think it would be better to limit that number to a dozen of suitably diverse scenarios, maybe even from downloading window in game(?), because even if it were several hundred variants, anyway they would not take into account all possibilities. And, of course, any data of income or size of the company will say what the game looks like with a given AI and whether it is interesting, so apart from "dry" tests descriptions containing the most important information would be useful, something like what I mentioned above.
For now, I have not published these descriptions anywhere. In total, I described about 20 AI. Not all descriptions are full and above all they are writte in Polish, so they won't help you much. Google Translator seems to explain it, but it's still quite lame for today If you wanted to publish it 1: 1, or even 1:10, I would have to verify everything again, because even now I see some inaccuracies in my description - it is not known whether AdmiralAI uses ships or doesn't use them. Checking each AI is always a few hours of fun. At present, I would not have much time to deal with it. However, I do not exclude, but neither do I promise, maybe I will come back to it next year. Then I would send you these descriptions. I don't mind if you use these descriptions or even a template for information.
Ps. Although I know this game rather well, the ELO ranking system tells me nothing at all - I could use some explanation
Ps2. These my assessments are quite subjective, but I think it could be more objectified.
256x256 random maps with default (fixed) settings sounds like a good choice for me, especially considering how long the simulation of the each game takes. General-purpose AI should do at least acceptable on such setup and the number of towns/industries is limited enough so that 2 AIs will have to compete about them with each other.
About the metric/win-lose condition - I think operating profit or the company value would be better than "balance-loan", as for me a "better" AI is the one which earns more, not the one that has more cash. It doesn't make a big difference for longer games (20+ years), but in the shorter ones, it discriminates the AIs that can have big income but spend all their money on new infrastructure/routes.
Are you ensuring that you use the latest AI version before the simulation starts?
<Samu> many others won't too
<Samu> looking for the best AI and then using easy preset, feels like...
<Samu> I dunno
<Samu> maybe I don't like losing
Default > Everything default except cargodist is enabled for passengers (not sure why it is disabled by default)
Ocean > Higher sealevel so ship based AI's perform better
NewGRF > FIRS + OpenGFX+ Road Vehicles + OpenGFX+ Trains + Fish + AV8. Should be different enough to catch overfitting AI's while not having too specific rules for AI's to understand.
@mjm I update the AI's during each match, so the next match should have the latest version. AI's are updated quite rarely so I think this should be fine. I would use profit instead of cash for ranking, except it seems like when I parse the save format the "profit" is always 0. I may be parsing it wrong, but it seems like the profit is not actually stored.
@Samu I'm sorry but it seems right now a bit too complex to manage setting different settings dependent on the AI type
Anyways, what is the git repository of your script? You have a description about running it (https://openttd.mousetail.nl/static/doc/about.html), but the repository link is nowhere to be found Not making any promises to run it as I have no random servers lying around at home, but I'm interested in playing around with it for a while.
- Replaced some obscurity based security with actual security, so it's safe to release the source now
- Added tentative support for "metrics", so you can have a seperate ranking perhaps for most money, or more resource effective, or whatever people think of. Currently, "Money" which behaves like the old system is the only metric.
I don't mean the available settings of each AI, but this:
Code: Select all
[script] settings_profile = hard
Add it to here, https://openttd.mousetail.nl/static/con ... efault.ini, or for a new preset.
@GeekToo Unfortunately, it seems like the openttd save game doesn't actually store whether or not a AI crashed. You might be able to extract it from the logs but that would be a lot more effort. Feel free to submit a pull request if you have any ideas.
Also, I have updated the configuration to use "hard" difficulty by default. Thanks.
Can you add a license to your repository?
I'd also like to echo an earlier comment that looking at balance-loan is not a good metric, especially not for the early years. Currently for the 1976 rankings the top 3 is "Orders assistant AI", "IdleMore" and "Idle". AIs that are working well don't yet have enough cash at that moment so they're still capable of spending all their cash on new routes, which is working as intended.
As for reading 0, I think this code may be part of the problem:
https://gitlab.com/mousetail/openttd-sa ... ler.py#L28
This is reading an array of length 32, but NUM_CARGO is 64.
Users browsing this forum: No registered users and 1 guest