Moderator: OpenTTD Developers
There aren't many AIs that do not crash.
What happened to basic quality control? It's so bad, if you download a random AI and play for 15 minutes, you have a good chance you get a crash. It has gotten completely out of hand.
I think some drastic measures need to be applied: I suggest to remove ALL AIs that are known to be broken from the "in-game extension downloader", alert the developer and tell them their AI is only going back when the critical bugs have been fixed. The official "in-game extension downloader" should not be a free-for-all for every possible add-on, no matter how broken. That's just terrible for the players.
What would also help a lot in discovering ton of bugs if the AIs would be automatically tested to check for common bugs or rare corner cases. Also performance test on extreme cases like 4096×4096 with many towns and industries.
Crashed either instantly or after only a few minutes:
- WrightAI v3
- TracAI v1
- AIAI v97
(too lazy to post crash logs, just test your scripts!)
Places bad stations:
- AIAI v97 (random bus stops in the middle of nowhere, no attempt to connect to road)
Failed the 1 town test:
- CluelessPlus v38
- HeliFerry v4
- TeshiNet v4
- BorkAI v23
- AIAI v97
- OtviAI v422
- SimpleAI v14
- Trans v190723
- AdmiralAI v25
- trAIns v2
- SynTrans v15
- PathZilla v6
- Terron v209
- DictatorAI v183
- ChooChoo v412 (places sign "ChooChoo: BuildNewNetwork" at random locations)
- Chopper v10 (places 1 unused airport)
- CivilAI v23 (builds 1 street depot and unused bus stop)
The "1 town test" is a game with only 1 town and no industries. Not a single AI I have tested is able to figure this one out. The solution is to build two stations in the same town.
Doing nothing nothing:
- If ChopperAI starts in a year in which the helicopter was not invented yet, it does nothing. It would be nice if starting the company could be prevented when it's clear the AI couldn't handle it.
AIs that place annoying signs by default (!):
- OtviAI v422 (sign at HQ)
- CluelessPlus v38 (sign at top of map with random nonsense)
- Terron v209 ("Road expansion: 0%", "Rail expansion: 0%")
- AIAI v97 (4 annoying HQ signs, disabable)
- ChooChoo v412 (disabable)
(IMHO there should be a global setting to prevent AIs to place any signs, I don't know why AI coders like to spam signs.)
Most AIs work 'well' only under very specific conditions, and if the map doesn't have X or Y or the year is Z instead of A, the entire thing has a heart attack and acts like Linux has kernel panicked or Windows has blue screened on it.
AIs also often have very stupid ways of dealing with some routes, as detailed here where the pathfinding looks like it was done by a blind drunk horse.
They're often there so the player (or developer) can look at what the AI is thinking of in real time. Although most of them (should) clean up the signs after they've finished doing whatever they were signposting. For example, the one you mentioned 'ChooChoo: BuildNewNetwork' is showing you the AI is looking at building a new network, i would presume at that location, and then when it has finished deciding, it should delete it and either not build it or build it there.
When used properly, allowing AIs to build signs is very helpful overall.
- Transport Coordinator
- Posts: 269
- Joined: 22 May 2006 18:25
- Location: London-ish.
Many of those used to work, but as OpenTTD has moved on they've not kept pace with changes to the game. You also have to consider these are volunteer projects. It's quite possible the authors don't play with corner cases like a single town or a very large map, and would rather spend their time elsewhere. For example, with my game script Villages Is Villages I do my best to handle the strange cases people have made bug reports for over the years, but it's still a pain downloading several 100MB of (often hard to find) newgrf files to track down some weird corner case. I mostly do it because people have asked nicely and sometimes the technical challenges of why it's broken are interesting even if I don't care for the gameplay setting.
I do think there is an issue with old, broken content, duplicates of the same file for minor version changes, and a fair amount of low effort stuff. Conversely there's also a big problem with, "I decided I didn't like the content system, so now you can only play on this server if you can hunt down the Reddit thread this specific version was posted on, good luck". If you tweak one you worsen the other. In my case, if there was some sort of approval process or formal review I wouldn't have uploaded Villages Is Villages or Timberwolf's Road Vehicles, which means you also wouldn't have Trains, Roads or Stations. (Some might argue this would have been a good thing).
How to fix it? I think maybe culling some of the things which hard crash for everyone on a version-by-version basis would be useful, behind a filter so you can still download them and play around if necessary. But that needs in-game telemetry and then some kind of abuse protection, unless someone manually has to check and approve each crash report. And it doesn't solve the "it's broken for me, but perfectly fine for someone else's use case"
This is also as good a place as any to mention I'm finding this strident "OpenTTD is totally broken and all the community content is awful and it needs to be fixed NOW" style of problem reporting rather grating. It's an excellent way of discouraging people from making anything. Certainly if the people who'd reported the first problems with Villages Is Villages on 4096x4096 maps had done it in a "this is terrible for all players and it should be removed from the content downloader until it's fixed" style I'd probably have stopped bothering uploading content for anyone else to use.
AIs should not place player-visible signs by default, unless there's a really good reason. Anyway, that's just my opinion. Maybe there are some instances in which AIs add useful non-debug signs without being annoying. I just haven't seen that yet. Feel free to show me an AI that does.
AIs that are no longer maintained would not be an issue if they would at least not crash. P
Don't get me wrong: I'm all for messing around, and experimenting with AIs, and building lots of crazy things, and I do not want to take that away. But I'm talking specifically about the OFFICIAL add-on system, i.e. the thing that is visible to the average NON-TECHNICAL player. It makes OpenTTD look bad if the majority of AI scripts is broken. There needs to be at least a bare minimum of quality assurance, like "no crashes". You can experiment with scripts all you want, but "getting approved to the content system" should have at least some requirements. Like "no crashes". There might be some grace for scripts that crash in only very academic, theoretical situations (although this still needs to be discouraged, obviously), but there should be no mercy for scripts that reliably crash within 5 seconds or under common circumstances.
I also don't buy the compability argument. Compability really does not matter if the AI does not work in the first place.
You might say I come over as a little rude, but to be fair, the problem is serious if things are so broken. It's very bad for gameplay experience when an AI crashes, it ruins the game. You can restart the AI, but this will destroy the company, including everything the AI has built.
To clarify the "no crash" requirement: No crashes under default settings and without any NewGRF active. So the absolute minimum. I admit that testing an AI with every possible NewGRF combination is not a reasonable requirement, it would be too much, sadly.
I'm not saying that every AI is "completely broken", nor do I want to say that everyone is just a fool. But I do want to show a systemic problem from a purely player perspective.. The problem is not even that we have 'broken scripts' but that that the broken scripts manage to get their way to the player, without any noticable content curation. The apparent lack of curation is the problem I want to talk about.
I also think that OpenTTD needs a stable, high-quality, general-purpose, well-tested default AI. Currently, OpenTTD ships with no AI at all, and AI development is not part of “official” development either. Which is … sad. If OpenTTD had just one reasonable default AI, the current problems with the community AIs wouldn't be that big of a deal anymore, because then there's always an AI to fall back on.
To summarize, this is what I suggest:
- Refuse AIs from being shown in the content manager if they are provably highly unstable (crashing a lot under common circumstances even w/o NewGRFs)
- Add the AIs back when the worst bugs have been dealt with
When I think about it, the problem is more about the apparent lack of content curation, and less about the scripts themselves.
- Transport Coordinator
- Posts: 263
- Joined: 11 Aug 2019 18:15
- Location: Massachusetts, USA
One potential improvement to the content download system might be a star-review system where users could rate content, allowing good stuff to float to the top and things which no longer work to sink out of sight. A good example is the Steam Workshop for a game like Cities: Skylines. It would also help with the constant questions of, "I'm new, what NewGRFs should I get?" (which is an understandable question, as there isn't really a good way of seeing what's available). But extending the UI and database to add such a feature is a big project.
Scripts that are known to crash a lot should not be visible to normal players in the first place. Minimal content curation is the answer. Maybe a simple reporting system would be better. Players report broken scripts and if it's confirmed to be real and serious enough, it is pulled & author is alerted.
- Transport Coordinator
- Posts: 269
- Joined: 22 May 2006 18:25
- Location: London-ish.
The problem with curation and testing is someone has to take the time to do it. An organic solution like star ratings would be the most elegant, as it doesn't require a thankless task to become someone's (that word again) job. Maybe some sort of "AI CI" that runs the AI on a dedicated server for a bit before you can upload, but... ah, someone has to pay for the computing resources and we're back there again. I'm not sure if it's particularly feasible with the current BaNaNaS infrastructure to do such a rating system, but I haven't looked too closely at it. You'd also need some way to handle "this used to work, but now it's broken in 1.10.x" situations and also the challenge that new sets come along all the time. I'm kind of against strong curation as funnelling people to a small number of existing sets rather than giving a fair chance to all authors, even though I see a benefit to new players in providing a few core combinations like UKRS2+HOVS+BATS, "full Andy" and so on that work well together. It'd also be nice to hide all of the beta versions, old abandoned sets, and things without going to the extreme step of the blacklist used for ripped-off sprites and spam.
Another challenge is a lot of stuff that's now broken used to be widely used, downloaded thousands of times and in theory likely to have gathered hundreds of 5* ratings. In the AI case, plenty only need simple fixes to handle some of the new game concepts that have been created, but sometimes it's difficult to contact the authors to get those uploaded. For example I've fixed a couple of bugs with CivilAI for my own use, but it's not my AI to re-upload and Pikka isn't around any more to implement those changes. Re-uploading carries the risk of licence violation for non-GPL or non-CC stuff, and of course worsens the problem of "now there are two versions of this AI, which should I use?" facing new players.
It's a bit like the serious base underlying my rather tongue-in-cheek comments about "the price for complaining about extra zoom sets is you have to make an extra zoom set" - just once it'd be nice to see this kind of thing expressed as "I made a PR to allow hiding AI-built signs in the transparency options" or "I've started an attempt to build a bug-free reference AI, please download and test", i.e. actually doing something about the problem rather than just complaining about it as if no-one's ever spotted it before.
I see now the problem that manual curation needs a curator … Yeah, that's a problem indeed. Especially if those people disappear.
I wonder how content manages to get into the content system … Does it need an approval from anyone? Or can everyone just post an add-on and it instantly appears for everyone? I mean, is there any verification required?
I am still not convinced the star rating system is going to cut it. You have basically mentioned all the problems with star ratings yourself. All reasons why I am skeptical. It's not a great solution.
Well, since OpenTTD promotes non-free software I'd say that's kind of a self-inflicted wound then. The solution, of course, is “boycott all proprietary content”. Also note that not even all CC'd content is really free/libre, there is sometimes the awful "NoDerivs" or "NonCommercial" clause which is incompatible with free software licensing. Yeah, I think just let those add-ons rot. They don't deserve support from the FOSS community.Re-uploading carries the risk of licence violation for non-GPL or non-CC stuff, and of course worsens the problem of "now there are two versions of this AI, which should I use?" facing new players.
Talking about automation: Some kind of automatic testing would be really great. I mean, I do know that community AIs have many bugs, but I can't really put my finger on where exactly the bugs lie. Automating the testing away, even just partially, would probably help a ton finding all those "hidden" bugs. But I admit I have no idea how exactly that would work in practice. I'm not sure if even a dedicated server is really required to do with. On the other hand, where are all those donations going to, anyway?
Hmmm, I am out of ideas right now. To be honest, I can't really think of a great solution on how to solve this once and for all. Curation was my main point here, but if that's not possible, I guess we're doomed.
Maybe the only path forward here is give up on the problem altogether and write a big fat notice that every add-on you see might be completely broken and that the OpenTTD team takes no responsibility whatsoever.
What do you think about taking/forking SimpleAI and turning it to the "official default AI"? Heck, even if SimpleAI is included into official OpenTTD verbatim (except renaming it to something like "DefaultAI"), that would already be a huge win. I think SimpleAI is "good enough" for now. It's closest to TTD behavior, it builds the basic things OK enough, it's GPLv2'ed. The only downside for now is that it doesn't support ships. But that's still so much better than nothing.
I think the simple act of adding any default AI would be a massive improvement of OpenTTD that cannot be understated. Not really because SimpleAI is that great, but because it would be the first time in years since OpenTTD shipped with an AI (the last time was in 0.7, I guess, i.e. ages ago). So in ther words, the day OpenTTD adds a default AI would be historic. xD
Users browsing this forum: No registered users and 4 guests