How to create widely accessible scenarios?

[Nagyzee]

The last few months I've been planning 3 rather detailed scenarios on huge maps. I've been tinkering around a lot with heightmaps and now thanks to oberhümer's Place houses manually patch I can continue with them in the scenario editor with a chance to finish in a reasonable amount of time.

I'd like to make them as widely available as possible so I thought I'd ask what I should look out for to make that happen. I plan on playing them with Chill's patchpack myself but make them with trunk + only the house placer patch (which is savegame compatible with trunk) so as to not make the scenarios obsolete in a few months. That should work, right?

My main problem is how to approach the newgrf issue. I like to play with tons of them, many of which are not available through the in-game downloads. I've read that adding newgrfs is safer than removing them so my plan is to include only the most necessary grfs. This would surely include industry and house sets and probably not include most of the only eye-candy graphic enhancement grfs. But what about vehicle sets? Is it safe to add them later or should I preferably include them from the start? Also what else should I be vary about? Any advice on the "risks of adding newgrfs vs. accessibility" dilemma is most welcome.

[Eddi]

there is currently (very early) development to make scenarios independent from GRFs, but you probably won't be able to pick the exact place of houses then, only a vague city size.

[Nagyzee]

Well, then that wouldn't help me much even if it was already implemented. I'd like to include towns designed down to the last house much like the Cindini map has. Thanks for the interesting info anyhow.

[Terkhen]

Houses can be controlled by NewGRFs too. Therefore, any approach to scenarios that is truly NewGRF-independent will not be able to store specific house types and their locations.

[Nagyzee]

I'm sad to hear that, designing the towns is quite a central thing for me. Anyhow I'd need advice about the current system mostly as I intend to make my scenarios in the close future. I know that even adding newgrfs to an existing scenario is not supported but I'd like to make an informed compromise in the safety vs. freedom of choice (or accessibility - as in only including a few easy to find grfs) dilemma.

Could you please share that in general what kind of grfs are mostly safe to be added later and which need to be present from the start? For example according to my knowledge simple graphics-changing grfs usually fall into the first category while industry sets are definitely in the latter. I'd need some further guidelines like that.

---

I'm currently working on a composite heightmap of a part of Oceania for a 2048x2048 scenario. It will include coastal Queensland, parts of Papua-New Guinea (most importantly Bougainville Island and Rabaul area, maybe also Port Moresby), Solomon Islands, Vanuatu, New Caledonia and either a part of Japan or Hawaii in the currently empty top right corner. If I manage to, I'd like to somehow cram the Northernmost part of New Zealand into the bottom right corner, too. Heights are not perfect yet either, not all coasts are drawn perfectly. Anyhow that's how it looks now:

OceaniaTestMap.png

(483.01 KiB) Downloaded 1 time

Focus would be on shipping and (narrow gauge) plantation railways a la colonial style first then later on air transport as well. If you have some suggestions for the scenario, I'm happy to hear those, too.

[Eddi]

it is often not intuitive which GRFs are "graphical only", and keep in mind that normal users cannot add any GRFs at all.

GRFs that you should not add after the scenario started:
industry GRFs, house GRFs, vehicle GRFs which offer "regearing" (e.g. NARS2)

GRFs which are fairly safe to add, but not remove:
station GRFs, vehicle GRFs (as long as no vehicles exist), track GRFs (as long as no tracks are built), ...

GRFs which are graphical only:
catenary GRFs, tree GRFs

[Alberth]

Sorry for the long post, but I tried to explain the situation a bit, as well as in which direction it might go, and the text exploded somewhat

The reason for removing the simple changing of newgrfs in 1.1 is based on limitations in the game engine. Adding or removing a newgrf is just like adding or cutting away a piece of the program. Theoretically, you can fix the problems in the game engine, but it will take many years, and there will still be limitations. For example, one cannot *ever* just cut away pieces of program that is being used. Even for the parts where it can be solved, checks need to be added, thus raising CPU usage, and reducing the supported size of the game.
In other words, with a lot of effort, perhaps one can make it work nicely for a number of cases, but never in general, and never without adding more costs.

The inevitable conclusion therefore was, that the above route is not (and never will be) a feasible direction to overcome the problems with newgrfs in a game. That conclusion resulted in the fact that any problem reported due to changed newgrfs instantly became a non-solvable bug of the program. To close that stream of bugs, the proper solution would be to remove access to the newgrf facilities from a game for everybody. However, that was considered to be a too harsh solution for the current newgrf and scenario developers, so a quite closed doorway was created, where the key is still available if you know where to look.

The addition of the mostly closed doorway is, in my view, a step towards full removal of doing newgrf changes 'the old way' for all.
Besides closing a hole filled with bugs, the change made everybody very much aware that a better way of handling newgrfs is needed.
(The newgrf change problem has existed for many years, yet nobody bothered to even notice it until now. Without adding the doorway, that situation would not change.)

Unfortunately, the scenarios got hit very badly by that move.

A scenario is basically a savegame without companies. (They always were that, but nobody realized that fully, let alone that someone even considered how to change that.) In the past, the 'old way of changing newgrfs' allowed enough flexibility to customize scenarios *slightly* (without crashing or turning ugly in other ways) for a player. In the current situation, playing a scenario is really equivalent to getting a savegame without companies from somewhere.
On the positive side, a scenario developer now has full control over what newgrfs are used by the player (since they cannot change it any more).
On the negative side, in many cases, a scenario developer has no interest in fully defining the newgrfs. He/she may want to leave it entirely free, or only define a few (and theoretically, he may even want to express 'these are recommended').

It is currently not clear how to handle the latter (the 'somewhat free' scenarios). One *possible* direction (but there are probably other solutions too) is to create a new file format that allows automatic generation of the world on demand.
(Think of it as automatically opening the various main menu windows to add all needed settings, and finally press 'generate' automatically.)
For some of the scenario data, you can do this today, for other data (such as placement of towns, roads, and industries, for example), functionality needs to be added.

The advantage of such a new file format is (in my view) that it is independent of the program data (since you generate a new world every time). The disadvantage is that you (as scenario developer) probably don't have full control over all aspects (such as the precise house or industry at a certain tile). Like the new file format, how much this 'not-full control' is a problem is not clear at all (ie it is called 'somewhat free' for a reason ).

If you have a better idea, please tell.

I'm sad to hear that, designing the towns is quite a central thing for me. Anyhow I'd need advice about the current system mostly as I intend to make my scenarios in the close future. I know that even adding newgrfs to an existing scenario is not supported but I'd like to make an informed compromise in the safety vs. freedom of choice (or accessibility - as in only including a few easy to find grfs) dilemma.

As explained above, relying on the ability to change newgrfs is a dead road. Players don't have the option to change them. Very maybe, it may improve slightly, but I wouldn't hold my breath. As far as I know, nobody is working on it.

The option you have today, is to create a scenario that fully defines the newgrfs used. This will give you full control over all aspects of the scenario. Since such a scenario file is like a savegame, creating alternatives for your users boils down to creating several scenarios.

Somewhen in the future, the option of 'somewhat free' scenarios may become available in some form.
However, the combination of fully defining the precise houses, yet allowing changes in other newgrfs in some way, is likely to be impossible for a long time (but this is a guestimate, as everything is extremely unclear at the moment). Newgrfs hook very deeply into the game program, affecting possibly almost every aspect of other newgrfs.

[FooBar]

While we're at this subject, allow me to make a small suggestion: let the game "understand" what grfs are save to add later and what are not.


One way to do that is add a flag as action14 that marks a grf save to add later or not. This leaves the decision whether a grf is save to the newgrf developer, which may or may not be a good idea. What might be good about it is that it allows grf developers to mark a grf 'safe' during development while it actually isn't, making development easier. It also could shift the responsibility of "the game crashed due to adding this newgrf" from the OpenTTD developers to the particular newgrf developer. Newgrfs which do not have the "safe" flag set are considered unsafe by default and cannot be added to existing savegames.


A different approach would be to have a comprehensive list of action and action/feature combinations that are safe to add to a running game. As soon as the user tries to add a grf to a running game, the particular grf is parsed. As soon as an action is encountered that is not on the safe list, parsing stops and adding the newgrf is denied.


In both cases, the same approach can be taken for removing grfs from existing games, as some grfs will be safe to add but unsafe to remove again.

[planetmaker]

his leaves the decision whether a grf is save to the newgrf developer, which may or may not be a good idea.

Sorry, not a good idea.

Given the history of NewGRFs with bugs, assumed but de-facto missing compatibility, this will fail. Also: generally no NewGRF can be considered save to add as another NewGRF might check for its presence and then disable itself or even other NewGRFs. And that's of course only the worst, there are many subtle and non-obvious way newgrfs might interfere with eachother which might only become apparent if you add e.g. a 2nd or 3rd of a type or similar.

[FooBar]

Hmmm...too bad.
Seemed like a good idea in my head though

[Nite Owl]

This is going way out on a limb and is, most likely, not even feasible but here goes.

A recent addition to the Scenario Editor is the ability to export a height map. If a scenario created by someone else is loaded into the scenario editor and you do not like the settings or the NewGrfs involved you can export the height map, change the settings and NewGrfs to your taste, import the height map, and then recreate the scenario. This would involve a lot of work on your part comparing the two scenarios so you can get the towns and industries and rivers and trees in the exact right locations. In the end, after all of the effort, you would have a duplicate of the original scenario suited to your individual wants without any of the possible bugs that making those changes directly to the original scenario may have caused.

Here comes the not feasible part. Would it be possible to create a program, or the specifically needed code within OpenTTD, to do the duplication work for you? You would export the height map, make your changes, import the height map and then the program or code would compare the two scenarios and regenerate the towns and industries and rivers and trees exactly as they are located on the map in the original scenario to your altered duplicate scenario. This most likely falls into the category of TMWFTLG but it is an intriguing idea nonetheless. My guess would be that each scenario would have to have a seed number that describes the location of each item to be duplicated in the same sort of way that the current terrain generator does. Fortunately for me those kind of specifics are for you true code geniuses to figure out. I am just a crazy idea man.

[Alberth]

Here comes the not feasible part. Would it be possible to create a program, or the specifically needed code within OpenTTD, to do the duplication work for you? You would export the height map, make your changes, import the height map and then the program or code would compare the two scenarios and regenerate the towns and industries and rivers and trees exactly as they are located on the map in the original scenario to your altered duplicate scenario.

I am sorry, but I don't see the non-feasible part here.

In fact, Terkhen, planetmaker, Rubidium, and myself discussed possibilities along these lines, and had an even more crazy idea, namely, don't just save the heightmap, but also save the other data. For example, rivers could be saved as a (mostly) transparent png where the non-transparent parts denote river. Roads could be done in much the same way (but how to handle town-owned road?). Any decent bitmap editor allows layers of bitmaps, so alignment of those pictures is quite doable, even outside OpenTTD.
The challenge is towns, industries, etc. How do you save a newgrf-independent description of those?

I am just a crazy idea man.

The global idea is not at all crazy imo, but the devil is in the details. How much to save of the remaining data is still an open question.

[FooBar]

Roads could be done in much the same way (but how to handle town-owned road?).

Maybe two different colours? Or two separate overlays.

As for industries, the northernmost tile of the industry can be saved. If you do that together with the industry ID and layout number, you have (I think) all the information needed to place the exact same industry, provided that exactly the same industry grf is loaded. If a different industry grf (or none) is loaded, the game could randomly select one industry of the same type (primary/secundary for instance) and place that on the given location instead.

For town buildings, one could essentially do something along those same lines. As well as for trees (which do not need a fallback system).

For objects I would take a slightly different approach: if the object newgrf used is not available, objects will not be placed.

What also comes to mind is XML to store all information.


These are just some things that come to mind. If they're useful, use them. If not, then hopefully they will spawn some better ideas.

[Kogut]

Warning, idea

-Heightmap
-treemap
-rivermap
-location and sizes of towns as XML/locations of buildings + IDs used in case of the same house newgrfs
-the same for industries
-roadmap
-XML of bridges and tunnels

Together: scenario file

+ tool to convert scenario file int mix of mentioned files and mix into scenario

[Terkhen]

That's more or less what we discussed, except that the format is INI (way simpler, and OpenTTD can already read it) and that OpenTTD itself would be able to load the scenario (stored as a tar file) and use it (applying the NewGRFs currently selected).

EDIT: I'm not interested in yet another XML discussion, let's just leave it at "OpenTTD already supports INI and a simple format should be enough for this feature"

[FooBar]

XML was just an example, by no means intended to open a can of worms. If ini fulfills the same purpose, that will work just as well. And it is indeed probably easier to implement as openttd already supports it.

Good to hear that work is on the way. Or at least some sort of specification on how to implement it!

[Terkhen]

Or at least some sort of specification on how to implement it!

To my knowledge there is currently no specification of the format to use. Ideally it should be extensible (so we can easily add more features in the future) and take care of positions in the map. As an extension, it should also allow to store enough information about an item to make a guess about a possible good replacement for the item when converting the scenario into an actual savegame with a NewGRF selection that might be different (or the same!) to the one used for creating the original map before converting it to the new format.

[FooBar]

What I meant was that it is good to see that you guys are thinking about a way to implement it.
The term "specification" was probably ill-chosen.

[Nagyzee]

Well, this discussion took an entirely different route from what I expected but it was all quite interesting and educating. And I also got the answer I was looking for here:

it is often not intuitive which GRFs are "graphical only", and keep in mind that normal users cannot add any GRFs at all.

GRFs that you should not add after the scenario started:
industry GRFs, house GRFs, vehicle GRFs which offer "regearing" (e.g. NARS2)

GRFs which are fairly safe to add, but not remove:
station GRFs, vehicle GRFs (as long as no vehicles exist), track GRFs (as long as no tracks are built), ...

GRFs which are graphical only:
catenary GRFs, tree GRFs

This is exactly the type of information I needed. Thanks a lot for that. Of course I plan to release my scenario as already quite well-rounded and playable with regards to newgrfs, but I'd like to add some more for myself (and recommend those for others) and also I'll probably avoid adding (non-industry and non-housing) newgrfs still in intensive development until right before the release of my scenario. For example this way I can start working on my scenario now and add the next 2cc trainset beta in 1-2 months instead of adding a 2cc set nightly right at start and then being stuck with it. Apart from including hard to find newgrfs adding in-development ones (like 2cc set and FIRS) is my biggest fear.

Another question just to make sure: Does BaNaNaS keep older versions of newgrfs available for compatibility reasons?

[planetmaker]

Does BaNaNaS keep older versions of newgrfs available for compatibility reasons?

It does. Upload of a newer version of the NewGRF will hide it, but it will remain accessible and available for download when you have a savegame or scenario which requires a particular version.