Integrated plugin system

[christophedlr]

Hello,

I know OpenTTD with the several years and I'm very happy.
It would be that there is a plugin system running with DLL/SO.

It would be possible to have new features without change the executable of game.
Those that add new functions, changes could offer their without losing other changes.

Yourself, you would have less a changes the executable of game.


I'm part of TTFF community, and the players feel the same.


P.S. : Please excuse me for my bad english.

[Alberth]

How are the NewGRFs not a plugin system?

[planetmaker]

It would be that there is a plugin system running with DLL/SO.

There's NewGRFs and NoAI; those and more can be downloaded from ingame. They provide exactly what you probably ask for.

[deamonhunter11]

He's asking about something that features that can only changed by patches. If not graphics and such can be found in the check online content.

[Alberth]

NewGRF is a *lot* more than graphics. It also contains code that is executed as part of the game loop.
NoAI code is executed to replace actions of a player, so you can do anything a player can do (with the same restrictions too ).


Note that adding a plugin system based on executable code is not better in any sense. If your plugin code is supposed to do something useful, it means that the game code must check whether a plugin exists, and call it. That in turn assumes that there is a pre-defined set of hooks that you can change, exactly what NoAI and NewGRF have.


Also with a plugin system with DLL/SO, you cannot plugin outside the pre-defined set of hooks.

The current system of patching the source code, and re-compiling the program does provide that capability, it literally enables you to change anything.
Note that re-compiling is not so bad, to build a DLL/SO, you need to compile anyway, so you can just as easy compile the whole program.

Edit: Patching the source also has the advantage that it works at all platforms supported by OpenTTD

[christophedlr]

Yes but there are two changes possible.
For example, if I want to add a system of contract, I could not have the Watch Companies system.
If it is possible to have such a thing with the current system , I 'm interested.

[Rubidium]

For example, if I want to add a system of contract, I could not have the Watch Companies system.

You can't do those things by simply loading some .dll or .so. You would need to add hooks for those .dll/.so files to use, which means modifying OpenTTD. It's likely that for each "feature" you'd need other hooks and thus more modifications.
As you would need to make changes to vanilla OpenTTD for your .dll/.so files to work, it's totally pointless to introduce such a form of plugins as it could be, as easily, compiled into the game as in either case you won't have a vanilla system but in the case you don't use .dll/.so files it will be easier to compile, install and distribute!

[christophedlr]

But compiling requires an impossibility to use or other changes , so get the source code of other changes and apply.

[planetmaker]

As OpenTTD is GPL-licensed, no-one may distribute compiled versions without making an offer to supply the source code anyway. So that is no limitation at all.

[christophedlr]

I agree , but if we are within the grounds of the community TTFF , apply the mods autrse of each proposed new function , we have not finished.

This can be done with a subversion server to handle this (and doing their modif ) , but we return to the original problem , namely that if someone outside has a good function, we must add our mods in the end it's quite heavy to implement.

[Rubidium]

How many times do we need to say that a PLUG IN system must have something it can PLUG IN? It needs HOOKS in OpenTTD's code because otherwise the plug in has NO idea where to put its changes.

[Alberth]

Let me try too

You are not understanding plugins, I think.
You ever saw a movie where a technician opened a huge case, and you see wires running all over the place? That is kind of the internals of OpenTTD (ours is much less messy of course ).

Now you have an extension (a plugin).
It comes in a nice closed box. It has a plug attached to it.

To connect it to the OpenTTD program, you need a hook from the program.
In wires, you need a second plug in the OpenTTD case that matches the plug of your extension. You need to make a hole in the OpenTTD case for the second plug, and add additional connections from/to the wires of OpenTTD internals and the second plug in the OpenTTD case.
Once you have done that, you can plug your extension in, and hey presto, it works.

If you were the only person with an extension, that would be managable. However:


Then person B comes along with another extension. His plug is different from yours, so he needs to do the above too.

Then person C comes along with another new plug, etc.

In fact, each extension to the OpenTTD program is so unique, that you never reach the point that a plug already exists for a new extension. In other words, we would be providing new variants of plugins forever (constantly adding more plugs and more connections between wires and plugs), until OpenTTD eventually collapses due to the overhead and added complexity of all the mostly unused plugs.
(Note that wires from existing plugs to the internal wires of the program also count as internal wires, so you are not only adding a plug, you are adding more wires at the same time. Now think what happens if you add a 100 plugs in that way.)

Also, what if you want to extend an extension? Your plugin system does not allow that. (or perhaps even you want to change a plugin, so it co-operates nicely with yours, something often done by patch-packs authors.)

So instead of OpenTTD supplying the connection between its wires and your unique plug, we give access to the wires itself, allowing you to make the optimal connection.

[Muzzly]

Before you start reading. I do respect everyone participating in openttd development and do not want to affront someone.

Current situation:

There are 4 groups of people ( Iwill not include GRF creators )
A. developer (12 persons according to readme.txt)
B. patch authors (probably more than 12)
C. patch pack creators (I think less than 12)
D. Players who are participating in tt-forum (many)

What has each group of people currently to do with patches?
1. Developer. If he has time, look through the patch and then accept or reject it.
2. Patch creator probably has to update it for each trunk change.
3. Each patchpack creators have to do the same Don Quixote way - merge patches and compile it for players.
4. Each player , that want to play patched version has to search for latest binary, or compile it on his own. A lot of outdated binaries , but non of them is compiled with all patches available

So I was thinking , is the no other way to save patch creators, patchpack creators and players time and let them work for the same goal with developer.

The way it could work: (see image)
Developers (later devs.) do their job and work on left side ( blue elements), patch creators work on right side ( red elements ). If some patches interfere too much with each other then:
A) Either skip it form public svn, till patch developers communicate with each other and make one common patch.
B) Or put it under one #ifdef GROUPED_PATCHxxx with other patch that it interferes at most

In other words public SVN could be a factory that
accepts:
- red code
- blue code
produces:
- Openttd 1.x.x “equipment”
- Patchpack prototype


Disadvantages:
1. Code in public SVN would be harder to read because of many #ifdef /#endif
2. Devs will get extra work, syncing their code to the public one

Advantages:
1. Patch authors do not have to modifier they patch for every current trunk change and concentrate on patchpack version looking for bugs, because patches can interfere with each other
2. Common player has a choice to take release, testing or patchpack version. You will agree that currently not every user can (or has time) to merge and compile his own version.
3. Patchpack player will be happier, because patch cannot expire and there will exist a version with either all or almost all compatible patches.
4. Because trunk and patches are placed in one place (public SVN), devs. and patch authors cannot ignore each other anymore and will interact on the same level.
5. In case something bad happens to public SVN, devs have always a possibility to continue to work on their internal SVN and compile a release version, till public SVN can be compiled again.
6. A lot of community time will be saved. How many patchpacks was created and will be created. Is it not better that users invest their time by playing patchpack and searching for new bugs?

Summarized. Again lets look at 4 groups of people:
A. developers. A bit more job, because they have to sync trunk to public SVN. Now they are acting as patch creators bumping they code to public trunk.
B. patch authors. Have more free time, which they could spend for improving they patches.
C. patch pack creators. Only one of them has to create a patchpack, the rest of them have nothing more to do, they can just test or improve public code.
D. Players who are participating in tt-forum. Have also more free time, because they do not have to search for binaries all over the forums. They could play patchpack or release version and search for bugs. Or even join 2nd or 3rd group.

I think in this case first 3 groups are acting together as developers. Because they are improving the same code, but on different parts

[Yexo]

The way it could work: (see image)
Developers (later devs.) do their job and work on left side ( blue elements), patch creators work on right side ( red elements ).

In your image it's impossible to work on "just the private svn". Say one developer does a change on the private svn server that cannot be merged with the public svn, who is responsible for merging that change in the public svn?
A) The developer: in this case you're making the developers responsible for updating all random patches that some forum user could think of. This means a lot more work for the developers that could also be spend in fixing bugs / implementing new features properly.
B) One of the patch authors: Very soon there would be an update which no patch author merges to the public svn so public svn becomes broken. At that point we're back at the current situation with only the "private svn" working.

If some patches interfere too much with each other then:

This can't happen, because when someone tries to create a patch against the public svn that conflicts he won't be able to create it. Or he'll solve the conflicts when he encounters them, solving the problem along the way.

Advantages:
1. Patch authors do not have to modifier they patch for every current trunk change and concentrate on patchpack version looking for bugs, because patches can interfere with each other

So you are proposing to shift the workload of updating patches from the patch authors to the developers.

2. Common player has a choice to take release, testing or patchpack version. You will agree that currently not every user can (or has time) to merge and compile his own version.

Of course I agree that not every user can or has time to merge/compile his/her own version. The question is: do they really need to? Arguably the most stable versions are the official openttd releases (sometimes stable, sometimes nightly). Why would a user want to play with a patchpack that has a) no multiplayer compatibility (because not enough users play it) and b) has several bugs that can cause crashes and c) your savegames are broken after each savegame update in trunk? Now there are several users that can live with those drawbacks if they get the latest shiny new features in return, but that isn't the majority of players.

4. Because trunk and patches are placed in one place (public SVN), devs. and patch authors cannot ignore each other anymore and will interact on the same level.

Again: who has to do the actual work integrating trunk changes in the public svn?

5. In case something bad happens to public SVN, devs have always a possibility to continue to work on their internal SVN and compile a release version, till public SVN can be compiled again.

See above, who is responsible for fixing the public svn?

6. A lot of community time will be saved. How many patchpacks was created and will be created. Is it not better that users invest their time by playing patchpack and searching for new bugs?

Users that are playing patchpacks generally don't report bugs.

Summarized. Again lets look at 4 groups of people:
A. developers. A bit more job, because they have to sync trunk to public SVN. Now they are acting as patch creators bumping they code to public trunk.

You can change "a bit" in "a lot more", which is a serious drawback.

B. patch authors. Have more free time, which they could spend for improving they patches.

Which they won't do because their patch is already in "public svn" and they don't feel responsible for it anymore.

C. patch pack creators. Only one of them has to create a patchpack, the rest of them have nothing more to do, they can just test or improve public code.

One of them? I thought you just said the devs were responsible for keeping public svn up to date?

D. Players who are participating in tt-forum. Have also more free time, because they do not have to search for binaries all over the forums. They could play patchpack or release version and search for bugs. Or even join 2nd or 3rd group.

"Players participating in tt-forum" is not the same group as "people playing custom builds and looking for binaries all the time". Furthermore people generally don't report bugs in patchpacks.


Now: if you still think a "public svn" would work, you can easily create it yourself. Finding hosting somewhere shouldn't be a problem, make one patchpatck author responsible for syncing that repository to trunk and give every patch author write access to it. It's essentially the same as your proposal, only one (or more) patch patck authors are responsible for syncing the public repository with trunk, which means there is no extra workload for the developers. Personally I doubt it's going to work for longer than 2 months or the first big/hard to merge commit in trunk.

[Rubidium]

Some notes:

Creating a public svn is a disaster waiting to happen; I'm almost certain people will forget to #ifdef some of their code and as such its no base for the official releases and/or nightlies. Even then, the public SVN adds no benefit whatsoever to the "devs", thus they are not going to do the work and finally due to the many #ifdefs you'll get an explosion of different binaries which *all* need to be compile tested before committing. Will the latter happen? I guess not. Does the latter add an extra maintaince burden to patch (pack) makers? I guess it will. Thus you will have to end up with a repository *without* #ifdefs.

That brings me to the next point: compatability. Patches are most of the time created without a clue about the way OpenTTD works meaning they either break saveload or break networking (forgetting to save important stuff == breaking networking).

But then, for patches to be ultimately accepted they must exist as separate entities as well; fixing a bug in a patch in the patch pack without fixing the bug in the patch means the bug isn't solved. So in fact having a SVN repository will "kill" patches and make it even less likely for stuff to end up in trunk.

Thus... you would probably end up with some public git/mercurial repository where each patch has its own branch so it is easy to extract the patch. This brings a lot of merging and the likely, but it's definitely the better alternative. And... if two patches collide/depend on eachother, you can merge those two into a branch and do the stuff that merges those there before merging that into the main branch. See e.g. the system used by cargodist. Nevertheless savegame compatability will remain a mess due to the many merged things and things being merged at different times.

This method *will* make it (somewhat) easier for patch pack makers as they don't have to diff patches anymore; they can just merge them into their patch pack branch, although you probably end up with one patch pack merging everything. Still it'll be possible and probably quite easy to create other patch packs. In any case you'll have to decide who does the merging of patches into the main branch.

Using SVN for this is not a good idea because SVN is basically not suited for this type of development.

Even so...
- the OpenTTD developers will not be the ones that are going to merge trunk changes into all those patches.
- you're talking about a "private" SVN, which basically means that the OpenTTD developers have nothing to gain from your "public" SVN.
- giving OpenTTD developers more work means either less stuff ends up in the official releases, offical releases are more buggy or developers quit meaning no official releases or bug fixes.
- the number of bug fixes coming from external parties in the official OpenTTD is an extremely small portion; the OpenTTD developers are the ones that fix this. The patchers don't do this.
- making a public SVN doesn't magically create binaries.
- patch (pack) users aren't very critical when it involves bugs. Some bugs in patches go unreported over two years and once it hits trunk we'll get multiple reports about it *including* ones saying that it has happened for two years with that patch. Thus, patches need to reviewed and tested thoroughly by developers meaning it needs a lot of time.

I will definitely not be maintaining anything in that "public" SVN and I doubt whether any of the other OpenTTD developers are interested in doing that work. We care about quality, those patch packs care about quantity; the more patches the better!

Finally... usually after a few months a patch pack creator gets bored and ditches the whole patch pack. So I guess this is going to crash and burn as well. But then, what you are effectively proposing is a Community Integrated Version, which died within a mere two weeks.

Long story short: don't expect the developers to do the community's work.

[Muzzly]

First of all, thanks for your responses.

Again: who has to do the actual work integrating trunk changes in the public svn?

I was thinking that maybe developers, because they are changing something in trunk. The same way patch creators are responsible for changes in their patches. The developers DO NOT HAVE to fix bugs in patch code, they would be not responsible fixing patches to match changes of trunk. I mean if in trunk happens major changes and 3 patches would not work any more, then developers have nothing to do with it , you would simply try to insert your trunk changes in a most proper place of public code (just make diff with previous trunk version and apply this diff to current public svn). The patches have to look up to time to code and check if their patch is still working.

See above, who is responsible for fixing the public svn?

Both parts!!! And each part only for they part of the code. But of cause developers, have higher vote, because their "patch" (I mead whole trunk) has more code lines

You can change "a bit" in "a lot more", which is a serious drawback.

I cannot judge how much work it is. If it is too much work, then let it be. I have an another idea. There are another way, patch creators could make their patch for both versions (with #ifdef for public version and without #ifdef for trunk) . But I have to ask them first, if they would do this.

Which they won't do because their patch is already in "public svn" and they don't feel responsible for it anymore.

That is also not quite true. If code is in public svn it does not mean that code is final and patch will be accepted to release version. If patch does not work or has worked, but has started to malfunction with trunk changes, then just do not enable it (skip -DBUGGY_PATCH_0xx ). The developer only take responsibility for the patch only if they accpet this patch and join it with private trunk. The same way like it works now. And if they accpet it to truk ( private trunk) ,then all #ifdef are removed from patch. Only in this case YOU (devs.) have to worry about bugs. In this manner you can wait as long as patch will be 100% bug free and only then accept it, thus forcing patch creators DO SOMETHING for their patches

Furthermore people generally don't report bugs in patchpacks.

I cannot judge about previous patch packs. But in Chill's patchpack forum http://www.tt-forums.net/viewtopic.php?f=33&t=47622 people do report bugs.

OpenTTD developers have nothing to gain from your "public" SVN.

Yes that's true. For developers it would be no benefit in this concept. I was more thinking about players . Like in linux either you use stable dep packets or latest experimental with many new features and many bugs Currently the process of obtaining latest experimental openttd (with bug-full patches ) is quite difficult for general player.

Now: if you still think a "public svn" would work, you can easily create it yourself. Finding hosting somewhere shouldn't be a problem, make one patchpatck author responsible for syncing that repository to trunk and give every patch author write access to it. It's essentially the same as your proposal, only one (or more) patch patck authors are responsible for syncing the public repository with trunk, which means there is no extra workload for the developers. Personally I doubt it's going to work for longer than 2 months or the first big/hard to merge commit in trunk.

Yes it would be a good alternative. This is what I was thinking also about. And both parties would be happy in this case.

Long story short: don't expect the developers to do the community's work.

No problem. It was just a suggestion. I do not want to bother you with more work. I am pretty sure, that you have enough to do.

I just want to find an "algorithm" that makes patchpacker's work useful. If they do something it would be nice that this work could last longer than just 2 months! I think I am going to try with public SVN, hosting it somewhere outside and convincing patch creators to participate Anyway I am not loosing anything.


A bit another discussion:
I know that you are against creating plugin mechanism. But ...
If you are using firefox, is there no plugin installed in your browser ??? Does in not give you any "surfing" convenience ?
E.g. try mouse gestures, xmarks, downloadThemAll , all in one sidebar ....

Someone said that with plugins there would be more and more function to create into which plugins could be "pluged in". Yes that's true. But are you not creating sometimes new callbacks for GRF ?

[Yexo]

I know that you are against creating plugin mechanism. But ...
If you are using firefox, is there no plugin installed in your browser ??? Does in not give you any "surfing" convenience ?
E.g. try mouse gestures, xmarks, downloadThemAll , all in one sidebar ....

Someone said that with plugins there would be more and more function to create into which plugins could be "pluged in". Yes that's true. But are you not creating sometimes new callbacks for GRF ?

Yes, you can view newgrf as a "plugin system" for some graphics/behavior. However those callbacks that are added can be used by multiple newgrfs in different ways. You can see the NoAI framework in a similar way, it's also a "plugin system" for AIs. Now try to define a generic set of variables/funcions that would be useful for a "patch plugin system".

What I mean is this: for AIs there is a defined set of functions they can use. The same holds for newgrfs, there is a set of callbacks available. Those sets are expanded over time, but they are pretty generic, you can code wildly different AIs with the single AI api and also very different NewGRFs with only 1 newgrf spec. For "plugins" in the way you want it it's impossible to define a set of functions that are useful for multiple patches.

Also take a look what is happening with e.g. copy paste. Once it became part of a patch pack the developer lost interest as it was "accepted" by someone. Now every once in a while someone is syncing it to trunk with various degrees of "success" (if you may call it that). It reverting trunk fixes for example and then carrying those for many many months, me warning people about it... the revert being reverted and two months later someone updated the version that did still have the trunk revert reintroducing the same bug once again.

[Rubidium]

Again I'd like to point out MiniIN. The patch pack where the patches were updated by someone that was not the patch author, but upon inclusion of the patch in MiniIN the actual development of that patch stalled leaving the maintainer(s) of MiniIN to clean up the mess. Given it had savegame compatability just dropping a patch isn't trivial. That lead to the eventual demise of MiniIN.

The "patches" aren't tested properly still remains. What people are doing is primarily reporting the crashes, but not the inconsistencies or other issues such as desyncs. The people reporting bugs for trunk are way more focussed on inconsistencies than those that play with "put it all together and hope it works" patch packs.

Also: how are "mere" users supposed to know what -D flags they need to pass to disable a particular patch? How are "mere" users supposed to know what patches should be disabled because they are utterly broken? How are users supposed to know how to pass those flags? Are users supposed to compile their own version of the pack? Any idea how many different combinations you're going to get and how incredibly complex figuring out what happened exactly becomes without a way to precisely monitor what is in a particular binary and what isn't?

[Lord Aro]

*sticks head around the door*
I think what muzzy wants is a nightly version that is properly and regularly unstable, as the name sugests. This would generally mean either commits per dev or more devs to make those commits
So patches aren't as thoroughly tested before being committed into the nightly
*tiptoes out*

[Rubidium]

*sticks head around the door*
I think what muzzy wants is a nightly version that is properly and regularly unstable, as the name sugests. This would generally mean either commits per dev or more devs to make those commits
So patches aren't as thoroughly tested before being committed into the nightly
*tiptoes out*

Those "nightly" versions are called "integrated nightlies" or "patch packs", though nobody seems to have cared to properly set up a patch pack that can be automatically compiled at regular intervals since the MiniIN.