NewGRF Web Service

[MHTransport]

There is currently difficulty for users to get vehicle sets and for creators to distribute them.

I propose we create a simple web service for accessing them from the game. We have been using libcurl at work, and that is a simple cross-platform open source way to download a file.

The web service would consist of a url that the user adds to a list in OpenTTD OpenTTD would the http-get the url. The url would be a file containing an entry for each available graphic set. The entry would contain a uuid, a version time, the name of the set, a description and the version time of the exe needed to use it; and a file hash.

OpenTTD would parse the list and filter out any that required a newer exe. Then it would filter out an sets with the same uuid and a same/newer version that it already had installed.

It could then present to the user the list of the name and description of not yet installed graphic sets. Then the user could click get we and OpenTTD downloads it hashes it and probably installs it.

And you could have a button on the start page check for updated/new graphic sets.

Save games could store the url of the web services, the ur & uuidl of the graphic sets, and their hashes of all sets used. So when you post it to the web and it is loaded all uninstalled sets could be [optionally] retrieved and installed.

The same for connecting to a online server.

Now the fields above must be stored in the set except the hash and that should be authoritative.
The entry would contain a uuid, a version time, the name of the set, a description and the version time of the exe needed to use it

You could even store multiple translated descriptions.

So in a tar archive create the files:
for the graphic set uuid a UTF8 text/plain file called "uuid"
for the graphic version uuid a UTF8 text/plain file called "version" in ISO-8601 UTC time with seconds of when this version was prepared.
for the brief name, UTF8 text/plain files name_en_US, name_jp
for the description, UTF8 text/plain files desc_en_US, desc_jp
for the version, UTF8 text/plain file called "requires" with the time of the enabling revision in ISO-8601 UTC time with seconds.
then add the rest of the graphics.

[Rubidium]

OpenTTD will never support downloading of NewGRFs due to copyright related issues that have been discussed a gazillion times already and I do not want to do it all over again.

[sickie]

Hi.
Well, then this 'web service'/NewGRF repository should only contain NewGRFs licensed under a free(as in freedom)-licnese. Although I think that the NewGRF management in OpenTTD should be improved first.

[MHTransport]

OpenTTD will never support downloading of NewGRFs due to copyright related issues that have been discussed a gazillion times already and I do not want to do it all over again.

I missed that conversation. Can anyone link me to the forum posts.

I can see there might be issues with automatic download from saved game or server. This could be over come by adding a flag to the graphic set: allow distribution which if not present would stop the set being listed and thus automatically retrieved.

For other reasons I guess I will read the previous discussions.

[Rubidium]

Hi.
Well, then this 'web service'/NewGRF repository should only contain NewGRFs licensed under a free(as in freedom)-licnese. Although I think that the NewGRF management in OpenTTD should be improved first.

How can the service determine whether it is released under a free license? The only way to do that properly is via the following:

I can see there might be issues with automatic download from saved game or server. This could be over come by adding a flag to the graphic set: allow distribution which if not present would stop the set being listed and thus automatically retrieved.

The major problem with this is that the NewGRF developers of the most used NewGRFs (ECS, dbsetxl, long vehicles, PBI, UKRS and such) do not like their NewGRFs to be automatically downloaded. This basically comes to the point that downloading the NewGRFs is useless because you still need to download a major part of the NewGRFs.
Adding the flag is also quite dubious, because the owner of the server can simply say that all NewGRFs it uses has the flag set. This would make the NewGRF authors that do not want their NewGRFs to be automatically distributed 'angry', which might *very well* result in deliberately making the NewGRFs incompatible with OpenTTD.
So to sum it up: adding automatic downloading of NewGRFs adds a great risk in losing the ability to actually use a major set of the NewGRFs. So it is: ease of use of NewGRFs vs. use of NewGRFs.

I missed that conversation. Can anyone link me to the forum posts.

A simple search would've given you (for example) http://www.tt-forums.net/viewtopic.php?f=36&t=32764

[Bilbo]

While auto-download is not the solution due to copyright stuff, there could be done things to ease downloading:

1. Every GRF have ID.
2. Most GRF's are present at GRFcrawler
3. Combine 1 and 2 to create "autodownload" link that will open the page at grfcrawler listing all missing GRF id's, so you do not have to type it in grfcraler manually (you can't copy&paste it from openttd window). Of course you have to download each of the GRF's manually, but you will have list of all of them conveniently on one page, each having either direct download link or at least link to GRF's website (according of what have the GRF author put in the grfcrawler)

[Rubidium]

1. Is true, but for OpenTTD it is not only the GRF ID that counts, but also the MD5 checksum and that combination cannot be searched for.

2. When writing the new masterserver/updater structure to include NewGRF information, I found out that a pretty large amount of NewGRFs are actually NOT on GRF crawler. If they are on GRF crawler they link you to a site or a forum thread where you still have to search for the version of the NewGRF you actually need. George's NewGRFs are, for example, updated quite often which means that it is likely that the NewGRFs that your network game needs aren't on his website anymore.

3. When combining the above: the 'automatic' download links that will be 'generated' will still need the people to manually search the website/thread to find the correct one, which might not be the correct one after all (md5 checksums do not match). This makes the 'feature' not behave as it should is such a large number of the cases that people get really annoyed by it and they'll come flooding the problems forum because the download links given to them aren't the right links.

[Ailure]

I used to be in favor of a autodownload system, but after the first openTTDcoop GRF pack got released I stopped seeing it as necessary. Sure it's not a definate GRF pack, but it's just easy as Server admin to point at the openTTDcoop page and just tell users to download the GRF pack. And if the openTTDcoop pack doesn't cover the GRF's you want, you can always make your own pack... after asking the various GRF authors permission where it's needed of course.

...is there a "features that won't be done" list somewhere? If not, one should be made at the Wiki. There's that "common suggestions" thread but there's a difference between common suggestions and ones that wouldn't be favorable to implement. :/

[audigex]

Why stop everyone because some people don't want their work distributed?

Let those who license their work under a less restrictive licence have it distributed through a central location. The others can either distribute it through the forum as normal, or may choose to change the restriction.

There's no requirement for it to be include, but we may aswell help those who would like it. (And it may encourage some of the bigger grf's creators to make their grfs freely distributable, when others start getting more exposure than them)

[dihedral]

there is the openttdcoop grfpack, we talk to authers and ask if they are fine with us distributing their grf's and to what conditions.
and we only include what we are allowed to by the authors.

this (not) feature has been discussed many times, and the answer has remaind the same over years, starting that discussion again will not be helpful to anyone! just understand that it will not happen and let that topic rest.

[audigex]

There's nothing wrong with discussion, is there?

At the end of the day isn't one of the major aims to make getting and customising the game as easy as possible? Okay, it's not that hard to get the coop pack - but you have to be on the forum and find the right topic to even find it. It's not the most complex installation process for it, but there are several steps including traversing your hard drive - some people aren't even confident enough to copy things into the /data/ folder.

It's easy for us to forget that we're among the more involved group - and probably have a decent amount of experience in computers... try getting a 10 year old to install a grf pack and see how he does.

[dihedral]

what dont you understand? the devs have expressed themselves many times very clearly how they stand towards such an implementation.
it will not happen!

and yes, there is a bad thing about discussing something over and over again: it wasts time! the devs do have a life outside of openttd, they can either spend their time reading your (not so unique) posts and respond ywan (yet with another no), or they can actually invest their time in some more interesting stuff, like... fixing bugs? implementing decent features?
there is much to do, annoying the devs with such repeated topics will not do any good, not to you, not to the devs, not to the community.

further more, OpenTTD does not have to handle every persons computer issues! just because i cannot install a grf into my data/ folder does not mean OpenTTD has to do it for me.
yes, OpenTTD is also not responsible for the grf author's standing to the distribution of his/her grf's, and does not have to pay attention to the grf's licensing, this is in the responsibility of the server admin / player.
OpenTTD does not have a bad 'relationship' to the grf authors, and on top of that, grf author's can make their grf's incompatible with OpenTTD (dont ask what purpose that could have!). I am guessing you like the grf's you play with - best thing to do is not to get the grf authors annoyed with a redistribution of their work they do not agree with!

hope that helps your thoughts to the 'webservice'

[Ammler]

The main problem a repo will have, is the handling of different versions, because you need exactly the same version as the server uses, the REPO needs you to provide with many different vesions of every NewGRF.

Else it might be easy to add a new field to the GRFCrawler with a direct link to the GRF and the Readme/Guide. And then patch OTTD to use those links. It would then depense on the Authors if they use those new fields.

Greets
Ammler

[wolf]

hope that helps your thoughts to the 'webservice'

Please stop the BS rain.

[Ralph]

Disclaimer: I really have no prior knowledge of the newGRF 'scene' so to speak.

I think the original posters idea is a great one, as I currently don't play the game with any new graphics, partly because it seems to be such a complex process making sure its all compatible, etc, and having a nice simple interface on the menu would greatly encourage the more casual openTTD player to try using more of the great stuff the community has produced.

True some developers may not want their work distributed in this way, but its trivial enough to ensure that only those who do want in have their content auto-downloaded by forcing creators to first upload their content to the system (and agreeing to allow it to be distributed in the process). Once the system is up and running, maybe those hold outs will come to realise that having what they have created used and enjoyed is better than the power trip they get from making people jump through hoops to get it.

[Rubidium]

True some developers may not want their work distributed in this way, but its trivial enough to ensure that only those who do want in have their content auto-downloaded by forcing creators to first upload their content to the system

It is also trivial to fake a creator and upload their content to the system, essentially nullifying that uploading content to the system means the creator agreed with it.

[Ralph]

True some developers may not want their work distributed in this way, but its trivial enough to ensure that only those who do want in have their content auto-downloaded by forcing creators to first upload their content to the system

It is also trivial to fake a creator and upload their content to the system, essentially nullifying that uploading content to the system means the creator agreed with it.

Then moderate it. This would require more work by the community, but does not have any great technical skill/knowledge requirements on the volunteers involved. Once someone has made a submission give them the keys to update it themselves. Also you can have a mechanism in place to automaticly reject known bad files by their hash or something, to make sure they don't reach human moderators.

Progress should not be held back by the whims of a few..

[DredFurst]

maybe a better way of approaching a solution, instead of providing a universal download service, make TTD servers responsible for distributing their newgrf's ala every other game that supports custom content. So instead of querying a third resource you can just send them through TTD, or even provide a web url download to the file directly off of their host.

Doing it this way makes the download service the responsibility of the server owner not the openTTD developers

[Rubidium]

Doing it this way makes the download service the responsibility of the server owner not the openTTD developers

And then because people don't care about the wishes of the NewGRF authors they just distribute everything even when it is not allowed. This will result in NewGRFs made to automatically disable themselves when loading in OpenTTD.

And DO NOT think that disabling NewGRFs for OpenTTD is not going to happen because it already did, even does, happen.

So in the end there are less NewGRFs where people can chose of and the OpenTTD developers get blamed for it because we implemented the distribution. So even when it isn't our direct responsibility, in the end we will be held responsible for the consequences of the decision.

So in short: not going to happen and discussing about it is pointless because it is not going to happen, unless YOU get the written and signed approval of ALL persons that ever made a NewGRF that ALL of their NewGRFs may be distributed without ANY restrictions.

[Ralph]

So in the end there are less NewGRFs where people can chose of and the OpenTTD developers get blamed for it because we implemented the distribution. So even when it isn't our direct responsibility, in the end we will be held responsible for the consequences of the decision.

So in short: not going to happen and discussing about it is pointless because it is not going to happen, unless YOU get the written and signed approval of ALL persons that ever made a NewGRF that ALL of their NewGRFs may be distributed without ANY restrictions.

Would someone explain the reason an entire community is going to throw their toys out of the pram and sulk over the distribution of a few graphics? Clearly the 'free as in freedom' mindset I associate with GPL software is not present here.

It's a pity, because a feature like this would probably also really help the effort to get the game running without any original graphics, only free ones.