AutoTTD - OpenTTD Updater (and more?)

[Xaroth]

I've been working on this the past week and informed some people (well, the people on IRC will probably be aware of it) about this, and I thought it's about time I put it in the open for more oppinions.

AutoTTD

AutoTTD is a win32 application that will, much like OpenTTD AU, allow the user to automagically update their install to the version of the server they intend on playing on.
However, this is a completely different approach from Zuu's project.

AutoTTD is done in C#, which will consist of a Core Library (AutoTTD.Library), a Win32 Executable (AutoTTD) and possibly a console variant (for porting to unix/Mono).


The difference from OpenTTD AU will start at the front-end, AutoTTD has ported the basic client/server protocol for both Master Server, Game Servers and the Content Server, this allows AutoTTD to be aware of ANY OpenTTD server (listed or unlisted), and should be able to get all the required information from a server to inform the user what is required to play.
This comes in two forms, NewGRFs and Client Version.

While the client version will mainly be an automated downloader of the nightlies/trunk clients, I also plan on incorperating special builds like Head2Head (granted they are identifyable from normal servers)..
NewGRF's will be different, as this topic is (and has been) a hot potatoe for many threads about copyrights this, copyright that, yada yada yada. as such I've taken the stance that AutoTTD will ONLY check the Content Server to see if that has any of the missing (if any ofcourse) GRF's that a server requires to play, and the client will be made aware of that outcome (evidently if there are NewGRF's 'missing' from BaNaNaS that the server requires, the application will inform this to the user.). Either way, AutoTTD will NOT download ANY GRF's in this stage, while this might change, it'll be highly unlikely.


Source

I plan on keeping AutoTTD, like OpenTTD, as open source as possible.. for now I'll be looking at getting the current code hosted somewhere public (after adding licenses etc), and I encourage people to help (not just code-wise, but also suggestion-wise) with this project.

Seeing AutoTTD is done in C#, it -should- technically be possible to port the library over to unix systems through Mono. while I do have a nice network of unix systems around I don't use any of them as desktop, so don't expect me to make an unix frontend for it.. however, if you have experience with it, you're more than welcome to help out


I'll try to keep this topic up-to-date with my progress, for now I'll use this thread mainly for suggestions.

[Xaroth]

Roadmap/ToDo list:

• Thoroughly test UDP/TCP protocols (Master Server, Game Server, Content Server)
• Thoroughly test NewGRF identification (GRFID, MD5(Done)) [ needs more different NewGRF's to fully test ]
• Obtain NewGRF Name from GRF files / Add NewGRF info to protocols
• Test IPv6 compatibility
• Design Front-end
• Design console-variant

[Zuu]

Sounds like this will become a nice application with a solid foundation.

One mistake I made, was to include the word "auto" in the name. By doing this users would expect the program to read their minds and do what they think the program should do. So consider renaming it to something else. "OpenTTD Updater" is already taken however.

One name I came up with given the description of your program is:
"OpenTTD Multiplayer Launcher"

[Xaroth]

Well the planning is to make the app do as much automatically as possible, but I understand your point and I'll have to give it some thought

Rubidium came with a good point on IRC as well, which is IPv6 compatibility, which I've added to my to-do list.


[edit] (save me from double-posting) .. been fiddling with a basic lay-out for the App, seeing things should be kept as simple as possible:




Tempted to make the right area hide when nothing is selected, but the (dis)appearing might be a bit annoying at time.
As i put in earlier, i should have a look at getting GRF names, and turn the NewGRF section into a list view, red entries being not available, Green either installed or available

[Sacro]

C# doesn't have a 'Win32 execuable', it compiles to CIL which can be ran under Windows/Mono without needing recompilation.

[Xaroth]

Small update:

I tested the protocols, the IPv4 variants of client <-> Master Server, client <-> Game Server and client <-> Content Server (although i'm not 100% happy with that one yet) seem to work.. I still have to find me a willing dummy^H^H^H^H^H assistant to test it on a pure IPv6 environment.. but for now it'll work.

I also designed a small favourites system, based on a dual concept.
Favourites list will be saved in an XML format, and will be able to include external urls as favourites list (this allows, for example, for people to host a list of their private servers, which people can then reference to from their favourites list). external urls will be loaded up initial startup and parsed up until a depth of 5 (for now.. i might even reduce that to 3) .
External urls can contain query data to allow for user-specific listing, but that's more up to the people who use it later on.

I am now going to incorperate this stuff into a frontend, so it's function can be tested before I put the update things in.

[Bilbo]

Either way, AutoTTD will NOT download ANY GRF's in this stage, while this might change, it'll be highly unlikely.

Still, It can always at least open URL of download page of that GRF or URL with that GRF from GRFCrawler (if any of these is known...) - while downloading the GRF will still be manual task, people will save some clicks :)

[Xaroth]

Either way, AutoTTD will NOT download ANY GRF's in this stage, while this might change, it'll be highly unlikely.

Still, It can always at least open URL of download page of that GRF or URL with that GRF from GRFCrawler (if any of these is known...) - while downloading the GRF will still be manual task, people will save some clicks

more than doable, I was thinking of adding that as a link that opens when you click on the missing number of grfs, aka, not have it as a main feature, but still there so it works.


On that note, I need to get myself a proper SVN somewhere, so I can start releasing visible builds so people can test and give their oppinions on things.. as there's only so much I can test at this point in time.

[Bilbo]

On that note, I need to get myself a proper SVN somewhere

Sourceforge? You can choose SVN, CVS or GIT, according to your preferences ....

[Xander]

Just as a quick aside, maybe put (OTTD Updater) in the topic title? I was half expecting some newbie prattaling on about how TTD would be so much more fun if the tracks would build themselves...

[Xaroth]

Right, initial build uploaded to sourceforge (yes, TrueBrain warned me.. it's slow as hell, so no binaries, only the csproject files and the source.)

Source is uploaded to SVN (well, by the time I hit post it will be).. but it's a limited build for now, that might not even run... but people wanted to look at it, so here it is.

1) Favourites not done, some test code in AutoTTD.Browser to see how it behaves.
2) Updating/play not implemented, first things first, and then those.
3) Use at own risk, etc.


Decided to include the license as source file, will add it as separate file to output to target directory when active builds start being done.

Oh yeh, before I forget.. there's some non-standard stuff in there, credits granted where due, if something's missing.. let me know.

[Xaroth]

Small Concept change after a chat with Ammler.

I'm modifying the server info a bit, that it allows for server-specific download locations for specific grfs.

As some may be aware, not all grfs are located on BaNaNaS, and not all of them are on grfcrawler either.. either due to authors being gone, or something else... but if the server admin has permission to use and share a grf for that specific server, it wouldn't be transparent for users to actually find that grf without looking.

with this, you can specify a grf id and add a link per hash (to allow for multiple versions of the same grf depending on current server config) where the user can go to download the grf manually.

The client will -ONLY- look in the supplied list for the server it's connecting to, to ensure that no licenses are broken, and as said above, it's a feature for the remote favourite list so it will only be available if the server has it's own favourite list publicly available with those grfs (meaning there are two level of restrictions in place)

[edit] Added the system, put it to work, the xml layout shows a general gist of how favourites work, the grf entries are more aimed for servers with their own favourite list (which is where remote-inclusion comes in), so they can put links to grfs they use for their games

design2.png (15.53 KiB) Viewed 985 times

.

the screenshot shows that i have 9 out of 10 newgrf's (i removed one on purpose...) .. and the xml file that it loads (above it) shows the id and hash of that grf, resulting in the 10th comming under the "Special" entry rather than Missing.

[Xaroth]

Some updates.

1) Made favourites show up bolded on the listing, including when you are not loading the favourites list to indicate you already have it on your favourite list.
2) Made a separate form for the newGRF information, showing hash and id, and the status found by the program. you can access this by selecting a server and then double-clicking on the Yes/No behind NewGRF (yes, it will show you a blank list if you do it on a server that has no newgrfs )
3) Changed so that favourites will ALWAYS be loaded in the list. the program IGNORES servers on the master server list if they do not respond, but still show them if they are a favourite.. this mainly to allow users to be able to see if their favourite server is online or not.
4) Added initial behavior for adding/removing servers to your favourites list, nothing noticable yet
5) Changed the query routines to not query a server twice, it will filter out servers already in the list to be queried based on their ip/port. this includes master-server servers that are already added from favourite list


Also added an initial build to the SVN repository, allowing people to look around with it without compiling manually..

Keep in mind that it's not done yet, it will NOT update NOR start your game, for now it's just a pretty server-browser... so use at own risk.
You'll need .net 2.0 to run, though i've not verified that 100%... if you got 3.0/3.5 you'll be settled 100%

Known Bugs:

- the Ok button on the NewGRF info window doesn't work.. yet.

[Xaroth]

Small update:


- Fixed the Ok button

- Implemented a download mechanism that downloads (if needed) the server version required.. nothing installs -yet- .. currenly the app can differentiate the following type of servers:

-- Nightlies
-- Releases (including betas and RCs)
-- Head2Head
-- CargoDest (though that is set to just 1 revision number as there is only one, and both CargoDest and H2H share the same prefix letter for their builds... if this is to change I will adapt the app accordingly)

The app downloads the file and stores it in the %applicationdata%\AutoTTD folder, naming them <type>-<build>.zip .. so they are humanly understandable as well. the app will not re-download a file if it's already there, but will have an override for this (in case the file is corrupted?)
The urls will be configurable at a later point, both the format of the files and the base paths of the files.. but that is for a later point in development.


I will include the current build for the app with this, this was the latest running compile I used to test, this mainly to allow people to test without doing dodgy stuff with the ever slow sourceforge SVN servers or manually compile their own... if you don't trust this, don't download it... this compile comes with no guarantees of working or keeping your pc in one piece so if something breaks it's not my fault.

The build has some limitations.
1) Favourite lists are stored with the application, as a file called test.xml .. this file is not written upon closing the app but IS read when loading it... use the included test.xml file to adjust to your testing needs.. the default will point to my test server (which has 10 newgrfs, some available, some not), with 1 'special' entry, and to a second server on the same ip on a different port, that's not running.. this to test the case of 'dead' favourite servers.
2) Pressing the Update button while updating the master server list will cause unexpected behavior, so use at care.. it is safe to press the update button when the progress bar is not visible.
3) the Update process will NOT install anything, it will ONLY download the zip file to the %applicationdata%\AutoTTD folder .. feel free to test this feature.
4) The app only reads the current working directory (which you should be able to change from a shortcut) for the NewGRF's, as such it's probably best to copy some of your grfs over to the app folder to test validity etc... note that the server only re-reads newgrfs upon startup...

AutoTTD.Browser.zip

Use with caution, no guarantees etc.

(53.6 KiB) Downloaded 139 times

[Zuu]

Hello,

One thing to consider is to clean the download cache after some time. Or it will take quite a lot disk space after some time. I solve it by when I use a cached file I touch the file so that one of the time stamps changes. And then when I clean the cache dir, I just remove files that have older time stamps than configured age.

My program automatically also places a warning.txt file in the cache directory telling users that they shouldn't store their documents in there as they will get removed.

[Xaroth]

That might actually be a good idea.

I'll see what I can add, i'll most likely make an XML file that stores that data, as the %applicationdata%\AutoTTD folder will host more than just cache (or should, really).

Cheers for the pointers

[Zuu]

You can perhaps have the cache in a subdirectory keeping the cache apart from settings files for higher robustness.


When you get to a more working state as in when it starts to become useful you could add a section about your program to http://wiki.openttd.org/OpenTTD_Updaters with a short description about it. I started out that page to have a somewhat neutral page to post to users, especially since this page now have solutions available not only for Windows users.


I saw you have created a thread also in general forums. Perhaps ask a mod to lock one of them (and then link to the locked thread from the other one) unless its really necessary to have two threads. Most projects only have one thread.

[Xaroth]

I'm tempted to keep both of them active, as the main topic in General is more for releases and whatnot, and seeing I'm releasing the full source I'd like people who contribute to be able to do that in here, so that the topic in General doesn't get cluttered with dozens of diffs.

I will write something down for the wiki page, for now I want to wait until I can assign a build 0.2.0, as I want that build to have most of the 'basic' functionality in thats still missing (Favourites being the majority here).. as I can also move around some stuffs.

But first I have to sacrifice a virgin to the code gods, as I've not found any major issue with the app -yet- ... now just to hope there will be people who like the app so it can be used like it should be.

[Xaroth]

Right, small update, as I'm working on the favourites.

The format for the Favourites XML files are as follows (Server admins might want to keep this thing in mind when you want to release your own server xmls)

Code: Select all

<root>
	<favourites>
		<server ip="{server_ip}" port="{server_port}" />
		<server ip="{server_ip}" port="{server_port}">
			<grf id="{grf_id}">
				<item hash="{grf_hash}">{grf_info_url}</item>
				<item hash="{grf_hash}">{grf_info_url}</item>
			</grf>
			<grf id="{grf_id}">
				<item hash="{grf_hash}">{grf_info_url}</item>
			</grf>
		</server>
		<url>{inclusion_url}</url>
	</favourites>
</root>

{server_ip} and {server_port} should speak for themselves, really.. the interesting part comes with the grf entries.

AutoTTD will check if a favourite server has grf information defined, this means that grf information for server A grf Y will not show up on server B running grf Y ...
Bare in mind that AutoTTD will not automatically open any of those urls, this mainly to ensure that it is the user's choice to follow the link, not the choice of the person making the xml.

in {grf_info_url} you can use the tokens %id% and %hash% , both of which will be replaced by the grf ID and grf Hash ... I might be tempted to add %ip% and %port% , but that will be after the system works.
the <url>{inclusion_url}</url> part only works on the root list, I might make an option in the future to increase this to a depth of 2 (meaning your serverlist can include another serverlist).. but this would have to be combined with warnings etc as it's prone to abuse.

With this, one can build something like this, I used this as a test to verify it worked... but the data might not be accurate.

Code: Select all

<root>
  <favourites>
    <server ip="194.1.204.204" port="3979" />
    <server ip="91.198.87.142" port="3979">
      <grf id="4D470501">
        <item hash="921D08A4B3430BDCC14715B6A1288350">http://ps.openttdcoop.com/?id=%id%</item>
      </grf>
    </server>
  </favourites>
</root>

A favourite list like this (in this case I put it remote, http://194.1.204.204/server.xml ) will add Favourites to 2 servers, Moocified and #openttdcoop public server .. for the public server it adds a definition for one of the GRF's, and points that to their site. (in this case the link will be transformed to http://ps.openttdcoop.com/?id=4D470501 )

And for the record, yes, i am aware that the link replacement tags are useless as the items are added per-grf-per-hash-per-server .. but i'd rather see xml files with the replacement tags that make it readable for both admin and user, than some kind of dodgy mess that nobody can get anything wiser from.


That should be all for now, I've completed adding/removing servers to/from your favourite list by browsing, I am now working on adding servers manually and adding/removing server lists. (which is what this entire topic was about.)

[edit] some additions

You can add defined lists and defined urls within your xml file (and will not traverse across included xml files), as follows:

Code: Select all

<root>
  <favourites>
    <server ip="62.75.156.9" port="3981">
      <grflist key="testgrflist" />
    </server>
    <url>http://194.1.204.204/server.xml</url>
  </favourites>
  <defines>
    <url key="testurl">Test Url!</url>
    <grflist key="testgrflist">
      <grf id="4D470501">
        <item hash="921D08A4B3430BDCC14715B6A1288350">http://ps.openttdcoop.com/?id=%id%</item>
      </grf>
      <grf id="4D470502">
        <item hash="921D08A4B3430BDCC14715B6A1288350">
          <urllist key="testurl" />
        </item>
      </grf>
    </grflist>
  </defines>
</root>

this will define 1 url under the key "testurl" and 1 set of 2 newgrfs under the key "testgrflist".. as you can also see the second grf is including that defined url by using <urllist key="testurl" />.. this will be replaced during load with the url in the define.... this means you can use the tokens %id% %hash% (and soon %ip% %port% ) to create somewhat dynamic links without defining them for multiple servers.. you merely include the pre-defined list.

[jawsper]

You could host this project on Google code (http://code.google.com), free, easy, and svn too!