Introducing: musa. The new BaNaNaS upload tool

OpenTTD is a fully open-sourced reimplementation of TTD, written in C++, boasting improved gameplay and many new features.

Moderator: OpenTTD Developers

TrueBrain
OpenTTD Developer
OpenTTD Developer
Posts: 1312
Joined: 31 May 2004 09:21

Introducing: musa. The new BaNaNaS upload tool

Post by TrueBrain » 28 Oct 2012 22:27

Today I proudly introduce you to musa, the monkey upload service ape. Your new best friend for uploading larger files to BaNaNaS.


Why?
HTTP protocol is not designed for large uploads. Where HTTP/1.1 solved many issues for large downloads, they never resolved the issue for large uploads. You have to go to javascript or Java to solve this issue. And even then, solutions aren't that great or bulletproof.

So, to solve this issue once and for all, we designed a simple Python application. As extra bonus, it validates your package on your client, so you don't have to upload 300 MiB files to only find out after an hour something is broken in your .tar. When the client validates your .tar, the server most likely will too. Win-win.

Who (is the audience)?
Any person who uploads to BaNaNaS can (optionally) use this, but people who want to upload larger files (30+ MiB) need to use it. It also allows a bit more configuration.

How?
musa (and depgen) is available as a windows executable from http://bundles.openttdcoop.org/musa/push/LATEST/
If execution fails due to missing MSVCR libraries or due to an invalid application configuration or similar error, you need to install http://www.microsoft.com/en-us/download ... aspx?id=29
If you are on OSX or linux, checkout http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or svn://svn.openttd.org/extra/musa (svn), and install Python 2.6+ yourself, you can use it! (it possible works with Python 2.5, but no guarantees). See example.ini how you need to configure it.

More info?
Sorry, I am fresh out. The --help of musa.py is very clear I think, and example.ini has many comments. If you can figure out NML, you can figure out this too.

Source?
Of course! We - are - Open - Source! http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or http://vcs.openttd.org/svn/browser/extra/musa (svn)

Why now?
When we added (good) 32bpp support to OpenTTD a few months back, we knew people would start making 32bpp grfs, and that these files would be big. Very soon after, it became clear our current methods weren't up for the task of uploading but also distributing large files. A few months back the first questions started to be asked about it. 3 months ago we talked over solutions, and considered a stand-alone tool with its own protocol best. A few days later Rubidium wrote an implementation, named musad. This weekend I rewrote the balancer to handle this kind of traffic, cleaned up our infrastructure in relation to this, moved around files, and managed to get a version online. After several tests, we finally managed to upload zBase (270+ MiB) to BaNaNaS, and distribution worked. After patching up ottd_content to handle files this large, we now feel confident our whole system from user to backend back to the user can handle files this size. So ... we are ready for your 32bpp grfs! Your turn!

Future plans?
None. We think this is finished. Of course we will work on installer packages for Musa client, but generally speaking, this system is done. Up to the next :D

Credits?
Rubidium, for writing musa(d).
Rubidium, for fixing ottd_content in record time.
Myself, for programming 20 hours in this weekend fixing up the mirror system, the balancer, the infrastructure, getting musad to work, ..
Yexo, Rubidium and others for testing.


Who needs a hug?
(thumb pointing to me) *this guy* (thumb pointing to me)
The only thing necessary for the triumph of evil is for good men to do nothing.

Supercheese
Tycoon
Tycoon
Posts: 1660
Joined: 16 Dec 2007 22:24
Location: Idaho, USA

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Supercheese » 28 Oct 2012 22:48

Money may not be able to buy happiness, but I could buy you some hugs. :wink:
Eyecandy Road Vehicles | Fake Subways | Supercheese's NewObjects

"Fashions and cultures change, but steam trains shall always be majestic."
-Professor Hershel Layton

User avatar
Zephyris
Tycoon
Tycoon
Posts: 2826
Joined: 16 May 2007 16:59

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Zephyris » 28 Oct 2012 22:50

FYI, I love the name :)

bremerjoe
Engineer
Engineer
Posts: 97
Joined: 24 May 2011 21:01

Re: Introducing: musa. The new BaNaNaS upload tool

Post by bremerjoe » 29 Oct 2012 01:03

Hi Truebrain,
Here comes a bearhug from Germany for taking up another big project once again for the OTTD community. But you are not the only one getting a bearhug, Rubidium of course also gets one and also every other dev, mod or NewGRF provider for making OTTD so awesome! :bow:
Everybody can contribute to OTTD! It does not require you to be an IT specialist or designer. And it does not take days or weeks of your time either.
There are many simple yet important tasks like translations, investigating facts and details and many other that need work and if you help a bit then the experienced developers can focus more on making OTTD even more awesome!
OTTD = Awesomeness created by a whole lot of volunteers, be one of them if you like the game!

User avatar
Nite Owl
Tycoon
Tycoon
Posts: 1889
Joined: 06 Mar 2007 19:32
Location: In The Dark

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Nite Owl » 29 Oct 2012 03:35

Another hardy THANK YOU to all involved.
Humor is the second most subjective thing on the planet
------------------------------------------------------------
Brevity is the soul of wit and obscenity is its downfall
--------------------------------------------------------
Good Night And Good Luck - Read You Soon

User avatar
Zuu
OpenTTD Developer
OpenTTD Developer
Posts: 4553
Joined: 09 Jun 2003 18:21
Location: /home/sweden

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Zuu » 14 Apr 2013 20:41

Over the last 1-2 weeks I have worked on musa in order to allow me to upload the Beginner Tutorial with it and set the dependencies needed between all parts involved. Today it seems like a good day to announce some results from this.

AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios
First thing to mention is that musa now supports uploading a whole set of new content types. (AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios)

Dependencies
During my work, I found and fixed problems related to validation of dependencies. It should now be possible to upload content and have their dependencies set. All you need is their md5sum, uniqueid and type. If you are like me, you think that is not "just", so that is why I now will introduce you to depgen.py:

depgen.py
This utility script helps you to determine the dependency string for content that you want to declare as dependencies for the new content that you are about to upload. As you might know, in musa you need to declare dependencies on this format:

Code: Select all

<type of content>:<unique ID>:<md5 sum>
Each type of content have their own way of storing the uniqueid and the way to compute the md5sum varies. Thus, to reduce the headache, this tool come into play to generate the dependency declaration for content that you have on your disk.

depgen.py examples

Code: Select all

> ./depgen.py ../content_download/ai/CluelessPlus-34.tar
# CluelessPlus-34.tar
dependencies = AI:CLUP:7cc9cae1cc5892435aa8f3aa7454fd06

Code: Select all

> ./depgen.py ../content_download/newgrf/OpenGFX_Trains-0.3.0.tar
# OpenGFX_Trains-0.3.0.tar
dependencies = NewGRF:4F472B31:4e0b2d42462f295f36afba90d239a323
If you pass multiple tar files, it will output a comma separated list that can be pasted in your .ini file. There is also a --list argument to create a more human friendly list output where each content is on a row of its own.

Where do I find depgen.py?
When you check out musa (or clone it using hg or git), you also get depgen.py next to musa.py. If you got the initial version of musa, you will need to update it to get depgen.py.
My OpenTTD contributions (AIs, Game Scripts, patches, OpenTTD Auto Updater, and some sprites)
Junctioneer (a traffic intersection simulator)

User avatar
Zuu
OpenTTD Developer
OpenTTD Developer
Posts: 4553
Joined: 09 Jun 2003 18:21
Location: /home/sweden

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Zuu » 14 Apr 2013 21:01

Example - Beginner Tutorial musa setup

What is worth to know is that the Beginner Tutorial is itself not just a single bananas entry. Instead it is composed of three bananas entries that are contained in one hg repository.

Before continue reading this example, I suggest you make yourself familiar with example.ini which is included when you check out/clone musa. This example shows how you can auto-generate musa ini files.


In Beginner Tutorial, I have musa_desc_<content>.txt for each content that is part of the tutorial. These files contain the text description of each item. This text is fairly static.

Then for each content, I generate a musa_<content>.ini file using this python script: make_musa.py. In make_musa.py I have hard-coded dependency strings for dependencies outside of the Beginner Tutorial project. These dependencies are already on bananas when I call make_musa.py. What this script do however is to also to predict the dependency string that content to be uploaded will get. This is used to make the scenario depend on the GS and AI.

make_musa.py do however never call musa. For that I have a separate shell script, upload.sh which calls musa. This script uses hard-coded paths to musa which you will have to update to work in your situation. Note that this script is quite stupid and do not verify that uploading of previous entry succeeded. So I need to keep an eye on that. However the script still make sure I always specify the same files to include in the bananas package.


What may not be optimal in this solution is that it still depend on my old make_tar.py scripts which tar the GS and the AI. These tars are then analyzed to get produce the md5sum of these scritpts. A cleaner solution could be that make_musa.py iterate the files directly on the file system as upload.sh will not use the tars produced by make_tar.py but rather let musa make its own tars.
My OpenTTD contributions (AIs, Game Scripts, patches, OpenTTD Auto Updater, and some sprites)
Junctioneer (a traffic intersection simulator)

User avatar
Dave
Moderator
Moderator
Posts: 17203
Joined: 26 Dec 2005 20:19
Location: Harringay, North London
Contact:

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Dave » 15 Apr 2013 17:18

Congrats to all involved for making BaNaNaS even more appealing!
Official TT-Dave Fan Club

Dave's Screenshot Thread! - Albion: A fictional Britain
Flickr


Why be a song when you can be a symphony? r is a...

Supercheese
Tycoon
Tycoon
Posts: 1660
Joined: 16 Dec 2007 22:24
Location: Idaho, USA

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Supercheese » 15 Apr 2013 18:33

Dave W wrote:Congrats to all involved for making BaNaNaS even more appealing!
I see what you did there. :roll:
Eyecandy Road Vehicles | Fake Subways | Supercheese's NewObjects

"Fashions and cultures change, but steam trains shall always be majestic."
-Professor Hershel Layton

User avatar
Dave
Moderator
Moderator
Posts: 17203
Joined: 26 Dec 2005 20:19
Location: Harringay, North London
Contact:

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Dave » 15 Apr 2013 19:29

Supercheese wrote:
Dave W wrote:Congrats to all involved for making BaNaNaS even more appealing!
I see what you did there. :roll:
Cor! I didn't! But now I have hahaha.
Official TT-Dave Fan Club

Dave's Screenshot Thread! - Albion: A fictional Britain
Flickr


Why be a song when you can be a symphony? r is a...

User avatar
kamnet
Moderator
Moderator
Posts: 6778
Joined: 28 Sep 2009 17:15
Location: Eastern KY
Contact:

Re: Introducing: musa. The new BaNaNaS upload tool

Post by kamnet » 15 Apr 2013 23:45

TrueBrain never gets enough accolade for the bunch of work he puts in.

juzza1
Engineer
Engineer
Posts: 94
Joined: 22 Oct 2004 19:27

Re: Introducing: musa. The new BaNaNaS upload tool

Post by juzza1 » 12 Sep 2013 15:41

I'm trying to upload a big grf into BaNaNas using musa. However, after the program starts packacking the files, an error "No magic header" is generated and the program terminates. I was told this is because of invalid grf format, but I've no idea how to fix the problem. The grf was generated using the compiler at jenkins.openttdcoop.org with nmlc version 0.3.0.r2095. The part of grfid.py which generates the error:

Code: Select all

def get_grfid(f):
	try:
		reader = GRFIDReader(f)
		grfcontversion = 1;

		# Check version
		if reader.buffer[0:len(grfv2header)] == grfv2header:
			grfcontversion = 2
			reader.skip_bytes(len(grfv2header) + 4 + 1)

		if reader.read_size(grfcontversion) != 0x04 or reader.read_byte() != 0xFF:
			raise MusaException("No magic header")
The problematic grf is available from here. Any help would be highly appreciated. I'm also at #openttd, if you wish to contact me there.

Rubidium
OpenTTD Developer
OpenTTD Developer
Posts: 3815
Joined: 09 Feb 2006 19:15

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Rubidium » 12 Sep 2013 16:56

What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.

juzza1
Engineer
Engineer
Posts: 94
Joined: 22 Oct 2004 19:27

Re: Introducing: musa. The new BaNaNaS upload tool

Post by juzza1 » 12 Sep 2013 21:26

Rubidium wrote:What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.

Code: Select all

python musa.py -c myconfigfile.ini -d -u juzza1 -p XXX mygrf.grf
So yes, I'm only trying to package the actual grf file.

Rubidium
OpenTTD Developer
OpenTTD Developer
Posts: 3815
Joined: 09 Feb 2006 19:15

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Rubidium » 12 Sep 2013 21:49

That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.

User avatar
kyosuke1989
Transport Coordinator
Transport Coordinator
Posts: 269
Joined: 24 Mar 2008 13:04
Location: Finland

Re: Introducing: musa. The new BaNaNaS upload tool

Post by kyosuke1989 » 12 Sep 2013 21:59

Rubidium wrote:That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.
Could you try to upload that linked NewGRF to the BaNaNas?

juzza1
Engineer
Engineer
Posts: 94
Joined: 22 Oct 2004 19:27

Re: Introducing: musa. The new BaNaNaS upload tool

Post by juzza1 » 12 Sep 2013 22:38

So I tried to build it on a VM Fedora. Curiously, I'm getting a different error, but the grf still won't package. Am I failing on two separate environments? I'm really at a loss here, I'm doing everything by the book but getting nowhere. Rubidium, could you post the exact Python version you are currently using - maybe I could try package with that.
Attachments
musa.png
What I'm working with
(358.67 KiB) Downloaded 4 times

Rubidium
OpenTTD Developer
OpenTTD Developer
Posts: 3815
Joined: 09 Feb 2006 19:15

Re: Introducing: musa. The new BaNaNaS upload tool

Post by Rubidium » 13 Sep 2013 04:55

Python -V says: Python 2.7.5+
It is actually the Debian package with version 2.7.5-7.

In any case, I find it really odd that python behaves so differently on different platforms. Running musa with -v does add a tiny bit of extra information, though that's more the intermediate steps.

I'm quite flabbergasted by this, but don't have the time to investigate this issue any time soon (look at the work I did for OpenTTD over the last two months), i.e. don't have the time to install Windows or another operating system and spend time trying to figure out what's going wrong there.

frosch
OpenTTD Developer
OpenTTD Developer
Posts: 979
Joined: 20 Dec 2006 13:31
Location: Aschaffenburg

Re: Introducing: musa. The new BaNaNaS upload tool

Post by frosch » 13 Sep 2013 16:37

Can you please try again with new musa r25762.

Maybe then you get at least the same error on both platforms :p
⢇⡸⢸⠢⡇⡇⢎⡁⢎⡱⢸⡱⢸⣭⠀⢸⢜⢸⢸⣀⢸⣀⢸⣭⢸⡱⠀⢰⠭⡆⣫⠰⣉⢸⢸⠀⢰⠭⡆⡯⡆⢹⠁⠀⢐⠰⡁

juzza1
Engineer
Engineer
Posts: 94
Joined: 22 Oct 2004 19:27

Re: Introducing: musa. The new BaNaNaS upload tool

Post by juzza1 » 13 Sep 2013 17:08

After updating, I am indeed getting the "uniqueid is invalid" error on all environments. Just for the heck of it, I also installed Debian as a VM, getting the same error there aswell. Some details of python -v, if it helps:

Debian VM:
Python 2.7.3
GCC 4.7.2

Fedora VM:
Python 2.7.5
GCC 4.8.1

Windows 7 64-bit (main OS)
Python 2.7.5 (32-bit)

Post Reply

Return to “General OpenTTD”

Who is online

Users browsing this forum: Urmila67 and 2 guests