Introducing: musa. The new BaNaNaS upload tool

[TrueBrain]

Today I proudly introduce you to musa, the monkey upload service ape. Your new best friend for uploading larger files to BaNaNaS.


Why?
HTTP protocol is not designed for large uploads. Where HTTP/1.1 solved many issues for large downloads, they never resolved the issue for large uploads. You have to go to javascript or Java to solve this issue. And even then, solutions aren't that great or bulletproof.

So, to solve this issue once and for all, we designed a simple Python application. As extra bonus, it validates your package on your client, so you don't have to upload 300 MiB files to only find out after an hour something is broken in your .tar. When the client validates your .tar, the server most likely will too. Win-win.

Who (is the audience)?
Any person who uploads to BaNaNaS can (optionally) use this, but people who want to upload larger files (30+ MiB) need to use it. It also allows a bit more configuration.

How?
musa (and depgen) is available as a windows executable from http://bundles.openttdcoop.org/musa/push/LATEST/
If execution fails due to missing MSVCR libraries or due to an invalid application configuration or similar error, you need to install http://www.microsoft.com/en-us/download ... aspx?id=29
If you are on OSX or linux, checkout http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or svn://svn.openttd.org/extra/musa (svn), and install Python 2.6+ yourself, you can use it! (it possible works with Python 2.5, but no guarantees). See example.ini how you need to configure it.

More info?
Sorry, I am fresh out. The --help of musa.py is very clear I think, and example.ini has many comments. If you can figure out NML, you can figure out this too.

Source?
Of course! We - are - Open - Source! http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or http://vcs.openttd.org/svn/browser/extra/musa (svn)

Why now?
When we added (good) 32bpp support to OpenTTD a few months back, we knew people would start making 32bpp grfs, and that these files would be big. Very soon after, it became clear our current methods weren't up for the task of uploading but also distributing large files. A few months back the first questions started to be asked about it. 3 months ago we talked over solutions, and considered a stand-alone tool with its own protocol best. A few days later Rubidium wrote an implementation, named musad. This weekend I rewrote the balancer to handle this kind of traffic, cleaned up our infrastructure in relation to this, moved around files, and managed to get a version online. After several tests, we finally managed to upload zBase (270+ MiB) to BaNaNaS, and distribution worked. After patching up ottd_content to handle files this large, we now feel confident our whole system from user to backend back to the user can handle files this size. So ... we are ready for your 32bpp grfs! Your turn!

Future plans?
None. We think this is finished. Of course we will work on installer packages for Musa client, but generally speaking, this system is done. Up to the next

Credits?
Rubidium, for writing musa(d).
Rubidium, for fixing ottd_content in record time.
Myself, for programming 20 hours in this weekend fixing up the mirror system, the balancer, the infrastructure, getting musad to work, ..
Yexo, Rubidium and others for testing.


Who needs a hug?
(thumb pointing to me) *this guy* (thumb pointing to me)

[Supercheese]

Money may not be able to buy happiness, but I could buy you some hugs.

[Zephyris]

FYI, I love the name

[bremerjoe]

Hi Truebrain,
Here comes a bearhug from Germany for taking up another big project once again for the OTTD community. But you are not the only one getting a bearhug, Rubidium of course also gets one and also every other dev, mod or NewGRF provider for making OTTD so awesome!

[Nite Owl]

Another hardy THANK YOU to all involved.

[Zuu]

Over the last 1-2 weeks I have worked on musa in order to allow me to upload the Beginner Tutorial with it and set the dependencies needed between all parts involved. Today it seems like a good day to announce some results from this.

AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios
First thing to mention is that musa now supports uploading a whole set of new content types. (AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios)

Dependencies
During my work, I found and fixed problems related to validation of dependencies. It should now be possible to upload content and have their dependencies set. All you need is their md5sum, uniqueid and type. If you are like me, you think that is not "just", so that is why I now will introduce you to depgen.py:

depgen.py
This utility script helps you to determine the dependency string for content that you want to declare as dependencies for the new content that you are about to upload. As you might know, in musa you need to declare dependencies on this format:

Code: Select all

<type of content>:<unique ID>:<md5 sum>

Each type of content have their own way of storing the uniqueid and the way to compute the md5sum varies. Thus, to reduce the headache, this tool come into play to generate the dependency declaration for content that you have on your disk.

depgen.py examples

Code: Select all

> ./depgen.py ../content_download/ai/CluelessPlus-34.tar
# CluelessPlus-34.tar
dependencies = AI:CLUP:7cc9cae1cc5892435aa8f3aa7454fd06

Code: Select all

> ./depgen.py ../content_download/newgrf/OpenGFX_Trains-0.3.0.tar
# OpenGFX_Trains-0.3.0.tar
dependencies = NewGRF:4F472B31:4e0b2d42462f295f36afba90d239a323

If you pass multiple tar files, it will output a comma separated list that can be pasted in your .ini file. There is also a --list argument to create a more human friendly list output where each content is on a row of its own.

Where do I find depgen.py?
When you check out musa (or clone it using hg or git), you also get depgen.py next to musa.py. If you got the initial version of musa, you will need to update it to get depgen.py.

[Zuu]

Example - Beginner Tutorial musa setup

What is worth to know is that the Beginner Tutorial is itself not just a single bananas entry. Instead it is composed of three bananas entries that are contained in one hg repository.

Before continue reading this example, I suggest you make yourself familiar with example.ini which is included when you check out/clone musa. This example shows how you can auto-generate musa ini files.


In Beginner Tutorial, I have musa_desc_<content>.txt for each content that is part of the tutorial. These files contain the text description of each item. This text is fairly static.

Then for each content, I generate a musa_<content>.ini file using this python script: make_musa.py. In make_musa.py I have hard-coded dependency strings for dependencies outside of the Beginner Tutorial project. These dependencies are already on bananas when I call make_musa.py. What this script do however is to also to predict the dependency string that content to be uploaded will get. This is used to make the scenario depend on the GS and AI.

make_musa.py do however never call musa. For that I have a separate shell script, upload.sh which calls musa. This script uses hard-coded paths to musa which you will have to update to work in your situation. Note that this script is quite stupid and do not verify that uploading of previous entry succeeded. So I need to keep an eye on that. However the script still make sure I always specify the same files to include in the bananas package.


What may not be optimal in this solution is that it still depend on my old make_tar.py scripts which tar the GS and the AI. These tars are then analyzed to get produce the md5sum of these scritpts. A cleaner solution could be that make_musa.py iterate the files directly on the file system as upload.sh will not use the tars produced by make_tar.py but rather let musa make its own tars.

[Dave]

Congrats to all involved for making BaNaNaS even more appealing!

[Supercheese]

Congrats to all involved for making BaNaNaS even more appealing!

I see what you did there.

[Dave]

Congrats to all involved for making BaNaNaS even more appealing!

I see what you did there.

Cor! I didn't! But now I have hahaha.

[kamnet]

TrueBrain never gets enough accolade for the bunch of work he puts in.

[juzza1]

I'm trying to upload a big grf into BaNaNas using musa. However, after the program starts packacking the files, an error "No magic header" is generated and the program terminates. I was told this is because of invalid grf format, but I've no idea how to fix the problem. The grf was generated using the compiler at jenkins.openttdcoop.org with nmlc version 0.3.0.r2095. The part of grfid.py which generates the error:

Code: Select all

def get_grfid(f):
	try:
		reader = GRFIDReader(f)
		grfcontversion = 1;

		# Check version
		if reader.buffer[0:len(grfv2header)] == grfv2header:
			grfcontversion = 2
			reader.skip_bytes(len(grfv2header) + 4 + 1)

		if reader.read_size(grfcontversion) != 0x04 or reader.read_byte() != 0xFF:
			raise MusaException("No magic header")

The problematic grf is available from here. Any help would be highly appreciated. I'm also at #openttd, if you wish to contact me there.

[Rubidium]

What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.

[juzza1]

What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.

Code: Select all

python musa.py -c myconfigfile.ini -d -u juzza1 -p XXX mygrf.grf

So yes, I'm only trying to package the actual grf file.

[Rubidium]

That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.

[kyosuke1989]

That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.

Could you try to upload that linked NewGRF to the BaNaNas?

[juzza1]

So I tried to build it on a VM Fedora. Curiously, I'm getting a different error, but the grf still won't package. Am I failing on two separate environments? I'm really at a loss here, I'm doing everything by the book but getting nowhere. Rubidium, could you post the exact Python version you are currently using - maybe I could try package with that.

[Rubidium]

Python -V says: Python 2.7.5+
It is actually the Debian package with version 2.7.5-7.

In any case, I find it really odd that python behaves so differently on different platforms. Running musa with -v does add a tiny bit of extra information, though that's more the intermediate steps.

I'm quite flabbergasted by this, but don't have the time to investigate this issue any time soon (look at the work I did for OpenTTD over the last two months), i.e. don't have the time to install Windows or another operating system and spend time trying to figure out what's going wrong there.

[frosch]

Can you please try again with new musa r25762.

Maybe then you get at least the same error on both platforms

[juzza1]

After updating, I am indeed getting the "uniqueid is invalid" error on all environments. Just for the heck of it, I also installed Debian as a VM, getting the same error there aswell. Some details of python -v, if it helps:

Debian VM:
Python 2.7.3
GCC 4.7.2

Fedora VM:
Python 2.7.5
GCC 4.8.1

Windows 7 64-bit (main OS)
Python 2.7.5 (32-bit)