Transport Tycoon Forums

The place to talk about Transport Tycoon
It is currently Tue Jul 25, 2017 6:44 pm

All times are UTC




Post new topic  Reply to topic  [ 37 posts ]  Go to page 1 2 Next
Author Message
PostPosted: Sun Oct 28, 2012 10:27 pm 
Offline
OpenTTD Developer
OpenTTD Developer

Joined: Mon May 31, 2004 9:21 am
Posts: 1292
Today I proudly introduce you to musa, the monkey upload service ape. Your new best friend for uploading larger files to BaNaNaS.


Why?
HTTP protocol is not designed for large uploads. Where HTTP/1.1 solved many issues for large downloads, they never resolved the issue for large uploads. You have to go to javascript or Java to solve this issue. And even then, solutions aren't that great or bulletproof.

So, to solve this issue once and for all, we designed a simple Python application. As extra bonus, it validates your package on your client, so you don't have to upload 300 MiB files to only find out after an hour something is broken in your .tar. When the client validates your .tar, the server most likely will too. Win-win.

Who (is the audience)?
Any person who uploads to BaNaNaS can (optionally) use this, but people who want to upload larger files (30+ MiB) need to use it. It also allows a bit more configuration.

How?
musa (and depgen) is available as a windows executable from http://bundles.openttdcoop.org/musa/push/LATEST/
If execution fails due to missing MSVCR libraries or due to an invalid application configuration or similar error, you need to install http://www.microsoft.com/en-us/download ... aspx?id=29
If you are on OSX or linux, checkout http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or svn://svn.openttd.org/extra/musa (svn), and install Python 2.6+ yourself, you can use it! (it possible works with Python 2.5, but no guarantees). See example.ini how you need to configure it.

More info?
Sorry, I am fresh out. The --help of musa.py is very clear I think, and example.ini has many comments. If you can figure out NML, you can figure out this too.

Source?
Of course! We - are - Open - Source! http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or http://vcs.openttd.org/svn/browser/extra/musa (svn)

Why now?
When we added (good) 32bpp support to OpenTTD a few months back, we knew people would start making 32bpp grfs, and that these files would be big. Very soon after, it became clear our current methods weren't up for the task of uploading but also distributing large files. A few months back the first questions started to be asked about it. 3 months ago we talked over solutions, and considered a stand-alone tool with its own protocol best. A few days later Rubidium wrote an implementation, named musad. This weekend I rewrote the balancer to handle this kind of traffic, cleaned up our infrastructure in relation to this, moved around files, and managed to get a version online. After several tests, we finally managed to upload zBase (270+ MiB) to BaNaNaS, and distribution worked. After patching up ottd_content to handle files this large, we now feel confident our whole system from user to backend back to the user can handle files this size. So ... we are ready for your 32bpp grfs! Your turn!

Future plans?
None. We think this is finished. Of course we will work on installer packages for Musa client, but generally speaking, this system is done. Up to the next :D

Credits?
Rubidium, for writing musa(d).
Rubidium, for fixing ottd_content in record time.
Myself, for programming 20 hours in this weekend fixing up the mirror system, the balancer, the infrastructure, getting musad to work, ..
Yexo, Rubidium and others for testing.


Who needs a hug?
(thumb pointing to me) *this guy* (thumb pointing to me)

_________________
The only thing necessary for the triumph of evil is for good men to do nothing.


Top
   
PostPosted: Sun Oct 28, 2012 10:48 pm 
Offline
Tycoon
Tycoon

Joined: Sun Dec 16, 2007 10:24 pm
Posts: 1613
Location: Idaho, USA
Money may not be able to buy happiness, but I could buy you some hugs. :wink:

_________________
Eyecandy Road Vehicles | Fake Subways | Supercheese's NewObjects

"Fashions and cultures change, but steam trains shall always be majestic."
-Professor Hershel Layton


Top
   
PostPosted: Sun Oct 28, 2012 10:50 pm 
Offline
Tycoon
Tycoon
User avatar

Joined: Wed May 16, 2007 4:59 pm
Posts: 2759
FYI, I love the name :)

_________________
GRVTS/eGRVTS --- Generic Tram Set --- UK Town Set --- zBase ---RichardWheeler.net


Top
   
PostPosted: Mon Oct 29, 2012 1:03 am 
Offline
Engineer
Engineer

Joined: Tue May 24, 2011 9:01 pm
Posts: 97
Hi Truebrain,
Here comes a bearhug from Germany for taking up another big project once again for the OTTD community. But you are not the only one getting a bearhug, Rubidium of course also gets one and also every other dev, mod or NewGRF provider for making OTTD so awesome! :bow:

_________________
Everybody can contribute to OTTD! It does not require you to be an IT specialist or designer. And it does not take days or weeks of your time either.
There are many simple yet important tasks like translations, investigating facts and details and many other that need work and if you help a bit then the experienced developers can focus more on making OTTD even more awesome!
OTTD = Awesomeness created by a whole lot of volunteers, be one of them if you like the game!


Top
   
PostPosted: Mon Oct 29, 2012 3:35 am 
Offline
Tycoon
Tycoon
User avatar

Joined: Tue Mar 06, 2007 7:32 pm
Posts: 1889
Location: In The Dark
Another hardy THANK YOU to all involved.

_________________
Humor is the second most subjective thing on the planet
------------------------------------------------------------
Brevity is the soul of wit and obscenity is its downfall
--------------------------------------------------------
Good Night And Good Luck - Read You Soon


Top
   
PostPosted: Sun Apr 14, 2013 8:41 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Mon Jun 09, 2003 6:21 pm
Posts: 4530
Location: /home/sweden
Over the last 1-2 weeks I have worked on musa in order to allow me to upload the Beginner Tutorial with it and set the dependencies needed between all parts involved. Today it seems like a good day to announce some results from this.

AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios
First thing to mention is that musa now supports uploading a whole set of new content types. (AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios)

Dependencies
During my work, I found and fixed problems related to validation of dependencies. It should now be possible to upload content and have their dependencies set. All you need is their md5sum, uniqueid and type. If you are like me, you think that is not "just", so that is why I now will introduce you to depgen.py:

depgen.py
This utility script helps you to determine the dependency string for content that you want to declare as dependencies for the new content that you are about to upload. As you might know, in musa you need to declare dependencies on this format:
Code:
<type of content>:<unique ID>:<md5 sum>


Each type of content have their own way of storing the uniqueid and the way to compute the md5sum varies. Thus, to reduce the headache, this tool come into play to generate the dependency declaration for content that you have on your disk.

depgen.py examples
Code:
> ./depgen.py ../content_download/ai/CluelessPlus-34.tar
# CluelessPlus-34.tar
dependencies = AI:CLUP:7cc9cae1cc5892435aa8f3aa7454fd06


Code:
> ./depgen.py ../content_download/newgrf/OpenGFX_Trains-0.3.0.tar
# OpenGFX_Trains-0.3.0.tar
dependencies = NewGRF:4F472B31:4e0b2d42462f295f36afba90d239a323


If you pass multiple tar files, it will output a comma separated list that can be pasted in your .ini file. There is also a --list argument to create a more human friendly list output where each content is on a row of its own.

Where do I find depgen.py?
When you check out musa (or clone it using hg or git), you also get depgen.py next to musa.py. If you got the initial version of musa, you will need to update it to get depgen.py.

_________________
My OpenTTD contributions (AIs, Game Scripts, patches, OpenTTD Auto Updater, and some sprites)
Junctioneer (a traffic intersection simulator)


Top
   
PostPosted: Sun Apr 14, 2013 9:01 pm 
Offline
OpenTTD Developer
OpenTTD Developer
User avatar

Joined: Mon Jun 09, 2003 6:21 pm
Posts: 4530
Location: /home/sweden
Example - Beginner Tutorial musa setup

What is worth to know is that the Beginner Tutorial is itself not just a single bananas entry. Instead it is composed of three bananas entries that are contained in one hg repository.

Before continue reading this example, I suggest you make yourself familiar with example.ini which is included when you check out/clone musa. This example shows how you can auto-generate musa ini files.


In Beginner Tutorial, I have musa_desc_<content>.txt for each content that is part of the tutorial. These files contain the text description of each item. This text is fairly static.

Then for each content, I generate a musa_<content>.ini file using this python script: make_musa.py. In make_musa.py I have hard-coded dependency strings for dependencies outside of the Beginner Tutorial project. These dependencies are already on bananas when I call make_musa.py. What this script do however is to also to predict the dependency string that content to be uploaded will get. This is used to make the scenario depend on the GS and AI.

make_musa.py do however never call musa. For that I have a separate shell script, upload.sh which calls musa. This script uses hard-coded paths to musa which you will have to update to work in your situation. Note that this script is quite stupid and do not verify that uploading of previous entry succeeded. So I need to keep an eye on that. However the script still make sure I always specify the same files to include in the bananas package.


What may not be optimal in this solution is that it still depend on my old make_tar.py scripts which tar the GS and the AI. These tars are then analyzed to get produce the md5sum of these scritpts. A cleaner solution could be that make_musa.py iterate the files directly on the file system as upload.sh will not use the tars produced by make_tar.py but rather let musa make its own tars.

_________________
My OpenTTD contributions (AIs, Game Scripts, patches, OpenTTD Auto Updater, and some sprites)
Junctioneer (a traffic intersection simulator)


Top
   
PostPosted: Mon Apr 15, 2013 5:18 pm 
Offline
Moderator
Moderator
User avatar

Joined: Mon Dec 26, 2005 8:19 pm
Posts: 17021
Location: Harringay, North London
Congrats to all involved for making BaNaNaS even more appealing!

_________________
Albion: A fictional Britain

Official TT-Dave Fan Club
Dave's Screenshot Thread! - Flickr


Why be a song when you can be a symphony? r is a...


Top
   
PostPosted: Mon Apr 15, 2013 6:33 pm 
Offline
Tycoon
Tycoon

Joined: Sun Dec 16, 2007 10:24 pm
Posts: 1613
Location: Idaho, USA
Dave W wrote:
Congrats to all involved for making BaNaNaS even more appealing!

I see what you did there. :roll:

_________________
Eyecandy Road Vehicles | Fake Subways | Supercheese's NewObjects

"Fashions and cultures change, but steam trains shall always be majestic."
-Professor Hershel Layton


Top
   
PostPosted: Mon Apr 15, 2013 7:29 pm 
Offline
Moderator
Moderator
User avatar

Joined: Mon Dec 26, 2005 8:19 pm
Posts: 17021
Location: Harringay, North London
Supercheese wrote:
Dave W wrote:
Congrats to all involved for making BaNaNaS even more appealing!

I see what you did there. :roll:


Cor! I didn't! But now I have hahaha.

_________________
Albion: A fictional Britain

Official TT-Dave Fan Club
Dave's Screenshot Thread! - Flickr


Why be a song when you can be a symphony? r is a...


Top
   
PostPosted: Mon Apr 15, 2013 11:45 pm 
Offline
Tycoon
Tycoon
User avatar

Joined: Mon Sep 28, 2009 5:15 pm
Posts: 5683
Location: Eastern KY
TrueBrain never gets enough accolade for the bunch of work he puts in.

_________________
Do you like drones, quadcopters & flying toys? Check out Drone Strike Force!
Image

Base Music Sets: OpenMSX | Scott Joplin Anthology | Traditional Winter Holiday Music | Modern Motion Music
Other Projects: 2CC Trams | Sprite Sandbox & NewGRF Releases | Ideabox | Town Names | Isle of Sodor Scenario | Random Sprite Repository
Misc Topics: My Screenshots | Forgotten NewGRFs | Unfinished Graphics Sets | Stats Shack | RoadTypes?


Top
   
PostPosted: Thu Sep 12, 2013 3:41 pm 
Offline
Engineer
Engineer

Joined: Fri Oct 22, 2004 7:27 pm
Posts: 91
I'm trying to upload a big grf into BaNaNas using musa. However, after the program starts packacking the files, an error "No magic header" is generated and the program terminates. I was told this is because of invalid grf format, but I've no idea how to fix the problem. The grf was generated using the compiler at jenkins.openttdcoop.org with nmlc version 0.3.0.r2095. The part of grfid.py which generates the error:
Code:
def get_grfid(f):
   try:
      reader = GRFIDReader(f)
      grfcontversion = 1;

      # Check version
      if reader.buffer[0:len(grfv2header)] == grfv2header:
         grfcontversion = 2
         reader.skip_bytes(len(grfv2header) + 4 + 1)

      if reader.read_size(grfcontversion) != 0x04 or reader.read_byte() != 0xFF:
         raise MusaException("No magic header")

The problematic grf is available from here. Any help would be highly appreciated. I'm also at #openttd, if you wish to contact me there.

_________________
Projects I'm involved with


Top
   
PostPosted: Thu Sep 12, 2013 4:56 pm 
Offline
OpenTTD Developer
OpenTTD Developer

Joined: Thu Feb 09, 2006 7:15 pm
Posts: 3815
What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.


Top
   
PostPosted: Thu Sep 12, 2013 9:26 pm 
Offline
Engineer
Engineer

Joined: Fri Oct 22, 2004 7:27 pm
Posts: 91
Rubidium wrote:
What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.

Code:
python musa.py -c myconfigfile.ini -d -u juzza1 -p XXX mygrf.grf


So yes, I'm only trying to package the actual grf file.

_________________
Projects I'm involved with


Top
   
PostPosted: Thu Sep 12, 2013 9:49 pm 
Offline
OpenTTD Developer
OpenTTD Developer

Joined: Thu Feb 09, 2006 7:15 pm
Posts: 3815
That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.


Top
   
PostPosted: Thu Sep 12, 2013 9:59 pm 
Offline
Transport Coordinator
Transport Coordinator
User avatar

Joined: Mon Mar 24, 2008 1:04 pm
Posts: 258
Location: Finland
Rubidium wrote:
That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.


Could you try to upload that linked NewGRF to the BaNaNas?

_________________
Projects in progress: Finnish Trainset (DevZone)| Finnish Rail Infrastructure and Stations Set (DevZone)


Top
   
PostPosted: Thu Sep 12, 2013 10:38 pm 
Offline
Engineer
Engineer

Joined: Fri Oct 22, 2004 7:27 pm
Posts: 91
So I tried to build it on a VM Fedora. Curiously, I'm getting a different error, but the grf still won't package. Am I failing on two separate environments? I'm really at a loss here, I'm doing everything by the book but getting nowhere. Rubidium, could you post the exact Python version you are currently using - maybe I could try package with that.


Attachments:
File comment: What I'm working with
musa.png [358.67 KiB]
Downloaded 4 times

_________________
Projects I'm involved with
Top
   
PostPosted: Fri Sep 13, 2013 4:55 am 
Offline
OpenTTD Developer
OpenTTD Developer

Joined: Thu Feb 09, 2006 7:15 pm
Posts: 3815
Python -V says: Python 2.7.5+
It is actually the Debian package with version 2.7.5-7.

In any case, I find it really odd that python behaves so differently on different platforms. Running musa with -v does add a tiny bit of extra information, though that's more the intermediate steps.

I'm quite flabbergasted by this, but don't have the time to investigate this issue any time soon (look at the work I did for OpenTTD over the last two months), i.e. don't have the time to install Windows or another operating system and spend time trying to figure out what's going wrong there.


Top
   
PostPosted: Fri Sep 13, 2013 4:37 pm 
Offline
OpenTTD Developer
OpenTTD Developer

Joined: Wed Dec 20, 2006 1:31 pm
Posts: 972
Location: Aschaffenburg
Can you please try again with new musa r25762.

Maybe then you get at least the same error on both platforms :p

_________________
⢇⡸⢸⠢⡇⡇⢎⡁⢎⡱⢸⡱⢸⣭⠀⢸⢜⢸⢸⣀⢸⣀⢸⣭⢸⡱⠀⢰⠭⡆⣫⠰⣉⢸⢸⠀⢰⠭⡆⡯⡆⢹⠁⠀⢐⠰⡁


Top
   
PostPosted: Fri Sep 13, 2013 5:08 pm 
Offline
Engineer
Engineer

Joined: Fri Oct 22, 2004 7:27 pm
Posts: 91
After updating, I am indeed getting the "uniqueid is invalid" error on all environments. Just for the heck of it, I also installed Debian as a VM, getting the same error there aswell. Some details of python -v, if it helps:

Debian VM:
Python 2.7.3
GCC 4.7.2

Fedora VM:
Python 2.7.5
GCC 4.8.1

Windows 7 64-bit (main OS)
Python 2.7.5 (32-bit)

_________________
Projects I'm involved with


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 37 posts ]  Go to page 1 2 Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000-2017 phpBB Limited

Copyright © Owen Rudge/The Transport Tycoon Forums 2001-2017.
Hosted by Zernebok Hosting.