Moderator: OpenTTD Developers
So, to solve this issue once and for all, we designed a simple Python application. As extra bonus, it validates your package on your client, so you don't have to upload 300 MiB files to only find out after an hour something is broken in your .tar. When the client validates your .tar, the server most likely will too. Win-win.
Who (is the audience)?
Any person who uploads to BaNaNaS can (optionally) use this, but people who want to upload larger files (30+ MiB) need to use it. It also allows a bit more configuration.
musa (and depgen) is available as a windows executable from http://bundles.openttdcoop.org/musa/push/LATEST/
If execution fails due to missing MSVCR libraries or due to an invalid application configuration or similar error, you need to install http://www.microsoft.com/en-us/download ... aspx?id=29
If you are on OSX or linux, checkout http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or svn://svn.openttd.org/extra/musa (svn), and install Python 2.6+ yourself, you can use it! (it possible works with Python 2.5, but no guarantees). See example.ini how you need to configure it.
Sorry, I am fresh out. The --help of musa.py is very clear I think, and example.ini has many comments. If you can figure out NML, you can figure out this too.
Of course! We - are - Open - Source! http://hg.openttd.org/openttd/extra/musa.hg (mercurial) or http://vcs.openttd.org/svn/browser/extra/musa (svn)
When we added (good) 32bpp support to OpenTTD a few months back, we knew people would start making 32bpp grfs, and that these files would be big. Very soon after, it became clear our current methods weren't up for the task of uploading but also distributing large files. A few months back the first questions started to be asked about it. 3 months ago we talked over solutions, and considered a stand-alone tool with its own protocol best. A few days later Rubidium wrote an implementation, named musad. This weekend I rewrote the balancer to handle this kind of traffic, cleaned up our infrastructure in relation to this, moved around files, and managed to get a version online. After several tests, we finally managed to upload zBase (270+ MiB) to BaNaNaS, and distribution worked. After patching up ottd_content to handle files this large, we now feel confident our whole system from user to backend back to the user can handle files this size. So ... we are ready for your 32bpp grfs! Your turn!
None. We think this is finished. Of course we will work on installer packages for Musa client, but generally speaking, this system is done. Up to the next
Rubidium, for writing musa(d).
Rubidium, for fixing ottd_content in record time.
Myself, for programming 20 hours in this weekend fixing up the mirror system, the balancer, the infrastructure, getting musad to work, ..
Yexo, Rubidium and others for testing.
Who needs a hug?
(thumb pointing to me) *this guy* (thumb pointing to me)
Here comes a bearhug from Germany for taking up another big project once again for the OTTD community. But you are not the only one getting a bearhug, Rubidium of course also gets one and also every other dev, mod or NewGRF provider for making OTTD so awesome!
There are many simple yet important tasks like translations, investigating facts and details and many other that need work and if you help a bit then the experienced developers can focus more on making OTTD even more awesome!
OTTD = Awesomeness created by a whole lot of volunteers, be one of them if you like the game!
Brevity is the soul of wit and obscenity is its downfall
Good Night And Good Luck - Read You Soon
AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios
First thing to mention is that musa now supports uploading a whole set of new content types. (AIs, AI libraries, GSs, GS libraries, heightmaps and scenarios)
During my work, I found and fixed problems related to validation of dependencies. It should now be possible to upload content and have their dependencies set. All you need is their md5sum, uniqueid and type. If you are like me, you think that is not "just", so that is why I now will introduce you to depgen.py:
This utility script helps you to determine the dependency string for content that you want to declare as dependencies for the new content that you are about to upload. As you might know, in musa you need to declare dependencies on this format:
Code: Select all
<type of content>:<unique ID>:<md5 sum>
Code: Select all
> ./depgen.py ../content_download/ai/CluelessPlus-34.tar # CluelessPlus-34.tar dependencies = AI:CLUP:7cc9cae1cc5892435aa8f3aa7454fd06
Code: Select all
> ./depgen.py ../content_download/newgrf/OpenGFX_Trains-0.3.0.tar # OpenGFX_Trains-0.3.0.tar dependencies = NewGRF:4F472B31:4e0b2d42462f295f36afba90d239a323
Where do I find depgen.py?
When you check out musa (or clone it using hg or git), you also get depgen.py next to musa.py. If you got the initial version of musa, you will need to update it to get depgen.py.
What is worth to know is that the Beginner Tutorial is itself not just a single bananas entry. Instead it is composed of three bananas entries that are contained in one hg repository.
Before continue reading this example, I suggest you make yourself familiar with example.ini which is included when you check out/clone musa. This example shows how you can auto-generate musa ini files.
In Beginner Tutorial, I have musa_desc_<content>.txt for each content that is part of the tutorial. These files contain the text description of each item. This text is fairly static.
Then for each content, I generate a musa_<content>.ini file using this python script: make_musa.py. In make_musa.py I have hard-coded dependency strings for dependencies outside of the Beginner Tutorial project. These dependencies are already on bananas when I call make_musa.py. What this script do however is to also to predict the dependency string that content to be uploaded will get. This is used to make the scenario depend on the GS and AI.
make_musa.py do however never call musa. For that I have a separate shell script, upload.sh which calls musa. This script uses hard-coded paths to musa which you will have to update to work in your situation. Note that this script is quite stupid and do not verify that uploading of previous entry succeeded. So I need to keep an eye on that. However the script still make sure I always specify the same files to include in the bananas package.
What may not be optimal in this solution is that it still depend on my old make_tar.py scripts which tar the GS and the AI. These tars are then analyzed to get produce the md5sum of these scritpts. A cleaner solution could be that make_musa.py iterate the files directly on the file system as upload.sh will not use the tars produced by make_tar.py but rather let musa make its own tars.
I see what you did there.Dave W wrote:Congrats to all involved for making BaNaNaS even more appealing!
Cor! I didn't! But now I have hahaha.Supercheese wrote:I see what you did there.Dave W wrote:Congrats to all involved for making BaNaNaS even more appealing!
Base Music Sets: OpenMSX | Scott Joplin Anthology | Traditional Winter Holiday Music | Modern Motion Music
Other Projects: 2CC Trams | Modern Waypoints | Sprite Sandbox & NewGRF Releases | Ideabox | Town Names | Isle of Sodor Scenario | Random Sprite Repository
Misc Topics: My Screenshots | Forgotten NewGRFs | Unfinished Graphics Sets | Stats Shack | RoadTypes?
Code: Select all
def get_grfid(f): try: reader = GRFIDReader(f) grfcontversion = 1; # Check version if reader.buffer[0:len(grfv2header)] == grfv2header: grfcontversion = 2 reader.skip_bytes(len(grfv2header) + 4 + 1) if reader.read_size(grfcontversion) != 0x04 or reader.read_byte() != 0xFF: raise MusaException("No magic header")
Rubidium wrote:What is the exact command line you are passing? Are you trying to package the .zip or .tar file? Either of them won't work; you need to package the actual GRF.
Code: Select all
python musa.py -c myconfigfile.ini -d -u juzza1 -p XXX mygrf.grf
Could you try to upload that linked NewGRF to the BaNaNas?Rubidium wrote:That command line packages the grf just fine for me. Beyond that I have no idea what could cause this.
- What I'm working with
- (358.67 KiB) Downloaded 4 times
It is actually the Debian package with version 2.7.5-7.
In any case, I find it really odd that python behaves so differently on different platforms. Running musa with -v does add a tiny bit of extra information, though that's more the intermediate steps.
I'm quite flabbergasted by this, but don't have the time to investigate this issue any time soon (look at the work I did for OpenTTD over the last two months), i.e. don't have the time to install Windows or another operating system and spend time trying to figure out what's going wrong there.
Maybe then you get at least the same error on both platforms
Windows 7 64-bit (main OS)
Python 2.7.5 (32-bit)
Users browsing this forum: No registered users and 7 guests