Transport Tycoon Forums

NOTE! This script is now obsolete and disabled

(If this is your first time reading this post, read through it before visiting the link.)

http://jupix.info/openttd/gfxdev-nightlies/

The pack currently available there has very few contained graphics, even though there are lots on the repository. Read on for why this is and what should be done to fix it.


A bit of background

If you recall, there was some talk in January about automating the "megapack" creation process, now that we have our content in a central place, which is the repository. It was suggested that I implement the tar bundling feature at the repository.

I (silently) acknowledged the idea as a good one, but there were some immediate concerns and things that prevented me from implementing it right away. I'll now throw in a quote from the discussion.

Jupix wrote: Ben keeps hitting the nail on the head in that the repo was never designed as a player resource - rather, it's an artist / developer resource with all the inherent negative aspects in regards to user friendliness, one of which is that files are, by definition, divided to small, easily manageable and modifiable packages which in an ideal situation contain only one "item", like a house.

[ ... ]

Now, the reason I went with this approach in the beginning is money, basically. As you may know the repo's already running on a 1mbps upline which translates to about 80 - 110 kB /s real download speed divided between all the concurrent users and users of my other websites. That's not a lot if I were to distribute megapacks that can reach 100's of megabytes in size. To get a bigger pipe to the server requires taking it into a server hotel which costs a fortune and as usual I'm not getting a penny out of doing this.

[ ... ]

Furthermore, automating the process of creating big packs at the repo takes considerable knowledge of how tars are handled using PHP and to be honest I don't have that knowledge, at least not yet. Also, it takes considerable server effort and we're running on a 600 MHz processor and less than 200 MB's of RAM. Compression is obviously right out of the question and I'm not sure how low specs like those would handle hundred megabyte uncompressed bundles.

In short, the issues were:
1) philosophical issue with modifying the repository for this
2) bandwidth
3) programming knowledge and
4) server computing capacity

Well, now that I've had some time to work these out, I've reached the following conclusions.

Let's start with #3. It appears PHP has a very easy to use, although a bit buggy, tar manipulation library that I have now learned to use. So in short, this is now a non-issue. Although #1 still stands, I have implemented the bundle automation feature at a separate site that's rather closely knit together with the repository. This solves the problem of adding bloat to the repository app.

Now, as I was developing the tar bundling app, it became clear that the server performance problem did not exist so much for tar creation and manipulation, than it did for basic file manipulation tasks. Both need to be taken care of, so performance is an issue, but currently, for creating nightly builds, my server is somewhat adequate. A bundle of about 50 megs takes about 20 minutes to compile.

The only concern left therefore was bandwidth, but fortunately I received an offer from Ammler for hosting at the openttdcoop server. Once I get that automated, there should be plenty of bandwidth for you guys.


What does it do?

It takes graphics from the repository and bundles them into what we like to call a megapack. More information about sprite inclusion can be obtained at the compiler page.


My sprites are not included in the standard pack?

Your sprites need to meet the requirements for inclusion.

- Include sources in your release and pack it into a standard tar. Then set its type to "Standard tar".
- Finish models, model all construction stages, etc. to complete your item and update the repository entry. Then set its status to "released".


Important guidelines

If you're an artist or administrator, read this. This is how the automation process is supposed to work:

1. Repository contains individual items, like a house, or grass ground sprite set, or paved roads, or whatever.
2. Nightly bundle script takes these individual items and bundles them into a pack.
3. Updates to graphics are made at the repository, making changes to the individual packs.
4. Nightly bundle script takes the individual tars again, thereby distributing the fixes.
5. Repeat 3-4 indefinitely.

In order for the system to work, we need #1 to be the actual case. This means, there is work to be done for administrators (and artists who have bundled their work into multiple item packs). Divide individual works into individual entries using common sense (individual ground sprites can be bundled at the repository, but individual vehicles should not be, etc. If you're unsure, ask me.)

Some updates went live today; compiler now takes standard tars only, and there is a dev package with less strict requirements for inclusion.

The standard nightly pack is unchanged apart from the standardisation improvement - content flows in as repo entries are converted into standard tars and released with all zoom levels.

Great initiative Jupix, did not really have time to respond earlier, though I did follow the thread.

Maybe we can find a way to have your tars and the 32bpp-extra tar/zip in one download location? Like here: http://bundles.openttdcoop.org/32bpp-extra/nightlies/
That would both be easy for the end users and relieve your servers bandwidth a bit.

A content remark:
http://jupix.info/openttd/gfxdev-repo/i ... file&id=58 can be blacklisted too I guess, since it is in the 32bpp-extra project, and won't work for 'normal' tars without newgrf.

A remark about the 'standard tar': openttdw/d would link to opengfxe_extra or v.v. But that will never work in all situations, because the numbering is different, i.e. the main reason for the existence of the 32bpp-extra project. So every occurrance of a sprite in openttdw, openttdd or ogfxe_extra should be split off in your repo. And these dirs/links can be removed from the standard tar. I could make a new tar, but you also could try the symlink command in php, or exec "ln -s" if you're on a Linux server

I tried to arrange hosting at the openttdcoop server with Ammler, but nothing came of that. It's now been a month without anything happening.

Thanks for the heads up on the catenary package. I try to tag every extras release at the repo (as NewGRF) so that the autocompiler doesn't even consider including them.

Outside that, when the autocompiler runs, it will automatically ignore all extras anyway. I am directing everyone who wants those sprites to the extras project page.

Thanks for the explanation concerning the tar layout. Now I understand the dir structure that much more. I would really appreciate an updated template from you actually.

Here's an updated template tar:
changelog:
-removed openttdd, openttdw and ofgxe_extra
-added sources dir

To be honest, I have not yet tested it on windows, I created it on Linux, and testing would require a reboot, but I'm sure someone will report here if it wouldn't work.

Couple of improvements:

1. There was an issue which resulted in extraneous png files being included when they weren't supposed to; this has now been fixed in the standard autopack but still unresolved in the dev version

2. Packs are now automatically pushed onto the openttdcoop bundle server which shall be the primary download mirror from now on, as it offers download speeds at least an order of magnitude faster than my own server

3. For backwards compatibility, all xxxx_z2.png and mask equivalents are automatically renamed to xxxx.png and xxxxm.png. This enables the z2 sprites to work whether or not the user has the EZ patch installed.

Latest megapacks did not get pushed to the openttdcoop server, it seems, so the link on the Jupix megapack page is dead. Don't know if that is a structural problem, I just want to let you know.

Yeah, I know, scp isn't working via cron. Don't know why. Haven't had the chance to work around it. In the mean time the "old" jupix.info mirror works fine.

cron gets a very limited environment, so maybe the path to scp isnt available. You can source your profile files or bashrc etc before, or only use full absolute paths in the scripts called by cron.
You also need to start scp with -B for batch mode processing, as no userinteraction can take place.

Very nice that it now works to get easily also a 32bpp pack which works with trunk. It actually made me to just give it another look and it's quite nice

I wonder: is there any license attached to the sprites? I found some sprites where I'm quite well considering to convert them to 8bpp and integrate them into OpenGFX, namely ore mine and light house, maybe some field tiles.

Concerning cron: it doesn't get your path. You need to explicitly state every path fully and not assume any.

The vast majority of released sprites are under GPLv2 - I think that the lighthouse is the only exception.

The licence should be clear on the repository for the files available there.
All files included in the main bundle should be gpl v2, but I can't vouch for that, as I haven't filtered them myself.
The field sprites are defiantly gpl v2 or later.

Perhaps GPLv2 should be a requirement for entry into the automated bundle?

Thanks Wasila and Ben, I'll look there.

Personally I'd prefer, if the whole set would use one license - and there preferably also GPL v2(+) which is the same license as OpenTTD uses; it'd make it possible to distribute it jointly without 2nd thought.

We have set the standard as GPLv2, and all our major artists which are still around have agreed to it. Unfortunately, some of them are inactive and so we are stuck with things like CC-BY-SA. Jupix just hasn't yet made GPLv2 a requirement to be included into the automated build, but as I said I think all of them are except for the light house.

Guys, please read bullet 3 of the intro section of the pack distribution website. The autopacks of this thread are as official as the manually created megapacks of the past. In other words, not official at all. These packs are intended to replace those packs, not implement a new 32bit base set. Therefore there are and will not be any licensing requirements. Eventually there will be a base set replacement project which may or may not use the compiler used for these packs, and for that project, there will be a strict licensing and content quality policy.

To check the licensing of an individual part of the autopacks, use the links found in the pack distribution website's content listings.

So what are we waiting for for an official base set replacement project?

That discussion is not for this thread.

Some nice stuff in there !

May I point you to CoBlitz for a config-less additional mirror ?

All it takes is to add a pointer to when you create your pointer to First user to hit the link got the same speed as if he went straight to your site, but following downloads comes from the free CDN.
At least, worth a try.

Thanks for the tip, very interesting site. I wonder who pays for the bandwidth. Added the link, let's see how it works. (Let me know.)

Haven't had time to delve deeper into how to solve the scp file copy problem, the problem is that it wants to always authenticate using the key-passphrase. Adding the credentials to the ssh agent obviously lets me upload till the agent lets go of the credentials again, but as that seems to happen before the cronjob gets even one copy out, I need a better solution. It seems like a commonly used method to attach the ssh agent to the shell during bootup, but I don't really want to reboot my server for this.