32bit Graphics Automated Nightly Bundle

[GeekToo]

Bug report (is this the right place?) for:
32bit-gfx-nightly-megapack-2010-08-15-dev.tar

dbg: [misc] Warning: PNG Sprite '/ogfx1_base/4240.png' doesn't have x_offs and y_offs; expect graphical problems
dbg: [misc] Warning: PNG Sprite '/ogfx1_base/1694.png' doesn't have x_offs and y_offs; expect graphical problems

This can cause glitches when zooming in on the grain fields, or some trees

[Jupix]

Well, the more proper place would be somewhere more associated with those particular releases, but it's good that the bugs are documented somewhere.

I need to write a "sprite search" feature into the repo so particular sprites can be easily traced back to their respective repo entries...

[Dwight_K._Schrute]

Hey ,

I downloaded the last standard nightly pack (2010-10-02) and most things are fine. But the manager faces are still in 8bpp and the two trains are looking kind of strange. This happens with a clean nightly (r20834), too. So I don't think the patch is related with this.

[Zephyris]

The graphics in teh 32bpp pack are not designed for use with the nightly, they are supposed to be used with the extra zoom lervels patch. Using that should solve both your graphics problems.

[Dwight_K._Schrute]

Oh ok. I've thought this pack works with both trunk and extra zoom levels. So I've apparently missunderstood something...

[Zephyris]

Lots of the graphics will work with the latest nightlys, company colours and extra zoom level graphics will not.

[GeekToo]

There is something funny in the nightly (dev) pack. Some sprites are in there multiple times, but with different content. Maybe that explains the problems with the group of town houses. The file itself seems to be coded ok http://jupix.info/openttd/gfxdev-repo/i ... ile&id=260, but it looks like the sprites in the source dir are added to the ogfx1_base dir, maybe by the nightly script

[maquinista]

There is something funny in the nightly (dev) pack. Some sprites are in there multiple times, but with different content. Maybe that explains the problems with the group of town houses. The file itself seems to be coded ok http://jupix.info/openttd/gfxdev-repo/i ... ile&id=260, but it looks like the sprites in the source dir are added to the ogfx1_base dir, maybe by the nightly script

Yes, the sprites of the small file are the latest and finished sprites, and are the sprites that should be used.

[Jupix]

Here's Eric's interpretation of the problem, I think he's spot on.

It looks like the sprites from the source dir also are put into the ogfx1_base dir in the resulting tar.

Here's the code that's probably the cause of the problem. Note the comment at the end, which I think foresees this exact issue

Code: Select all

// In a normal situation we'll already have the
// OpenGFX directory structure in the source tar
if (strpos($rel['filename'], "ogfx")) $set = $spritename_chunks[count($spritename_chunks)-2];
// But in case the tar was structured to work with
// TT data files, we'll convert it to the OpenGFX format
elseif (strpos($rel['filename'], "trg")) {
	switch ($spritename_chunks[count($spritename_chunks)-2]) {
		case "trg1r":
		case "trg1":
			$set = "ogfx1_base";
			break;
			
		case "trgcr":
		case "trgc":
			$set = "ogfxc_arctic";
			break;
			
		case "trghr":
		case "trgh":
			$set = "ogfxh_tropical";
			break;
			
		case "trgtr":
		case "trgt":
			$set = "ogfxt_toyland";
			break;
			
		case "trgir":
		case "trgi":
			$set = "ogfxi_logos";
			break;
	}
}
// Because the devpackager may touch incorrectly structured tars,
// we need to just take a leap of faith and put into the base
// dir everything that hasn't got a proper dirname
else $set = "ogfx1_base";

I added a check right next to where I check for the content being PNG's, and not being extras, for it not being sources... Let's see if the problem is fixed in tomorrow's build.

[Jupix]

There should be a new build now, can anyone confirm whether the issue still exists or is fixed?

[Lord Aro]

Jupix,

can you fix the mirrors, especially the OTTDcoop one?
As you say yourself, your server isn't the fastest...

Can't check the pack right now, as this computer doesn't have unpacking software (or admin rights to install it)

[GeekToo]

Thanks for your effort Jupix, but the issue is not yet solved. Eg 1501 and 1502 are still double present in the tar.

[Jupix]

OK, need to take another look.

As for the ottdcoop mirror, I'm at a loss as to how to fix it. If it was easy or just a matter of RTFM, I'd have done it already.

[Jupix]

So the issue with the duplicate sprites & sources ending up in the base directory was caused by the very code I looked at earlier, only the fix I put in was bugged. (Didn't remember strpos returns 0 when the needle is at the beginning of the haystack.)

I reran today's pack and looked at it, I don't see any duplicate sprites or source graphics there now.

[GeekToo]

Just tested it, works fine for me. Thanks

[Jupix]

I spent about 9 hours today on and off trying to debug the cronjob. I failed to get it running properly and are about to turn in for the night. Sit rep follows:

• I got the cronjob to run almost immediately, but with only 2 tars out of 32 getting through to the pack
• The 2 "successful" packages are identical in layout, permissions and ownership to the ones that fail
• Moved the script from cron.daily to cron.hourly to debug it
• Chowned and chmodded a bunch of stuff that had been creating with various owners and masks after the server move of last summer
• Removed steps in the packager that create and remove a temp directory for the packaging process, and changed it to use a permanent temp place with correct ownership and permissions
• Learned that different versions of PHP handle permissions for uploaded files differently, so implemented forcing all that to a specific setting both at the repo and at the packager
•  Narrowed the issue at the packager down to a single line:

  Code: Select all

  exec("/bin/tar --append --file=/var/www/html/openttd/gfxdev-nightlies/files/$filename --transform='s,.*/,sprites/$set/,' $tempdir/$spritename");

• The variables are all setting correctly
• The command gets executed successfully
• Two tars get packaged with that line without a hitch, but 30 tars silently fail somehow
• A line of code exactly like that functions perfecly above, successfully appending readme.txt into the package
• While the script runs, all sprites get extracted into the temp dir successfully and are equal in ownership and permissions
• The 30 tars are not accidentally caught in any filter, since they get extracted, and all the extracting gets done within the same if-wrappers
• When run in the command line as a regular user, the script functions correctly, packaging all 32 tars
• When run in the command line as root, the script functions correctly, packaging all 32 tars
• When run as a cron.daily or cron.hourly entry owned by a regular user, the script runs, but only packages 2 tars
• When run as an @hourly root crontab row, the script runs, but only packages 2 tars

I've pretty much exhausted all my options. But tomorrow's a new day.

[planetmaker]

In cron none of the usual environment variables available in a shell are set. Give all paths and stuff explicitly and fully expanded.

[Jupix]

Yep, all paths in that script are full and explicit.

[Lord Aro]

My inexperienced head is telling me that the problem has now becom a case of: we can't help you without seeing your code (script + crontab)

If you're willing to post them, that is

[Jupix]

Well, already in the morning I concluded that it's the most time-efficient to just re-engineer the whole thing. After all, it's a pretty short and sweet script.

I just got it to run in cron.hourly so I'm pretty optimistic it'll run right in cron.daily, tomorrow morning at 4:00 am.

For the hell of it, though, here's the code of the old one: http://jupix.info/openttd/gfxdev-nightl ... kager.phps
I based the new one on that, but it handles the tar creation part differently, and I think in a much smarter way.

I also changed the front page a bit, so that
- it is now easier to compare # of tars and exact file size between today's and previous packs
- black list isn't taking up space at the top of the page
- the order of content makes more sense

Let's hope the packager runs tomorrow, so that I can start concentrating on other things than the broken compiler.