.png sprite loading removed from OpenTTD

[Vector]

When 32bit newgrfs were introduced, support for our current sprite distribution method was removed. That means everything we've produced no longer works in versions that are coming out. This thread is for discussing that. Original post continues...

So, thanks for the fixes. I've tried the dev pack, and, sure enough, it's got some empty images, too. Here's the list (may not be comprehensive, but I think it is):

Code: Select all

ogfx1_base/1370.png
ogfx1_base/1371.png
ogfx1_base/1372.png
ogfx1_base/1373.png
ogfx1_base/1374.png
ogfx1_base/1375.png
ogfx1_base/1376.png
ogfx1_base/1377.png
ogfx1_base/1688.png
ogfx1_base/1689.png
ogfx1_base/1690.png
ogfx1_base/1691.png
ogfx1_base/1692.png
ogfx1_base/1693.png
ogfx1_base/1694.png
ogfx1_base/2727.png
ogfx1_base/2728.png
ogfx1_base/2729.png
ogfx1_base/2730.png
ogfx1_base/2731.png
ogfx1_base/2732.png
ogfx1_base/2929.png
ogfx1_base/2930.png
ogfx1_base/2931.png
ogfx1_base/2932.png
ogfxc_arctic/203.png
ogfxc_arctic/204.png
ogfxc_arctic/205.png
ogfxc_arctic/206.png
ogfxc_arctic/207.png
ogfxc_arctic/208.png
ogfxc_arctic/209.png
ogfxc_arctic/210.png
ogfxc_arctic/322.png
ogfxc_arctic/323.png
ogfxc_arctic/324.png
ogfxc_arctic/325.png
ogfxc_arctic/326.png
ogfxh_tropical/38.png
ogfxh_tropical/39.png
ogfxh_tropical/40.png
ogfxh_tropical/41.png
ogfxh_tropical/42.png
ogfxh_tropical/43.png
ogfxh_tropical/44.png
ogfxh_tropical/45.png
ogfxh_tropical/46.png
ogfxh_tropical/47.png
ogfxh_tropical/48.png
ogfxh_tropical/49.png
ogfxh_tropical/50.png
ogfxh_tropical/51.png
ogfxh_tropical/52.png
ogfxh_tropical/53.png
ogfxh_tropical/54.png
ogfxh_tropical/55.png
ogfxh_tropical/56.png
ogfxh_tropical/61.png
ogfxh_tropical/62.png
ogfxh_tropical/63.png
ogfxh_tropical/64.png
ogfxh_tropical/65.png
ogfxh_tropical/66.png
ogfxh_tropical/67.png
ogfxh_tropical/68.png
ogfxh_tropical/69.png
ogfxh_tropical/70.png
ogfxh_tropical/71.png
ogfxh_tropical/72.png
ogfxh_tropical/73.png
ogfxh_tropical/74.png
ogfxh_tropical/75.png
ogfxh_tropical/76.png
ogfxh_tropical/77.png
ogfxh_tropical/78.png
ogfxh_tropical/79.png
ogfxh_tropical/80.png
ogfxh_tropical/81.png
ogfxh_tropical/82.png
ogfxh_tropical/83.png
ogfxh_tropical/84.png
ogfxh_tropical/85.png
ogfxh_tropical/86.png
ogfxh_tropical/87.png
ogfxh_tropical/88.png
ogfxh_tropical/89.png
ogfxh_tropical/90.png
ogfxh_tropical/91.png
ogfxh_tropical/92.png
ogfxh_tropical/93.png
ogfxh_tropical/94.png
ogfxh_tropical/95.png
ogfxh_tropical/96.png
ogfxh_tropical/97.png
ogfxh_tropical/98.png
ogfxh_tropical/99.png
ogfxh_tropical/100.png
ogfxh_tropical/101.png
ogfxh_tropical/102.png
ogfxh_tropical/191.png
ogfxh_tropical/192.png
ogfxh_tropical/193.png
ogfxh_tropical/194.png
ogfxh_tropical/195.png
ogfxh_tropical/196.png
ogfxh_tropical/197.png
ogfxh_tropical/198.png
ogfxh_tropical/369.png
ogfxh_tropical/370.png
ogfxh_tropical/371.png
ogfxh_tropical/372.png
ogfxh_tropical/373.png
ogfxh_tropical/374.png
ogfxh_tropical/375.png
ogfxh_tropical/376.png
ogfxh_tropical/377.png
ogfxh_tropical/378.png
ogfxh_tropical/379.png
ogfxh_tropical/380.png
ogfxh_tropical/381.png
ogfxh_tropical/382.png
ogfxh_tropical/383.png
ogfxh_tropical/384.png
ogfxh_tropical/385.png
ogfxh_tropical/386.png
ogfxh_tropical/387.png
ogfxh_tropical/531.png
ogfxh_tropical/532.png
ogfxh_tropical/533.png
ogfxh_tropical/534.png
ogfxh_tropical/557.png

HTH

[GeekToo]

The list is nice, but can be extended to every sprite we have in the repo. Recent development has removed the possibility to load png files, in favour of graphics data in grf file format. So for trunk:

• the nightly bundle is useless

• all our pngcodeced pngs are useless

• 32bpp-extra newgrf is useless

• wiki page overviews of 32bpp tars are useless

.

• pngcodec and scripts for it are useless

.

• standard tar template is useless

.

Commit log:
(svn r23898) -Remove: PNG sprite loader.2 weeks ago, by michi_cc

Just tried it, and indeed, not one 32bpp sprite visible anymore.
So we're back at an empty list. Anyone feel like creating a couple of thousand lines in nfo format, so the pngs can be grfcodeced?
It's the second time a trick like this is performed, before the introduction of 32bpp in trunk there was a 32bpp branch, that used nfo and pngs in a tar. Then, in trunk pngcodec was introduced, making all the previous tars useless.

This may need some discussion in the organisation thread, how to continue graphics packs. And also how it is possible that one commit makes literally thousands of hours of work useless.

[planetmaker]

I understand your anger in some way. And we had quite some discussions about the zoom level topic, basically over the last 3 months at least. The decision as it is now was made for technical reasons only after looking at all the options, including keeping the way the 32bpp-EZ patch treated things:
* The current OpenTTD implementation now only one mechanism for all colour depths and zoom levels for loading.
* He bananas support for all sprite types came for free with this solution.
* The old solution to put all sprites into a tar next to the grf proved quite badly compared to this when looking at performance.
* Writing (New)GRFs now also only uses one method to add sprites of every allowed colour depths and zoom level, treating all new combinations equally; thus NewGRF authors don't have to learn another one, two or three tools to add additional sprites.
* The format is extensible to either other bit depths or further zoom levels without changing anything in the format and does not depend on file names

And for invitation and reference: Now it's IMHO a good time to start and extend OpenGFX by 32bpp sprites and sprites for other zoom levels. More so as every grf always needs the 8bpp 1x zoom sprites unconditionally. Thus currently my idea and plan is, as far as my time allows, adding slowly both, sprites for different zoom levels as well as 32bpp. NML will support these soonish.

Graphicswise I'd like to focus on those 32bpp which come with a blender model (where applicable. Doesn't make sense for menu). This will hopefully allow to generate the sprites for all required zoom levels via a nice blender script (as far as I looked, blender can be scripted to that extend - but I've not undertaken any detailed effort to do so, so far). Going for this scripted generation approach makes it a bit easier to adjust sprites as it only needs doing so in one file.

At the same time as implementing this, I'd like to see taken care that the 8bpp and 32bpp version don't diverge too drastically (i.e. rough shape and dimensions should agree). Thus the 32bpp should resemble the 8bpp equivalent - or vice versa. Which way around is chosen IMHO depends on the model and sprites in question. Though of course I see no way to automatize the process of 32bpp->8bpp conversion when done this way around. I shamelessly boiled down Jupix 32bpp whitepaper to what I personally consider the important essence. It's found under my user wiki page.

Cheers,
pm

[Vector]

Well, the list was primarily for Lord Aro and anyone else working with tars to fix their broken ones, which I still think they should do. But the question now is, how would one go about getting the new graphics without the click-fest that the megapacks were designed to eliminate.

[Leanden]

I expect there will eventually be a recoding of the openttd.grf file to automatically pull 32bpp sprites.

Does this mean that openttd now supports seperate sprites for z0 and z1, even if you can't code them with NML currently. Might be worth me redrawing the sprites im currently working on to 4x their current size to allow for this.

[planetmaker]

I expect there will eventually be a recoding of the openttd.grf file to automatically pull 32bpp sprites.

Does this mean that openttd now supports seperate sprites for z0 and z1, even if you can't code them with NML currently. Might be worth me redrawing the sprites im currently working on to 4x their current size to allow for this.

It means you can supply 32bpp sprites for 4x, 2x, 1x, 0.5x, 0.25x and 0.125x zoom levels. And you can supply 8bpp sprites for 4x, 2x, 1x, 0.5x, 0.25x and 0.125x zoom levels.

[Alltaken]

Out of curiosity, How does this change the process of creating and testing graphics quickly?

Previously it was possible to chuck a few png images over the top of existing sprites in a tar and see graphics ingame instantly, is this process still relatively quick?

[Rubidium]

is this process still relatively quick?

Creating a small NewGRF with those graphics should not be significantly more work. It mostly requires setting up the boilerplate once and then it's simply updating offsets/sizes/filenames and recompiling it.

The new format also makes it significantly easier to add 32bpp to a NewGRF as you do not need to rename many if not all filenames each time a sprite is inserted at some point.

Finally I'm not seeing why thousands of hours of work became useless. It should be relatively easy to create a script that constructs most of the nfo (or nml) you need to compile the GRF by means of "file" (to get the dimension), "pngcodec l" (to get the offsets), sed/awk (to reorder the output so it's right for grfcodec) and bash (to tie it all together).

For example the following creates most of the nfo for the real sprites:

Code: Select all

for i in `ls *.png|sort -n`; do FOO=`(file $i | cut -d' ' -f 5,7|sed s/,//; pngcodec l $i | tr ' ' '\n'|grep offs|sed 's/._offs=//;s/^$/0/')|tr 'n' ' '`;printf '\t| %12s %5s %4s %4s %4s %4s %6s\n' $i `echo $i | sed "s/.*_//;s/.png//;s/.*m/mask/;s/z1/32bpp/;s/z0/32bpp/;s/^..[^bs].*$/32bpp/"` $FOO `echo $i | sed "s/.*_//;s/.png//;s/.*m//;s/z0/zi2/;s/z1/zi4/;s/^[^z][^i].*$/normal/"`; done

[Edd.Dragon]

Hello!

As I understand from this topic, now I can use 32bit-gfx-nightly-megapack-2012-02-19.tar with latest ottd builds to get "extraquality"? Or I been mistaken?

openttd.cfg:
blitter = "32bpp-optimized"
fullscreen_bpp = 32

Tar is placed in Data (which no longer used? or not?) and in Baseset folders. But nothing happens

[planetmaker]

As I understand from this topic, now I can use 32bit-gfx-nightly-megapack-2012-02-19.tar with latest ottd builds to get "extraquality"? Or I been mistaken?

You're mistaken. The file format for sprites changed.

[Jupix]

I understand your anger in some way. And we had quite some discussions about the zoom level topic, basically over the last 3 months at least. The decision as it is now was made for technical reasons only after looking at all the options, including keeping the way the 32bpp-EZ patch treated things:
* The current OpenTTD implementation now only one mechanism for all colour depths and zoom levels for loading.
* He bananas support for all sprite types came for free with this solution.
* The old solution to put all sprites into a tar next to the grf proved quite badly compared to this when looking at performance.
* Writing (New)GRFs now also only uses one method to add sprites of every allowed colour depths and zoom level, treating all new combinations equally; thus NewGRF authors don't have to learn another one, two or three tools to add additional sprites.
* The format is extensible to either other bit depths or further zoom levels without changing anything in the format and does not depend on file names

The reason this causes anger is that there was no grace period before everything we worked on for 3 years went down the toilet in a single devastating splash.

I don't think the devs ever announced on this forum, the official communication venue for 32bits, that support for our content was going away on so-and-so date. I was told on IRC png loading wouldn't last forever, but I never knew the change was right around the corner.

My initial thought was to start working towards converting everything to .grf's once a process for that was figured out. Obviously you devs have different ideas and are of course free to do whatever you want. What makes me sad is that I wasn't consulted on any of it, and actually, have been left completely out of the loop.


To add insult to injury, last week, this week and the one after next, I'm more busy than usual, due to exams @ uni. I have been reading and answering PM's but was planning on touching on this once they were out of the picture. That does seem like a too long schedule now but there's not much I can do about it, RL is more important.

[FooBar]

Don't take this too personal. Yes, an ETA on the removal of png loading would have been better. But let's see what happened.

The 32bpp replacement set is far from complete. The sprites are still all there, but cannot be loaded for a short period of time. Maybe a couple of weeks. There's still more than a month left until the release of 1.2.0, when the mainstream users will switch. Mainstream users are still able to use the existing pack with 1.1.5 and even 1.2.0-betax.

So what did we lose? Mostly the people involved in the development of this cannot load their work in a recent game version. That doesn't mean nothing can be drawn or modeled in the meantime.

What is there to gain? The possibility to put a release on Bananas so that it is available for easy installation for everyone switching to 1.2.0 in over a month.

In the defence of the OpenTTD devs: I think this change came very fast for them as well. 32bpp grfs came far in the release of the 1.2.0-betas. In order for it to be in the final release, the png loading had to be removed before 1.2.0-RC1, as this is where 1.2.0 branches off.

So what should you do? Take some weeks off of 32bpp development. Focus on your exams, these are far more important right now.
When those are over, we can expect full support in NML, and with a bit of luck someone will figure out a script that takes the PNG files, reads the offsets and generates the required NML. From there it's a short walk to generating new nightly bundles and uploading the 32bpp graphics to bananas.

[Michi_cc]

and with a bit of luck someone will figure out a script that takes the PNG files, reads the offsets and generates the required NML.

Substitute NML with NFO and read five posts above. And even changing the print statement therein to produce NML syntax doesn't look like magic to me.

-- Michael Lutz

[Alltaken]

Somehow it doesn't bother me (but then again i've only just turned up after a few years away)

However, I see that Jupix tool might now have gone to waste. This would be a huge shame.

I really like the idea that PNG's and tars and other such simple mechanisms for distribution are used. The reason being the end-use format of PNG's is effectively identical to the development and source format. i.e. Blender spits out PNG's you can see and edit PNG's in any software etc... Suddenly the introduction of a GRF means there is a layer between PNGs and the game. Which with the correct advantages and tools would not be a problem.

If Jupix system can create GRF's while graphics developers can still upload PNGs and Blend Files, then i'd be quite happy, and we'd then have a single community wide conduit between "creation and consumption"

What is needed to make this happen, or to support Jupix to make this happen? Can this happen?

What i would almost prefer more is a web-based graphics editing system whereby you drag your image onto the website, asign the offsets, settings etc... and it is done on the website and distributed once it is flagged as complete. There is no reason that the graphics could not be previewed on the site alongside other in-game graphics, rather than ever needing to load them ingame.

Cheers,

[FooBar]

The best solution IMO would be to at some point expand the repository website with fields to store the offsets. For existing sprites these can probably be imported from the png files.

Then a script can be written that generates the required NFO/NML directly from the repository database.

I realize that this makes aligning the sprites a bit more difficult (as there's no easy way for an artist to test ingame), but this could become a two-step process. Step one is adding the sprite to the repository with some guesstimated offsets. Step two is waiting for the nightly compile. Step three is revisiting those sprites using the nightly ingame and using the sprite aligner to find the correct offsets, which then can be entered in the website.
A flag to mark offsets as correct on the website would be useful to see what's done and what's not.

[Alltaken]

I realize that this makes aligning the sprites a bit more difficult (as there's no easy way for an artist to test ingame), but this could become a two-step process. Step one is adding the sprite to the repository with some guesstimated offsets. Step two is waiting for the nightly compile. Step three is revisiting those sprites using the nightly ingame and using the sprite aligner to find the correct offsets, which then can be entered in the website.
A flag to mark offsets as correct on the website would be useful to see what's done and what's not.

Time allowing, i'd be happy to help Jupix or someone else create the interface on a website to be able to move a sprite left/right up/down to align it as though it were ingame.

It would need some basic parts:

• An Uploaded Image file, an aligned background grid of sprites (i.e. 64 128 256 etc...)
• A set of inputs with pixel offsets etc...
• Some javascript to update the CSS to match the offsets
• Someone else can deal with the database side of things which i try to avoid.

Naturally vehicles would be slightly different, but if smartly done all the angles could be aligned on a single screen quite quickly. Again, I'd be happy to help with the interface here.

Cheers,

[FooBar]

A different method would be to template all future graphics, like any other decent newgrf project does. Then there only needs to be a drop-down of sorts to select what template to use for the sprite.

[Lord Aro]

I don't think it (should) be too difficult to add a similar build environment to the devzone on the repository
Alternatively, we could migrate totally to the devzone, just keeping sources (perhaps) on the repo

Either way, i think we should wait until NML supports 32bpp-ez before continuing, making the coding process easier

[planetmaker]

Time allowing, i'd be happy to help Jupix or someone else create the interface on a website to be able to move a sprite left/right up/down to align it as though it were ingame.

That'd be quite nice to have. Also possibly NewGRFs could profit from this. At the DevZone we could and happily would provide you with all the necessary infrastructure.

Also andythenorth afaik once had a similar idea with BANDIT

[coyoteelabs]

Time allowing, i'd be happy to help Jupix or someone else create the interface on a website to be able to move a sprite left/right up/down to align it as though it were ingame.
Cheers,

You can already do that (more or less) with my Visual PNG codec program. I released it a while back to help with the sprite offsets. It also includes a crop tool to remove the minimize the sprite size as much as possible. It's licensed under GPL v2 and runs on Windows.

I'll try to update it to output the required code for GRF / NML when I have some time.