zBase (32bpp base set by Zephyris)

[CargoLiner]

just a quick test, the following powershell 3.0 script ran from your \infrastructure\road\roadbits\road_stations directory produced the following output

Code: Select all

$grfName = "TestCompile"
$GFXdir = ".\"

Write-Output "grf {
    grfid: `"AB\01\01`";
    name:   string(STR_GRF_NAME);
    desc:   string(STR_GRF_DESCRIPTION);
    version: 0;
    min_compatible_version: 0;
}" | out-file -FilePath .\$($grfName).nml -Encoding "UTF8" -Append

$si = (Get-ChildItem .\ -Recurse -Include *.png -Exclude *m_*.png| 
select fullname, basename,
@{n='Index'; e={($_.BaseName).split("_")[1]}},
@{n='ZM'; e={($_.BaseName).split("_")[0]}}|
Group-Object -Property Index -AsHashTable).GetEnumerator() | sort {[int]$_.Key}

foreach ($i in $si) {
Write-Output "replace hd_$($i.Name) ($($i.Name)) {[0, 0, `".\gfx\empty.png`"]}" | out-file -FilePath .\$($grfName).nml -Encoding "UTF8" -Append

foreach ($z in $i.Value.GetEnumerator()) {

switch (($z.BaseName).Split("_")[0]){
"256" {$zm = "IN_4X"
$xoffset = -124
$yoffset = -132}
"128" {$zm = "IN_2X"
$xoffset = -62
$yoffset = -66}
default {$zm = "NORMAL"
$xoffset = -31
$yoffset = -33}
}

if ((Test-Path (".\$($z.ZM)m_$($z.Index).png")) -eq $true){
$mk = ", `".\$($z.ZM)m_$($z.Index).png`""
}
else {$mk = ""}


Write-Output "alternative_sprites (hd_$($z.Index), ZOOM_LEVEL_$($zm), BIT_DEPTH_32BPP)
{
[$($xoffset), $($yoffset), `"$($z.FullName)`"$($mk)]
}" | out-file -FilePath .\$($grfName).nml -Encoding "UTF8" -Append


}
}

[FooBar]

About the cropping: NML can do that for you, so there's no real need to optimize png files to be as small as possible. So that won't be an issue.

Looking at your NML file: I wouldn't use absolute paths, they're a nightmare when it comes to compiling on different systems. Myself I've also been thinking about automating this as much as possible. A Windows script is probably not the way to go (the devzone server runs linux), but your ideas can certainly be used.

[CargoLiner]

the absolute path is actually a function of the $_.fullname where $_ is actually the file object itself so it will fetch the absolute path of the system that the script is running on, i can probably attempt to do a relative path lookup or just make assumptions

i'd figured that linux was being used and that the compile script was probably running server side, but in the off chance that the nightly build is done locally and then pushed to the repo then the script is pretty easy to tailor to your purposes, though it probably would not take too much for me to convert to perl or python

im currently going through some of the zBase Build repo i do notice some script that run through PNG files to try and determine y_offsets which would suggest some cropping maybe occurring

[Zephyris]

That looks very interesting. It might be worth chatting with Alberth and planetmaker who have also tried some automated coding...

*edit*

im currently going through some of the zBase Build repo i do notice some script that run through PNG files to try and determine y_offsets which would suggest some cropping maybe occurring

The few scripts that are there aren't currently in use, but they may be in the future. You can ignore them for now

[planetmaker]

About the cropping: NML can do that for you, so there's no real need to optimize png files to be as small as possible. So that won't be an issue.

Oh well. It's not an issue for a single file. But for 10000 files added up. With different revisions, that makes possibly a huge difference which easily adds up to tens, if not hundrets of megabytes of repo size... (try to clone ogfx+trains which grew to 1GB repo size this way. And not because it was mis-used in anyway, but just by using it normally as one works. Keeping files as small as feasible, would probably cut off something around 100 and 300 MByte)

[FooBar]

Hmm, didn't realize that. If ogfx+trains is already 1 GB, this can easily grow to 20 or more. Which is quite significant

[Zephyris]

The biggest part of the repo size is the .blend files. These are not compressed and easily reach several megabytes. Two backup files (.blend1 and .blend2) are also automatically created whenever a .blend file is saved which triples the storage usage. I don't know enough about managing a repo to know if it would be possible but the best way to minimise the repo size is automatically compressing all .blend files and refusing to track .blend1 and .blend2 files.

[FooBar]

.blend1 and .blend2 can be easily ignored by means of .hgignore.

Simply add this to it:

Code: Select all

*.blend1
*.blend2

I don't know if it's possible to automatically compress certain files.

[planetmaker]

Compressing files is IMHO not a good really desirable solution; we want usually the sources to work with, and not compressed sources. Compression for the sake of transfer of repos is a matter of the transfer protocol, not of the repo itself (and binary files are not that well compressible anyway).

Not tracking files which are automatically (re-)generated of course is desirable as they then do not contain any new information.
But I don't know blender enough to know whether those files are necessary. If they're not needed and automatically re-generated by blender, then they imho shouldn't be added to a repo.

[Alberth]

zBaseBuild is just a repo for experiments in coding the zBase graphics. It started 2 days ago (one day at my local hard disk, and one day as zBaseBuild). So far, it's just doing some terrain sprites.
It assumes that ../opengfx is the OpenGFX repo, and ../zbase is the zBase repo, and pulls the 8bpp and 32bpp images from them.

Eventually, the build stuff will end up in either zBase or OpenGFX.


The first thing that I noticed is that the currently normal approach of using CPP for generating NML fails with zBase. Reason is that files like "64_0001.png" to "64_0024.png" contain 19 terrain sprites (yep, 5 are not used), but in CPP you cannot compute a file-number from a base+offset, and format it into a 4-digit number. (A pattern like "64_<type>_<number>.png" would have been much easier here.)

I can see two alternatives at this time:
- Python, a general purpose programming language, also highly usable for scripting. (Also used by NML, so no new dependencies.)
- m4, a dedicated macro processor.

Note that any solution MUST be cross-platform, Windows only, or Unix only is not going to work (although the latter is slightly less worse, as the servers for the nightlies all run Linux).

At this time, I opted for Python, as I am most familiar with it. Xotic750 has an m4 approach that I still have to look at.

My first attempt is to generate the right NML directly (basically a few 'print' statements with a table, and some start offsets and filenames), but it feels as being too low level, too many details to handle at the same time. On the other hand, it will be flexible, 'magic' like inserting a crop operation at some point will be easily possible.

'm4' is more like CPP, and may be easier to understand for non-programmers. On the other hand, the use of the right quotes at the right spot is critical, and m4 is not helping at all here, it happily produces horribly wrong output.

I am currently considering a more table driven approach, that is, you make a file with all the numbers, and a script produces the NML from it. Ideas are welcome, my experience in coding baseset-ish NewGRFs is about 2 days.


I am not doing any cropping at the moment, I let NML do that. the 'get_y_offset' was just for me finding a magic number
If you want to add cropping, please save the offsets in a file. It would be a huge waste of time having to find that information again. (This also fits nicely in a table driven approach )


As for the backend, I currently generate NML, but it takes ages to compile. I have heard that grfcodec is orders of magnitude faster in processing. For something as simple as a bunch of real sprites, it might make sense to generate NFO directly. NFO does not do cropping afaik, so that needs to be inserted then.

[Zephyris]

But I don't know blender enough to know whether those files are necessary. If they're not needed and automatically re-generated by blender, then they imho shouldn't be added to a repo.

The .blend1 and .blend2s are completely unnecessary, I hadn't worried about them before because I hadn't realised they were such big files. I'll try and get them excluded.

[FooBar]

For something as simple as a bunch of real sprites, it might make sense to generate NFO directly. NFO does not do cropping afaik, so that needs to be inserted then.

I agree. Especially if all code is auto-generated, there's no real need for NML. Nobody is going to read the generated output anyways.
grfcodec -c does cropping, which should be fine.

The table approach you proposed seems sensible to me. It may be low-level, but why make it more complicated than necessary? The table can list sprite numbers and what offset template to use for that sprite. These templates can simply be defined in the (pyton?) code in a separate file.

[planetmaker]

Yes and no.

One of the advantages of NML over the NFO is the use of sprite templates, which especially make sense for the repetitive tasks of coding 19 (or 23) ground sprites, the track or road sprites.
Also there's no need to e.g. create all black&white font sprites in 32bpp or the GUI elements like window close box. This would all need completely new coding while it can simply be taken as-is from OpenGFX when sticking to NML.

The real fun with an NFO-choice begins when coding the extra grf which definitely is need. Hint: the rivers alone are 1000 lines of nml but allow for gradual snowyness of the grass or desert transition they reside on.

Thus my suggestion would be to write NML from that script. It would also allow the use of sprite templates for repetive ground sprites etc. Then let nmlc write the nfo which then is compiled by grfcodec for speed. And we'd have readable grfs, should nmlc learn to encode sprites faster.

[FooBar]

Templates aren't the biggest problem, they can be implemented in a different way for NFO as well. For especially the extra grf NML is indeed the better choice, I have to agree.

[Zephyris]

FWIW I think it would make sense for the nfo/nml generation to work off very simple tab-separated text files which define which grf file the sprites belong to, the sprite numbers, the folder to look in to find the 32bpp sprites, the render number corresponding to that sprite and the x/y offsets for its alignment. As people are saying let grfcodet take care of automatically croppping the sprites.

Code: Select all

grffile spriteno path                 index   xoffs yoffs
base    2456     terrain/temperate/    0000    0     0    
base    2457     terrain/temperate/    0007    0     0    
base    2458     terrain/temperate/    0006    0     0    
base    2459     terrain/temperate/    0005    0     0    
base    2460     terrain/temperate/    0004    0     0    

[Alberth]

Such a table is simple, but that leads to much duplication since you don't exploit similarities in structure.
Also, it should take the 8bpp graphics into account which you also need for making a newgrf.

With respect to NML vs NFO, I got convinced that only NML is feasible in the long run. Given the long life time of a base set, the decision should be made based on minimizing long term maintenance effort, not what is the easiest just now.
At this moment, NFO generation and NML generation is anout an equal amount of work. When you want to add other features to it (and that WILL happen), coding them in NFO is a a lot more work than in NML. In addition new coders learn NML only, so the number of people understanding NFO will decrease.

[FooBar]

Good points there Alberth, I'm convinced now and will no longer suggest NFO

[Alberth]

As an alternative, what about this format:

Code: Select all

[root]
8bpp-xsize = 64
8bpp-xoffset = -31
064-xsize = 64
064-ysize = 64
064-xpos = 0
064-ypos = 0
064-xoffset = -31
064-yoffset = -32
128-xsize = 128
128-ysize = 128
128-xpos = 0
128-ypos = 0
128-xoffset = -63
128-yoffset = -64
256-xsize = 256
256-ysize = 256
256-xpos = 0
256-ypos = 0
256-xoffset = -127
256-yoffset = -128



[3924]
parent = root
sprite = 3924
8bpp-filename = "../opengfx/sprites/png/terrain/bare-03-temperate.png"
8bpp-xpos = 1
8bpp-ypos = 1
8bpp-ysize = 31
8bpp-yoffset = 0
064-filename = "../zbase/terrain/terrain_temperate/64_0073.png"
128-filename = "../zbase/terrain/terrain_temperate/128_0073.png"
256-filename = "../zbase/terrain/terrain_temperate/256_0073.png"

Basically, you have to define a lot of values for each sprite, where sprites share common parts. An INI file like above can handle that nicely. Rather than define the full set of values for each sprite (which is what the table approach does), put the common stuff in the "[root]" section, and refer to it from the other sprite-sections by "parent = root".

Having a parent saves a lot of typing. However, you can do more.
If you also allow several parents, you can do tricks like

Code: Select all

[terrain_flat]
parent = root
sprite-offset = 0
8bpp-xpos-offset = 0
8bpp-ysize = 31
8bpp-yoffset = 0
32bpp-file-offset = 0

[terrain_w]
parent = root
sprite-offset = 1
8bpp-xpos-offset = 80
8bpp-ysize = 31
8bpp-yoffset = 0
32bpp-file-offset = 7

; and 17 other entries (omitted for briefness)

[terrain]
parent = terrain_flat terrain_w terrain_s terrain_sw terrain_e terrain_ew terrain_se
	terrain_wse terrain_n terrain_nw terrain_ns terrain_nws terrain_ne
	terrain_enw terrain_sen terrain_steep_n terrain_steep_s terrain_steep_w terrain_steep_e


[temperate_grass_0]
parent = terrain
32bpp-file-base = 73
sprite-base = 3924
8bpp-filename = sprites/png/terrain/bare-03-temperate.png
064-filename-base = terrain/terrain_temperate/64_
128-filename-base = terrain/terrain_temperate/128_
256-filename-base = terrain/terrain_temperate/256_

You define 19 sections, one for each tile-variant in a terrain-type. The "[terrain]" section is empty, except for the 'parent' entry. It means there are actually 19 sections all called 'terrain'. The final section "[temperate_grass_0]" is also copied one time for each 'terrain', so this entry expresses 19 sprites instead of just 1.

The small additional changes like "32bpp-file-base" and "32bpp-file-offset" which added together construct a 4-digit file name number are crucial here.

[LaDoncella]

hmmm.. interesting

[CargoLiner]

@Alberth
forgive my ignorance, i understand the usefulness of such a setup when using spritesets or sprite templates, but from what i understand, all the sprites and variation of will be rendered to their own individual PNG file at full frame, the offsets will be constant for every sprite of a given zoom level (i may be wrong in this assumption but this is what i got from the discussion so far, would be great if someone can clarify this further), so for all intent and purposes there will not be 19 different blocks to define terrain, as all definitions will be identical so a single definition would be used, in fact we could take this a step further and say that there is generally speaking only 3 major definitions one for each zoom level for all sprites

the biggest hurdle would be to itemize and group them into replace and alternative graphics blocks (along with if statements for different climates), if we are doing more than just replace graphics in the NML then that's a different kettle of fish, but i was lead to believe this is a base graphics replacement project

@Zephyris
a simple file would work, i suggest the grfFile field is not required, this can be deduced from the file name (i.e. Base.ini would be for Base grf build), remove the x and y offs (im presuming full frame render), this would then make maintenance much easier as we need only concentrate on spriteID path and render number, would all render be assigned a unique number or would there be instances of the same number being used in a different directory (from what i deduce from your blender setup, this is dependent upon the batch render jobs, each batch starts at 0000 and is incremented for each frame in the render), if not then i suggest we pair the path and the render index together and separated by a : (or some unique identifier) that way we can then add multiple renders to a single spriteID for different climates which the IF statements can key into

Code: Select all

2456     terrain/temperate/:0000	terrain/artic/:0000		terrain/jungle/:0000
2457     terrain/temperate/:0007	terrain/artic/:0007
2458     terrain/temperate/:0006
2459     terrain/temperate/:0005
2460     terrain/temperate/:0004	terrain/jungle/:0004