Smaller ZIP/7z package (by from ~3.5% to ~28% for OpenTTD)

[CobraPL]

Greetings,

how about making smaller ZIP/7z/7z-SFX package?

openttd-0.7.0-beta2-windows-win32.zip

1. 3419306 bytes - original size
2. 3306146 bytes - repacked using 7-zip - with: pass=10 fb=258
3. 3299626 bytes - Optimized using ZIPMAX. - 96.50% of original size
4. 3249106 bytes - optimized PNGs + ZIPMAX
5. 2984825 bytes - optimized PNGs + main exe packed with UPX 3.03 with "--lzma --best" + ZIPMAX - 87.29% of original size
6. 2490051 bytes - optimized PNGs + main exe packed with UPX 3.03 with "--lzma --best" + 7z format made with maximum compression, solid archiving, 64mb dictionary size and single thread.
7. 2463158 bytes - optimized PNGs + 7z format made with maximum compression, solid archiving, 64mb dictionary size and single thread - 72.04% of original size; for 10000 downloads it saves ~8.9gb of traffic
8. 2603446 bytes - optimized PNGs + 7z format made with maximum compression, solid archiving, 64mb dictionary size and single thread - SFX archive, so you can unpack it on any computer w/o 7-zip. 76.14% of original size

Some notes:

• Recent WinRAR versions support 7z unpacking.
• As you can see, UPXing main exe causes better compression for ZIP and worse for 7z - because 7-zip uses BCJ2
• UPXing (any) exes causes false alarms made by pesky AV programs. Luckily, only eSafe cause problems for OpenTTD.
• NSIS installer uses the same LZMA compression as 7-zip. So it owns any other solution.
• See also: Optimizing PNG files (by ~45% for OpenTTD)

Peter

[Conditional Zenith]

Better optimization would be achieved by using something other than zip.

[CobraPL]

Better optimization would be achieved by using something other than zip.

I agree, first post edited. But perhaps DEVs want to stay with zip because of compatibility. BTW. I saw some projects with both .zip and .7z releases.

[MrFrans]

I saw some projects with both .zip and .7z releases.

That is probably a good idea.

[Rubidium]

Is that 'better' compression tool available on freshly installed computers? Guess not. Care to teach people how to install such a tool just to install OpenTTD? I don't think you're willing to do that and even if you would most people would be scared away because "it doesn't work".

Anyhow... by far most people download the installer which is significantly smaller than what you can reach by optimizing the zip. Furthermore the creation of the packages happens completely automatic on a Unix machine. After all... most "simple" users want the installer because extracting the zip and placing the TTD graphics in the right place is very hard for them.

We're also very hesistant to changes of the system we use to build OpenTTD as that requires much testing. Especially when you want us to install new and unknown applications. It's quite a fragile ecosystem and minor changes can have very unwanted effects such as loss of network connection (strangeness between the ISP's hardware and the software used to build our binaries) which is very inconvenient when you've got no (quick) terminal access.

So long story short: in my opinion the effort use another untested/unknown tools to create the packages outweighs the gains of it.

[FooBar]

For 10000 downloads it saves ~92mb of traffic

That's hardly worth the effort...

[XeryusTC]

3% isn't that much of a saving on a package IMO, hardly worth the effort.

[Wold]

We can save a about 25% using kgb for the whole archive, but is it worth it?
I think the idea is to use what every windows has - WinZip. Quick and simple, isn't it?

[Conditional Zenith]

Yes, zip for all it's problems is the most widely supported general-purpose compression format, and can be handled by most operating systems out of the box.

And I agree, a 3% saving is barely worth the effort.

Having said that, I don't see why there can't be another format as well, and 7zip makes the most sense IMO. It compresses well, has an open source implementation on all 3 major platforms, and a linux commandline implementation. However it sounds like the OTTD server goes offline if you look at it funny (which I don't get, but the devs say it is so), and zip alone works well enough.

[Rubidium]

The server goes offline under certain loads. We cannot reproduce this on our home computers and do not want to unneededly compromise the stability of the server by experimenting in that (fragile) ecosystem which seems to be quite stable lately.

[CobraPL]

Guys, from the beginning I thought about optimizing only major builds, for example 7.0.0 final or betas/RC, not nightly ones. I didn't mind envolving/changing system - rather doing it by hand, once for major release. Anyway, devs are aware of such technique and will do what they want...

[TrueBrain]

Guys, from the beginning I thought about optimizing only major builds, for example 7.0.0 final or betas/RC, not nightly ones. I didn't mind envolving/changing system - rather doing it by hand, once for major release. Anyway, devs are aware of such technique and will do what they want...

The whole compile process (both nightly as stables, as any branch) is completely automatic and done by our compile-farm. Doing any manual actions on them is not why we created the system

Either way, back on the original subject: 3% is not worth the CPU time required and the effort to test and double check the changes. We have plenty of bandwidth, and if we run out of that, we have a mirror-network in creation. So that won't be done for sure

In general, I see more and more people using 7-zip. I personally still don't have in installed on any of the boxes I manage (which are in fact a lot ). Nevertheless, it might be worth the effort to look into this, and make it possible to generate 7z packages for a few targets. Based on the amount of downloads we can see if it is really used. I will prioritize this on my list as 'very low', and see if I can add it in the next few weeks.

Anyway, tnx for the pointers

[CobraPL]

First post edited, detailed info

[CommanderZ]

3 or 30, OPENTTD is packed so it is easy to download in one file, not to be smaller. That megabyte really doesn't matter nowadays. You can unpack basic ZIP on every Windows PC with Win XP+ without installing any additional software, 7z hardly.

[Conditional Zenith]

Yes, ditching zip is a bad idea, we know that already. Having something else as well wont stop people from downloading the zip file if zip is the only format they can handle. Anyway, "very low" is being generous I think.