For several years we are running our own Compile Farm, with great success. Since the introduction of it, we can produce binaries at a much more steady rate (normally you had to wait between 1 hour and several weeks before flavor N uploaded the release binary), and the quality of these binaries is fully in our control. From my point of view, this has been one huge success.
Over the years the software running the CF has changed. We started out with some jails and cross compiling. These days we use Bamboo, and run several Virtual Machines to create all the binaries. Windows binaries are no longer cross compiled, and we use chroots to create debian/ubuntu deb-packages. The only target still cross compiled is Mac OS X; this is unlikely to change, as we don't have access to Mac hardware. We have come a long way, and the quality has been higher and higher.
What you don't see, is how very annoying it is to maintain the CF these days. (I am sure I wrote another dev-blog about that in the past ) When a Debian or a Ubuntu create a new "latest", a lot of tasks have to be preformed:
- Create a new chroot
- Upgrade, test, ...
- Create a new "executable" in Bamboo
- Copy an existing job, and change all the settings correctly (this goes wrong so often, as there are many settings )
- Test-run on next release
- Pray
This often means we are slow with new targets, and nobody really wants to do it. Besides that, only me and Rubidium can do it, as it requires a lot of know-how of the systems involved.
As you might understand, that has to change.
Additionally, lately there has been the requests if we can compile random patches on the interwebz to publish binaries from. You can see several threads where people publish their patch, and then they have to wait for supporter A to publish a Windows binary, so people can play that patch. I am really happy (and a bit proud) we have people doing that; but from a SysOp point of view, there are a few issues with that. Mostly .. we have a CF, why not use it for these things?
Well, the main reason is: security. The CF at the moment is build on trust. It means that if anyone would start a task on it which .. say .. starts a DDoS from the Makefile, it would do that. Not really something I want to open up for the public in one way or the other. And I don't want the other devs to filter every patch, just because I was incapable of securing the stuff.
Luckily enough, I am not incapable (ghehe), but over time the CF has grown into kinda of a monster, which makes it hard to change things enough to get the above requested security. The whole CF is on a separate network, and isolated from everything .. but because it has to publish its files, and for some projects (Simutrans mainly) it has to download files from other websites .. there are a few .. holes between this network and the main OpenTTD network. Well .. just one big huge hole, which says: forward all network Limiting that network is possible, but I don't fancy breaking stuff over and over till we found what should be open and shouldn't be.
So .. solutions!
Today I have been fiddling with Docker. For a while now I had the idea that could resolve most of our issues, and make maintaining the CF a lot easier. And as it turns out, it is! So next I will talk a bit about how I plan to use Docker. If you have any tips and suggestions, do let me know
So what is Docker? Docker is a containment method. Where Virtual Machines run a complete machine, Docker runs on your own kernel, like a jail/chroot, but a lot more contained. So what do I mean with that? Well, without boring you with all kinds of details, I can prepare a Docker that, when started, compiles stuff at a certain directory, and shuts down when that is done. Even more, I can make him do that without granting him any network access.
This of course is ideal for what we want. Even if horrible DDoS code would be started in such Docker, it wouldn't be able to do anything. The worst thing it can do, is waste CPU cycles till the task is being killed. That is more than acceptable to me.
During testing and installing all of this, I found out some awesome things. In random order:
- Bamboo supports Docker fine.
- Bamboo starts a Docker, mounts a volume, executes a command, destroys the Docker, and fetches some files from the mounted volume
- Docker can disallow network traffic from/to the containers
- Docker supports 32bit containers, although officially they say they don't. It required some fiddling around to get Debian and Ubuntu i386 containers, but in the end it was trivial
- I ended up creating my own base images, to make 32bit vs 64bit more clear
- I can prepare a Docker so you too can compile OpenTTD like the CF would; exactly the same
This opens a few doors. For example, because of the last point, I can easily share Dockers with other people, so they can improve on it. Take Mac OS X, we cross compile that. If I can get this in a Docker (I haven't tried this part, but I am sure it will be successful ), other people can use it too to make test-builds etc.
Also other projects, like Simutrans, which require a few other dependencies, are no longer difficult to maintain.
And behind another door, I can allow any dev to create a project to compile a random patch from the interwebz, without having to worry about security. This means we can create binaries for people with patches in threads (upon request or something most likely), creating a bigger userbase for them.
Sadly, we will still need a Windows VM to make the native Windows builds. But I just bribe Rubidium to maintain that (he is anyway ).
So what do I have at the moment? Well, let me show you:
Code: Select all
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
openttd-cf ubuntu-amd64-trusty 08fef76552b7 14 seconds ago 525.9 MB
openttd-cf debian-amd64-wheezy 3a78f4f338e8 5 minutes ago 440.4 MB
openttd-cf debian-amd64-jessie a33aea9789b2 5 minutes ago 564.4 MB
openttd-cf ubuntu-i386-trusty a5b2ad73d51e 5 minutes ago 500.4 MB
openttd-cf debian-i386-jessie 2786a64dd1b9 12 minutes ago 560.5 MB
openttd-cf debian-i386-wheezy 9508cf0e0a08 12 minutes ago 432.2 MB
ubuntu-i386 trusty 53e2401d7c20 23 minutes ago 177.6 MB
ubuntu-amd64 trusty 8441e92dcb79 24 minutes ago 187.9 MB
debian-i386 jessie ecc313246f26 26 minutes ago 122.2 MB
debian-i386 wheezy 4391b229012e 28 minutes ago 82.4 MB
debian-amd64 jessie 482859149a4e 31 minutes ago 125.1 MB
debian-amd64 wheezy 17c30a806df6 34 minutes ago 84.89 MB
Dockerfile:
Code: Select all
FROM @OS@-@ARCH@:@RELEASE@
MAINTAINER Patric 'TrueBrain' Stout
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends build-essential debhelper fakeroot libfile-fcntllock-perl
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends libsdl1.2-dev zlib1g-dev liblzo2-dev liblzma-dev libfontconfig-dev libicu-dev
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends subversion git mercurial patch
ADD release.sh /usr/bin/
ENTRYPOINT ["release.sh"]
CMD ["dev"]
Code: Select all
ln -sf os/debian debian && mkdir bundles
fakeroot make -f debian/rules binary
mv ../*dbg*.deb bundles/openttd-${VERSION}-linux-${OS}-${OS_RELEASE}-${ARCH}-dbg.deb
mv ../*.deb bundles/openttd-${VERSION}-linux-${OS}-${OS_RELEASE}-${ARCH}.deb
I am currently very happy with the result. In Bamboo a single command compiles via Docker a whole target. This makes it very trivial to create/add new targets, or update current ones. I am very very happy with it.
So what is next? To put it in a list:
- Update Bamboo to 5.9
- Install a Docker VM
- Deploy my new code there
- Start building debian/ubuntu via Docker
- Start building linux nightly and CI via Docker
- Check in code in our VCS (so everyone can use it, update it, etc)
- Investigate Mac OS X via Docker
- Swicth from Windows 2003 to Windows 10 for Windows VM
- Rewire how the CF publishes its files (it now goes over a few layers, don't get me started )
- Create a wiki page explaining the ins and outs of how we use Docker
So yeah, a long way to go before it is all done etc, but the progress has been awesome.
Hope you enjoyed this read. If not, why did you bother? You only have yourself to blame here