For year and day we have a balancer running on binaries.openttd.org, which redirects clients to proper mirrors all over the world. A few years ago I wrote a python script that does that, with the idea to once replace it with something more powerful. Now finally, that day has arrived. The reason it took so long is two-fold. First, the current script is much more stable than I ever expected. Second, for work I program a lot .. when I get home, I am not in a programming mood
But lately the need for a rewrite has increased drastically, so finally ... OpenTTDMirror/2.0 is a fact.First, what does the old balancer do? Well, when you make a GET-request to any binaries.openttd.org, it redirects you to any mirror we have in rotation. It uses GeoIP to find the one closest to you, does some magic with the load, and picks a relative close one but not overloaded (by us). The balancer assumes all mirrors are identical to the master, and makes no attempts to validate if a mirror is in sync or even online.
A secondary system rsyncs the files to the mirror, triggered by scripts and who-knows-what-else.
This method works very well, but there are also many things wrong with it. First of all, it has no clue when a mirror is being sync'd, so it has to guess a bit. Well, a lot in fact. Basically, it assumes that if a file is on his local filesystem, it also has to be on the mirrors. This gives issues with the first minutes of any release. It is also one of the reasons we already have a release (like 1.2.2) on the mirrors for a while, before we notify anyone about it. It is a silly restriction, and rather annoying.
Much more important, and annoying, is that all mirrors have to be full. We, as developers of OpenTTD, and more specially: me, love having an archive. I wouldn't dream of removing the 0.3 binaries from our archive. Not that it is useful, but it is nice to keep and have. We have every source of every nightly we ever published. Why? Because we can! But .. every mirror is atm 44k files big. And those old files are accessed ... rarely. So why do we mirror them? Not because we want to, but because we have to.
When a mirror went offline, or rsyncs failed, it still directs you to such mirror. Often this leads to problems, users complaining and the likes.
Lastly, I have had offers of mirroring under the condition only certain IP ranges would be able to access it (countries with expensive transits (non-local-routable-traffic), think Australia etc).
So no more! I rewrote the whole balancer, and made it much more powerful. With the exception of the last entry, which will come when we really do get a mirror which demands that, it can handle all. A brief summary of the new system. It is rather complex, so I hope you can follow it a bit.
First, it now uses inotify. This means that if a file is changed, the balancer gets informed about it. This is mostly used to detect when new files are being uploaded to the binaries. After 2 minutes of no new files, it starts the rsync process (more on that below). It also detects changes in its configuration directory. When we add a mirror (or delete on), it is immediately applied (runtime). No longer we have to cycle the balancer etc.
The rsync process is now built-in the balancer, and much more clever. It starts off by syncing all the small files on the system. Next, it requests a total list of files that are on a mirror. Then it sends the larger files, requests the list again. Finally, it sends the biggest files (Mac OS X binary big), and then finishes off with asking the file list again. What happens with that file list?
When the balancer starts up, it thinks all mirrors are out-of-sync, and only serves files from the master. When those lists of files come in, it knows which are in sync with the master, and updates its internal database with which mirror has those files too. Next time you ask any of these files, it has a bigger pool of mirrors to pick from. When a file gets changed on the local system, it removes all mirrors from this entry, and once again only the master will serve it. Till the next sync. Then the lists come in again, and it updates the entries.
This system works very well. It also means we can only send a selective set of files to a mirror. And this can be anything. We are in control. Lovely.
So, mirrors no longer have to contain all our files, while when you request any of those files not on other mirrors, the master will serve them for you. And if a mirror cannot be rsync'd to, it will simply remove itself from rotation.
There are many more small and big improvements to the balancer, but I won't bore you with the details. Most noticeable to close off with, is that the whole system is threaded and non-blocking. Lovely, aint it?
The new balancer will go live in a few, depending how the final tests go is live. To close off this post, there are a few other reasons we had to rewrite:
The current balancer is still hosted by an old system of ours. I want to put that system to rest, but the mirror/balancer stuff is a very integrated part of it. This is one of the many small changes needed to move it away from that old system. It also opens the way for bigger BaNaNaS files, which is highly overdue.
At the moment the old rsync system is still chained. New binaries first arrive on the old system, does a lot of magic, then it is rsynced to its mirrors; now there is only 1 entry in that mirror system: the new binaries. When it arrives there, within 2 minutes it starts to send the files to the real mirrors.
Next on my agenda is moving away from this chain more. The new system has to generate the required index.html, it has to update the timestamps on the files when required, it has to update side-systems like finger, react on releases ... and many many more hooks we have running. Lastly, our CF has to be told it has to upload its files to the new host; only then I can remove the old mirrorsystem from the chain. It will be a long and bumpy road, but hopefully totally unnoticeable by you.
That all said and done, if you notice any issues with the new balancer, please do let me know ASAP. It is most likely a bug or a problem.
I thank you for reading, I hope it was somewhat coherent; it was a lot harder to write than some of my other posts .. it is hard to not get technical, while still keeping you informed.
