LeaseWeb, our provider, supplies us with 2TB a month. For months, this was plenty. We utilized more than 25%, but never more than 50%. Since the 1.0.0-beta1, this changed. After the first week of January, the estimated bandwidth usage for that month was over the 3TB. This we could not supply, not without paying (lots) of extras for overusage.
Since 2 years I have silently been collecting mirrors when ever I could get them. Because it went silently, and we have a very long list of demands, we collected 3 of them so far. One in CZ (supplied by Ignum), one in GB (supplied by peter1138), one in US (supplied by Zernebok). The idea was to slowly introduce a mirror system in 2010, offloading the downloads of people to close by their country. This idea is now accelerated.
Since two weeks, people are randomly distributed between the mirrors when access any file on http://binaries.openttd.org. And this is truly random. Of course this is far from a nice solution (I live in the NL, why would I start a download in US?), a better method was required. But it was required to make sure we wouldn't go over our max bandwidth.
Today we rolled out that method. At the bottom of this post I explain to you how it works. Basically, the balancer I created does its at most to pick a mirror close by you, but also keeps in mind load should be distributed over all the mirrors. We do need to finetune this system, and this will take days, if not weeks, if not months to do so. But the idea works. This topic is only partly about informing you about this.
What I really want to write about, is a request. A request to any ISP or other party that can supply us with diskspace and bandwidth. Our demands are high, we know. But all for a good reason. So, here it goes:
We are looking for people who can supply us with:
- 100 GB of diskspace
- at least 1 TB of bandwidth
- 100 mbit/s connection or faster
- IPv4 + IPv6 support (the latter is optional, but we do filter on it)
- HTTP access to that data (FTP optional, but it would be nice)
- rsync over SSH for pushing the files (SSH account can be restricted to only allow rsync, only in that directory, no problem)
- Server has to be located in a datacenter
- One mirror per country (with the exception of US)
If you feel you can donate the above to us, please email me (email listed at http://www.openttd.org/contact ) with the details of your offer, the AS you are directly connected to, the country, and geographical coordinates of the server. If you wonder why we need the latter, please read below.
I like to remind you at this point that we are an Open Source community, and cannot offer you any money in return. The only thing we can do for you, is name you in the list of mirrors, and at the mirror itself (check http://master.binaries.openttd.org/, you can see 'Sponsored by: LeaseWeb' there. There you can put your own banner).
We _are_ looking for another mirror in NL, as we want to offload all binary downloads from our main server (it already has to serve BaNaNaS, subversion, mercurial, ...)
[Update] We are no longer looking for an NL mirror. Thank you SNT![/Update]
Which mirrors do you have currently and where are they located?
CZ - Sponsored by Ignum (http://www.ignum.cz/), http://cz.binaries.openttd.org/openttd/binaries/
DE - Sponsored by Alexey Polyakov, http://de.binaries.openttd.org/binaries/
GB - Sponsored by peter1138, http://gb.binaries.openttd.org/binaries/
NL - Sponsored by Studenten Net Twente, http://ftp.snt.utwente.nl/pub/games/openttd/binaries/
US - Sponsored by Zernebok (http://www.zernebok.com/), http://us.binaries.openttd.org/binaries/
How does our mirror process work?
When we make a binary on our compile-farm, the files are distributed to all the mirrors afterwards. When this is done, the new binaries are announced. So after every nightly, the files are directly rsynced to the mirrors. Because of this there is no delay between mirrors. It also means there is no pull on the mirrors, but we push the files to them.
How is a mirror selected for a request?
Via GeoIP. When you request a file from http://binaries.openttd.org, a balancer picks up the connection, and checks your IP via GeoIP. Based on that output, it finds the closest mirror to you, geographical. Of course this is not always the fastest, but it is as close as we can estimate. To not overload a single mirror, a formula is used to pick the best mirror:
Distance * Priority of Mirror * Load of Mirror
The Priority is normally 1. The higher, the less chance a mirror is selected.
The Load is increased when someone downloads something from the mirror. This means that when you hammer on one mirror, at a given moment, you will get moved to another mirror. This should prevent one mirror taking all the load, while others do nothing.
If you have any suggestions or ideas about the system, let us know (via this thread). Mind you, we considered carefully what mirror system to use, and how to implement it, so don't be to judgemental about this approach, and think before speaking. Also, ask before saying, most things have a reason
, I know student associations that generate a lot of traffic with just movies of their own kite-surf activities (700MB a piece where it's not uncommon to get over 100 hits.)
