OpenTTD source migration and other changes

OpenTTD is a fully open-sourced reimplementation of TTD, written in C++, boasting improved gameplay and many new features.

Moderator: OpenTTD Developers

TrueBrain
OpenTTD Developer
OpenTTD Developer
Posts: 1312
Joined: 31 May 2004 09:21

OpenTTD source migration and other changes

Post by TrueBrain » 14 Apr 2018 22:31

Last weekend OpenTTD migrated its source (Subversion) and bugs (FlySpray) to GitHub (https://github.com/OpenTTD/OpenTTD). This change was long overdue, and after months of slow-chatting ideas back and forward, last month we finally reached a decision: we are migrating to GitHub.

This migration took a few weeks of preparation; and we are still in the process of dotting the 'i's and crossing the 't's, but the result is very pleasing and we hope you will like it too.

So what has changed?

We ditched Subversion, and now use git. git has been in widespread use for several years now, so although we understand it can take a bit of time getting used to, though there are plenty of manuals around to get you started. We strongly advise you to read the Guides of GitHub (https://guides.github.com). It is really good for the basics. The benefits for git over subversion are ample; to name a few: local repositories, your own branches, decentralized, local history, ..

Next we ditched FlySpray, and are now using GitHub Issue tracker. FlySpray was on the list for replacement for years. It hasn't been maintained upstream (well, recently there are some commits; but nothing that changes the issues we are having). A replacement was hard to find; but GitHub's Issue Tracker is sufficient for what we want.

But more importantly, GitHub has Pull Requests. This replaces "patches" on FlySpray. The way OpenTTD was handling patches (adding attachments to FlySpray issues) was very old .. and badly needed replacement. The inability to leave in-line comments and see what changed was really hurting reviews, up to the point where they didn't happen.

The other big change here is that Pull Requests are now validated by Continuous Integration before they are merged; meaning the main branch (master, or what used to be trunk) is a lot more stable, and reviewers of Pull Requests have fewer things to worry about (they can see it compiled on a various of compilers/platforms, instead of finding that out themselves).

In other words: the workflow for patches has now drastically improved; and we have been experiencing that over the last few days already.

The migration to GitHub has also another reasoning: it is much more open and reachable for other people. GitHub is very well known for a lot of developers; this means that we reach more developers that "just make a PR to fix this tiny thing". And because reviewing is easier, it is also much more likely to be accepted. Win-win-win-win.

The last few weeks we have been working on plans, scripts, and scenarios on how to get here. We imported all FlySpray issues (closed or not) to GitHub, keeping the number as it was on FlySpray the same - making #1 on FlySpray also #1 on GitHub. Next, last week, we have been cleaning up all the issues on GitHub. We looked at what could be closed, what needs attention, etc. We are down from over 400 issues, to only 235 issues at the time of writing, and we plan to prune it down even further.

We also use a new policy in regards to closing issues; keeping issues around with "a good idea" or "not really a bug but we should maybe fix it" turns out to have the reversed effect: nobody looks at it anymore. Although we really appreciate feedback and ideas, we will close enhancement requests that we don't expect to fulfill in the next year. Also bugs that are not really bugs but feature requests, are much more likely to be closed with the message: "we are going to pass on this one; thank you though". Many of those ideas etc do have a place on the forums; and if enough people like it, someone will stand up and make it :) We just don't want the Issue Tracker to become a huge sinkhole of endless ideas nobody is going to do anything with.

Some random facts: during this cleanup we found 35 tickets that went under the radar and still need triage; we invite anyone to help us with that. We are down to 65 enhancements, 46 tickets we consider "good first issues", 136 real bugs (well, most likely anyway), and still have to process 69 patches that were imported from FlySpray.

The latter is a bit difficult, as we would much rather prefer that the original author make the Pull Request than we. But slowly we will work on the list, and make a Pull Request ourselves. This will take some time.

We also have been working on making a better more robust Compile Farm. The old one wasn't really maintained and most targets didn't support C++11, let alone C++14 or C++17. This was hurting development.

So what we did over the last few months, is dockerize (https://www.docker.com/) the Compile Farm, and put the source of it up (on GitHub of course): https://github.com/OpenTTD/OpenTTD-CF/. This is still being worked on, and is one of the priorities in the next few weeks. These Docker containers allow you to locally build the exact same binaries as we do for releases, but also to validate your code under other compilers and platforms. Every Pull Request is now validated by these validation dockers and the results are reported back to the Pull Request. This comes back to the earlier point: developers see issues faster, meaning they need less time doing "housekeeping", meaning they have more time doing to good stuff: accepting Pull Requests ;)

The other thing we are working towards, is making it possible for anyone to easily get release binaries for their fork. This is mostly for Patchpacks, but also for big new features which needs good user-testing (NRT, for example). We are attempting to allow with some simple administrative actions the Compile Farm to create those binaries, and make them publicly available. This of course means that "official" OpenTTD gets into competition with Patchpacks; and we approve of this. We are fully aware it could lead to a Patchpack being more popular than "official" OpenTTD; and we will be upfront about that.

This does need some drastic code changes; for example savegames are currently not compatible between Patchpacks and "official" OpenTTD, and merging from one to the other can lead to nightmares in savegame compatibility. There are already solutions in Patchpacks available, and we will actively work towards getting them in the "official" OpenTTD.

So what more is there to come? Last week we closed off our local mirrors of Git and Mercurial in favour of GitHub. This means https://git.openttd.org/ and https://hg.openttd.org/ now redirect to GitHub. All the repositories hosted on there are no longer available. If you depended on them, we do apologize. The reason for this is simple: if we keep them alive, it might be that you, as user of such repository, will never notice it is not being updated anymore. And we don't really have another way to make you aware, other than shutting it down. A bit drastic, but sometimes you need to break eggs to ..

svn://svn.openttd.org/ is currently still available, but by the end of this weekend will also be unavailable. It can still be reached at svn://svn-archive.openttd.org/. Here too, we want people to know we moved to GitHub. Not seeing any updates is worse than not being able to reach the address, going to the website, and figuring out what is going on. Again, apologies for any inconvenience this might give.

Last in this list, the git on GitHub has been recreated. This means that if you have some branches of your own based on our old git mirror, you have to do some action before you can work with the version on GitHub. Mostly, this involves: git remote add upstream https://github.com/OpenTTD/OpenTTD; git rebase --onto upstream/master origin/master origin/<your branch>. This should fix most issues for most people. If not, drop by on IRC and let us help you!

In closing, OpenTTD has been doing everything "on its own" since 2004. For the longest time this was needed as nobody could supply what we were looking for. But the world has caught up; GitHub supplies us with everything we were looking for. It is a change, a huge change, but we truly believe it will lead to a better place; both for OpenTTD and all its fans, but also for all the developers out there.

If you are interesting in helping out finishing this migration, please step forward (get to IRC, and start talking). Mainly if you like Docker, if you are interested in cross-compiling, know how to use MSBuild from PowerShell, like writing documentation (e.g. CONTRIBUTING.md or README.md), enjoy cleaning up issues, know how to triage issues, etc.

And interested in helping out with OpenTTD? Go to our Issue Tracker, click on the "good first issue" label, and get going! Drop by on IRC any time for additional help and/or hints.

TL;DR
: Put up or shut up. Things are changing. Get moving. Or to leave it in the words of andythenorth:
We have moved the OpenTTD source to GitHub.

It was probably overdue.

Things will be better.

We tried not to kill any kittens in the process. Promise.
The only thing necessary for the triumph of evil is for good men to do nothing.

User avatar
Digitalfox
Chief Executive
Chief Executive
Posts: 704
Joined: 28 Oct 2004 04:42
Location: Catch the Fox if you can... 15 years and counting!

Re: OpenTTD source migration and other changes

Post by Digitalfox » 15 Apr 2018 03:05

Too all involved, thank you :bow:

Juanjo
Engineer
Engineer
Posts: 124
Joined: 06 Aug 2011 15:51
Location: Spain

Re: OpenTTD source migration and other changes

Post by Juanjo » 15 Apr 2018 09:21

I will have to catch up with the recent changes, but I really like the roadmap you have decided on. I miss the numbered revisions (r00001...) though.

Thumbs up for giving developers a way to ease the creation of binaries for distribution. That must be still under development, but where they will be stored? Which users and how will they trigger the creation of such binaries?

About the saveload versions with different branches, I am interested about how you decide to solve it, because that is an annoying issue.

Thank you for all your work.

User avatar
Telk
Engineer
Engineer
Posts: 60
Joined: 05 May 2008 14:04
Location: Seoul, South Korea
Contact:

Re: OpenTTD source migration and other changes

Post by Telk » 15 Apr 2018 09:33

Finally! It will be more familiar to me to reporting bugs.
Additionally, should I pull request when I have to change my language's translation via github (like OpenRCT2), or keep using WebTranslator?
(Of course I think there is rarely additional stuff to translate.)
Image
- OpenTTD Korean Translator
- TELKLAND site manager, the korean website of OpenRCT2/OpenTTD (https://telk.kr)

frosch
OpenTTD Developer
OpenTTD Developer
Posts: 979
Joined: 20 Dec 2006 13:31
Location: Aschaffenburg

Re: OpenTTD source migration and other changes

Post by frosch » 15 Apr 2018 15:41

Telk wrote:Additionally, should I pull request when I have to change my language's translation via github (like OpenRCT2), or keep using WebTranslator?
We just reenabled the WebTranslator/Eints, so translators can continue as before.
Eints validates the translations and then pushes them every day.

PRs are the way for adding completely new languages, but we do not want to handle PRs for updates to a few strings in languages we do not understand :)
Juanjo wrote:About the saveload versions with different branches, I am interested about how you decide to solve it, because that is an annoying issue.
I am not sure whether my link refers to the newest version, but something in the direction of:
https://github.com/JGRennison/OpenTTD-p ... ver_sl.cpp
But one could also go towards self-describing structures and map array. (read: store descriptions of the binary structures in the savegame, including string-identifiers for the fields, types and sizes).
⢇⡸⢸⠢⡇⡇⢎⡁⢎⡱⢸⡱⢸⣭⠀⢸⢜⢸⢸⣀⢸⣀⢸⣭⢸⡱⠀⢰⠭⡆⣫⠰⣉⢸⢸⠀⢰⠭⡆⡯⡆⢹⠁⠀⢐⠰⡁

Baldy's Boss
Tycoon
Tycoon
Posts: 1308
Joined: 23 Feb 2014 22:02

Re: OpenTTD source migration and other changes

Post by Baldy's Boss » 16 Apr 2018 19:00

I'm a FreeBSD user who fairly recently migrated TO Subversion for tracking ports,and there it's considered the new thing compared to CVSup.
Not the same in application development world?

peter1138
OpenTTD Developer
OpenTTD Developer
Posts: 1646
Joined: 30 Mar 2005 09:43

Re: OpenTTD source migration and other changes

Post by peter1138 » 17 Apr 2018 08:38

BSD is a bit weird like that. Nobody else has been using CVS for decades. I don't think many people have been using SVN for new projects for several years either.
He's like, some kind of OpenTTD developer.

Baldy's Boss
Tycoon
Tycoon
Posts: 1308
Joined: 23 Feb 2014 22:02

Re: OpenTTD source migration and other changes

Post by Baldy's Boss » 17 Apr 2018 16:44

peter1138 wrote:BSD is a bit weird like that. Nobody else has been using CVS for decades. I don't think many people have been using SVN for new projects for several years either.
https://wiki.freebsd.org/CvsIsDeprecated

but I only switched from portsnap to svn in late 2014 and from portmaster to Synth after that (Synth creates a local package repository),

TrueBrain
OpenTTD Developer
OpenTTD Developer
Posts: 1312
Joined: 31 May 2004 09:21

Re: OpenTTD source migration and other changes

Post by TrueBrain » 17 Apr 2018 18:00

Baldy's Boss wrote:I'm a FreeBSD user who fairly recently migrated TO Subversion for tracking ports,and there it's considered the new thing compared to CVSup.
Not the same in application development world?
So if I can make a prediction, in 14 years FreeBSD moves to git :D They are a bit lagging behind it seems ;)

CVS .. that is a long long long long time ago :)

But so yeah, the same, just delayed with a few years :) Most likely for very valid and good reasons. svn2git tools are very good too, so meh :)

PS: 2012 is no longer "recent" .. ;)

Okay, I take my comment back. They migrated in 2012 to SVN. Seems they are not lagging behind that much. Maybe in 2 years to git? :)
The only thing necessary for the triumph of evil is for good men to do nothing.

User avatar
Pyoro
Tycoon
Tycoon
Posts: 2465
Joined: 17 Oct 2008 12:17
Location: Virgo Supercluster

Re: OpenTTD source migration and other changes

Post by Pyoro » 19 Apr 2018 12:27

Interesting stuff. Thanks for the hard wörk :)

User avatar
Zephyris
Tycoon
Tycoon
Posts: 2826
Joined: 16 May 2007 16:59

Re: OpenTTD source migration and other changes

Post by Zephyris » 20 Apr 2018 10:46

Big news! Definitely sounds like a good long-term plan.

Baldy's Boss
Tycoon
Tycoon
Posts: 1308
Joined: 23 Feb 2014 22:02

Re: OpenTTD source migration and other changes

Post by Baldy's Boss » 20 Apr 2018 17:34

TrueBrain wrote:
Baldy's Boss wrote:I'm a FreeBSD user who fairly recently migrated TO Subversion for tracking ports,and there it's considered the new thing compared to CVSup.
Not the same in application development world?
So if I can make a prediction, in 14 years FreeBSD moves to git :D They are a bit lagging behind it seems ;)

CVS .. that is a long long long long time ago :)

But so yeah, the same, just delayed with a few years :) Most likely for very valid and good reasons. svn2git tools are very good too, so meh :)

PS: 2012 is no longer "recent" .. ;)

Okay, I take my comment back. They migrated in 2012 to SVN. Seems they are not lagging behind that much. Maybe in 2 years to git? :)
Turns out there IS in fact a https://github.com/freebsd for those so inclined but I can't see them outsourcing the central functions.

User avatar
Gymnasiast
Engineer
Engineer
Posts: 4
Joined: 03 Jun 2017 18:34

Re: OpenTTD source migration and other changes

Post by Gymnasiast » 21 Apr 2018 11:13

Ah, great news! I think this could easily lead to more outside contributions, since this lowers the bar a great deal. :D

KeldorKatarn
Transport Coordinator
Transport Coordinator
Posts: 274
Joined: 13 Apr 2010 21:31

Re: OpenTTD source migration and other changes

Post by KeldorKatarn » 23 Apr 2018 13:31

With that compile farm in place I MIGHT actually try to make my patch pack more compatible with other platforms. Having infrastructure in place that I can simply use will help a lot with that. Making the savegames compatible however will be a nightmare I think. Maybe JGR can help me integrate his system somehow, but it still won't be super easy. Sometimes we have the same feature but they use flags in different enum locations and such or the feature may have been introduce slightly differently so it's actually not the same feature at all.

I personally am not super optimistic about achieving savegame compatibility between patch packs but we'll see.

User avatar
JGR
Tycoon
Tycoon
Posts: 1935
Joined: 08 Aug 2005 13:46
Location: Ipswich

Re: OpenTTD source migration and other changes

Post by JGR » 23 Apr 2018 20:17

KeldorKatarn wrote:I personally am not super optimistic about achieving savegame compatibility between patch packs but we'll see.
It's quite possible, my patchpack can load several different variants of SpringPP savegames.
It is however a non-trivial amount of effort to get right and requires having near-complete feature coverage. Many key features cannot be gracefully ignored on load if the version doing the loading doesn't identically support a feature used in the savegame.
KeldorKatarn wrote:...but they use flags in different enum locations and such or the feature may have been introduce slightly differently so it's actually not the same feature at all.
afterload.cpp is full of these sorts of fixups and bit shuffling, both in trunk and in some of my additions.
I wouldn't worry too much about those.
Ex TTDPatch Coder, Grumpy Greymuzzle
Avatar by MoonsongWolf.
Patch Pack, Github
Dad-Coder since April 2018

User avatar
George
Tycoon
Tycoon
Posts: 4306
Joined: 16 Apr 2003 16:09
Skype: george-vb
Location: SPb, Russia.
Contact:

Re: OpenTTD source migration and other changes

Post by George » 16 Sep 2018 08:07

After a brief search, I could not find a place, where I can download OTTD trunk versions after R28004 released on 2018-04-06
What is a new place for them?
Image Image Image Image

Eddi
Tycoon
Tycoon
Posts: 7365
Joined: 17 Jan 2007 00:14

Re: OpenTTD source migration and other changes

Post by Eddi » 17 Sep 2018 09:03

it's not called "trunk" anymore, it's now called "master" and can be found on https://github.openttd.org

also, there's no revision numbers anymore.

nightly versions are currently unavailable, until the compile infrastructure is updated.
You might not exactly be interested in Ferion, but if you are, have fun :)

User avatar
George
Tycoon
Tycoon
Posts: 4306
Joined: 16 Apr 2003 16:09
Skype: george-vb
Location: SPb, Russia.
Contact:

Re: OpenTTD source migration and other changes

Post by George » 19 Sep 2018 17:10

Eddi wrote:it's not called "trunk" anymore, it's now called "master" and can be found on https://github.openttd.org
also, there's no revision numbers anymore.
nightly versions are currently unavailable, until the compile infrastructure is updated.
I cant find any precompiled windows versions there
Image Image Image Image

Eddi
Tycoon
Tycoon
Posts: 7365
Joined: 17 Jan 2007 00:14

Re: OpenTTD source migration and other changes

Post by Eddi » 19 Sep 2018 22:40

maybe you should slap a sysadmin into submission? just because we're not paying him doesn't mean he doesn't have to work.
You might not exactly be interested in Ferion, but if you are, have fun :)

User avatar
George
Tycoon
Tycoon
Posts: 4306
Joined: 16 Apr 2003 16:09
Skype: george-vb
Location: SPb, Russia.
Contact:

Re: OpenTTD source migration and other changes

Post by George » 20 Sep 2018 05:50

Eddi wrote:maybe you should slap a sysadmin into submission? just because we're not paying him doesn't mean he doesn't have to work.
Do you mean that there is no CF, so everyone has to compile himself?
Image Image Image Image

Post Reply

Return to “General OpenTTD”

Who is online

Users browsing this forum: No registered users and 1 guest