OpenTTD version control systems

[ic111]

Hello,

ok, seems that you guys are reinventing your version control systems, don´t you?

At least:
- At https://www.openttd.org/en/development, under "Obtaining the source code" only github is mentioned any longer
- Google still shows me the old text with the hg path
- Under https://wiki.openttd.org/Community, still the hg and svn paths are mentioned
- Both redirect to github
- Under github, "Clone or Download" states I can user git or svn
- In the development forum, the topic with the link to the ICQ archives has disappeared, and to be honest, I don´t know it; this prevents me from trying to read in the chat archives about what´s going up here
- 28 days ago, Frosch committed svn r27988 which edits in particular .hgignore, so at least a month ago something about hg was still there

So:
- Is hg / mercurial queues (which you more or less forced me to use years ago) discontinued or not?
- Is there a way to still use existing patch queues (i.e. mercurial queues)?
- What is the workflow to fetch the current trunk source code and apply an existing patch queue on top of it?

To be honest, at first glance I am not amused about what I see here. Especially, because apparently everything has changed, but neither in the Forum, nor on the Development page is a clear hint / description for the transformation from old to new world. Or did I miss some important link?

(... or (question to myself) should I now really start putting my patches into an own repository, as you will not accept them anyway because changes bigger than tiny changes don´t have chances to be accepted anyway? At least, migration from mq to git seems quite a lot of work (learn a completely new workflow) given that I only occasionally spent limited amounts of time for this, and code won´t be accepted anyway...)

[andythenorth]

Bear with us.

* the move is to github
* some details are still being worked out
* there is a lot of effort going on behind the scenes (visible in irc though)
* this is the most time and effort spent on OpenTTD infrastructure for a long time, and it's really important

The intention is to make OpenTTD development sustainable in the long run.

irc logs are at http://webster.openttdcoop.org/?channel=openttd

[HackaLittleBit]

To be honest, at first glance I am not amused about what I see here. Especially, because apparently everything has changed, but neither in the Forum, nor on the Development page is a clear hint / description for the transformation from old to new world. Or did I miss some important link?

That is an understatement.
I always thought that a forum is the place to make important decisions.
IRC is for chit chat!

[andythenorth]

I always thought that a forum is the place to make important decisions.
IRC is for chit chat!

That's historical at best . Most of the currently active devs use the forums very little, or not at all. There are various reasons for this, but I don't want to speak on behalf of other people.

Facts on the ground are that #openttd irc is currently the primary channel for discussion about the project

We'll need to update the related wiki pages, e.g. https://wiki.openttd.org/FAQ_development

[HackaLittleBit]

I am sad to hear this.

One of the cornerstones of any successful enterprise is good communication. I doubt that this is achievable with IRC.

[JGR]

I am sad to hear this.

One of the cornerstones of any successful enterprise is good communication. I doubt that this is achievable with IRC.

It can be achieved using IRC. Simple things like scheduling important meetings in advance and publishing minutes/a summary can make things more manageable.
Most people can't or won't sit in a channel for ~12 hours a day hoping to catch an interesting meeting taking place.

From my point of view, moving to git/github and modernising the build infrastructure is a reasonable move.
However throwing away the existing git repo isn't necessary and seemed odd to me, as any new changes could be committed on top of it in the usual way.
Using a separate history creates a fair amount of make-work for existing forks/branches (e.g. my one), and sort of defeats the point of having immutable public history.
I don't have time at the moment but at some point I will see what the most pragmatic option for merging/etc is as regards my repo (also on github).

[ic111]

That's historical at best . Most of the currently active devs use the forums very little, or not at all. There are various reasons for this, but I don't want to speak on behalf of other people.

Facts on the ground are that #openttd irc is currently the primary channel for discussion about the project

The problem with a chat is, that you have to join it every day to keep up to date. So: I think it is perfectly ok to discuss such things in a chat, but for those who regard themself part of the project, but do not have the time to join a chat every day, at least a short notice in the forum (where one sees at a glance which topics exist, which isn't the case in a chat protocol, where I have to read through pages of discussions about whatever, especially off-topic things) would be quite nice. Something like "I want to reinvent the wheel because rectangular wheels look nicer, lets discuss this in chat next week".

So, I might feel as being someone trying to improve this project by developing things, but if you only take people seriously that invest some hours every day in the chat then you simply kick people like me out of the project.

E.g. developing the rivers patch took me, say, 2 or 3 hours a day over several weeks. But this was only possible because other things in life weren't that time-consuming at that time. In other months, I don't have the time to think about OpenTTD at all - currently I have about 300 hours overtime in real life job...

Reading the chat of today makes quite clear that with such time constraints, you cannot be part of this project. Let me at least answer two posts:

what amazes me about those post ... who "forced" them in hg queues ..

I was quite clearly told that splitting things up into very small chunks is the only way to get accepted, and that doing so by using hg queues is by far the best way.

I really don't liked maintining those small chunks, and for two of the three major patch queues I wrote, in the end it was senseless effort anyway, as the technical base for doing so is now discontinued.

LordAro: 'devs are mean' is probably perennial, there are those who do, and those who complain, and those who can do politics and those who can't

Well, I have spent quite a lot of time developing patch queues, but spending the additional hours per day to "do politics"? No way. I have a real life. If sensefully being part of a project is only possible by chatting all the time, then that's it - not my world. The day has 24 hours, not more

[andythenorth]

LordAro: 'devs are mean' is probably perennial, there are those who do, and those who complain, and those who can do politics and those who can't

To avoid misunderstandings, that quote was me replying to LordAro, rather than LordAro typing

[wallyweb]

The main advantage of chat is that current items are discussed in real time. The big disadvantage is that there are often several discussions all happening at the same time making it very difficult to follow a thread. And then there are slow readers/typers such as myself.

[Alberth]

"Real time" in irc is very stretchy. Indeed you can pretty much have a normal conversation, if both are sitting behind a keyboard. On the other hand, anyone can and does walk out of the conversation at any time, and reply when he/she is back, which can vary between minutes and hours.

[TrueBrain]

DISCLAIMER: all opinions are those of my own, and might or might not be the opinion of others; I am not speaking on behalf of OpenTTD.

I never reply in topics like this; I always have an issue putting the words as such that no party is insulted Let's see how I do today (be gentle!)

So, first of, I really REALLY appreciate all the patches and patchpacks. I think they are awesome (always have, including in the time of MiniIN), as it shows people want different things .. people should be able to pick what they enjoy in a game, not what some few have said the game should be like. Just my opinion. And just to set the mood. I like people who work on games, and make patches, and try to make things better and more pretty and more playable, and more enjoyable, etc etc etc.

Second, I took the time to write this. Please have the decency to read it. Please don't cherry-pick. It is a message in its full, not in its parts.

Hello,

ok, seems that you guys are reinventing your version control systems, don´t you?
(..)

Just so you know, I cannot read on the forums if this is meant aggressive or not. That makes it a bit hard for me to understand where you at. Remember we are all people here, so if it was meant aggressively, please don't. It sets a mood. If you meant it as a honest question: good! I can deal with that

In basics terms, but this is more something for frosch123 to comment on, we are finally getting ride of an ancient VCS, and replacing it with something all the big boys have been for years. Any movement is better than no movement. So yeah, that makes me rather happy There is a much deeper story why such move is needed for a project like OpenTTD. But that is not for me to tell; as I would just be repeating others

What might be nice to know, years ago (almost 10 years, I think .. pfew, getting old here), I create an hg and git mirror, as back then it was nearly impossible to say which was going to win. Both had pluses and minuses. These for a long time were far from official, but, as it goes with these things, they sneaked into becoming official mirrors. They were mainly setup for people to make their patches via, and publish it so they could be accepted via subversion. For a really long time I expected Mercurial to win, as it had the Windows users on board. But .. time tells a different story. Just as a bit of background for you. I know, not helping you at all, but I just wanted to say it.

I can fully understand that this comes as a bit of a shock to you. I said this on IRC a few times, I fully understand where you are coming from. They way you word yourself, not sure that is the most tactful, but I get you. But what I would like to ask you, is to look at the bigger picture. In 1 year OpenTTD had a messily 300 commits. Most of them Translation commits. Something has to change. Possibly (hopefully?) this is that change. Will it work? Who knows. But any movement better than no movement

I especially understand frustration if you are finally used to mercurial. Sadly, it seems that mercurial is losing the race. Github is for sure a big part of that reason. Knowing git these days is very very valuable in Open Source. But .. everything new is difficult. The good thing is: this time OpenTTD didn't invent their own thing, but used something MANY projects are using. So there is a big help page called the internet (and of course, feel free to ask anything you like to ask .. but sadly, IRC is the best medium for that).

The good thing is, you used Mercurial Queues. Those are easy to port to Git. It is basically just committing them one by one in a branch, and you are done. From all the migrations I have seen/helped with over the weekend, I think you have it easiest. It is just the learning of a new VCS ... but stick with us there; things might become a lot easier because of that

One of the main things that annoyed the hell out of me in relation to OpenTTD and VCSes .. the review system. Manual patches and giving comments on that .. that is so 2004. That really was not helping anyone. Now with Pull Requests, comments are in the patch where it goes wrong. Hopefully that makes accepting patches easier, as it is more visible what is wrong, and it can be done on a much nicer pace. But I am rambling here

The other cool thing about GitHub: you WILL be setting up your own repository (by forking ours). This means even if OpenTTD does not accept your patches (to be perfectly clear: I am not saying yes or no with any weight to one or the other; just stating two paths), and a lot of people like your changes and download your version ... that too is a clear message. "Owh, but how are they going to download my version?!", you ask sarcastically (not saying you, ic111, are, just trying to be funny; sorry if it offends anyone), well .. there I, as SysOp, can help with. But we can talk about that in a later stage, if you give me the time. The plans are there .. execution takes a bit of time.

All in all: I get you. But please bear with us.




Than the subject of communication channels ... well .. I don't know. IRC always has been the primary for me, ever since I started on OpenTTD in 2004. Always used IRC first, and if I felt the need, tt-forums second. But this migration was not my party, so I am not involved in communications (I just made it technical possible and execute it). So I leave this touchy subject for others


Did I ever mention I am Dutch? I tend to be blunt .. don't take it personally. It isn't.

(..)
From my point of view, moving to git/github and modernising the build infrastructure is a reasonable move.(..)

Honestly, this should be a very welcome change for you, as patchpack owner. It is more than "reasonable". It fixes a lot of issues patchpacks have. But that it just my experience with VCSes speaking here

(..)
However throwing away the existing git repo isn't necessary and seemed odd to me, as any new changes could be committed on top of it in the usual way.
(..)

These remarks always annoy me. People rarely ask: why did you do that? They just assume you are stupid or something and jump to all kind of conclusions. It always makes me a bit nasty .. which I really don't want to. So please, PLEASE, can you just ask WHY we went for this solution, and not something that didn't break all forks? I am pretty sure we have a darn good reason for that. Give us the benefit of the doubt. PLEASE!

Lucky enough, the amount of work involved is very minimal, if your patches were already PullRequest ready. I have noticed over the weekend that not all git-forks were like that .. meaning a lot of merges in there, which don't always play nice. But, git is awesome, and so far we managed to port in relative ease all forks. Even the 229 commits NRT has .. they only have to fix ~20 commits, of which most are binary conflicts which had to be resolved anyway.

If you want, I can help you out here. But to put it simple: git has a command 'git rebase --onto'. It goes something like this:

Code: Select all

git clone https://github.com/OpenTTD/OpenTTD
cd OpenTTD
git remote add deprecated <your fork>
git rebase --onto origin/master deprecated/master deprecated/<yourbranch>

Few assumptions here: branches are up-to-date on your own fork, and they are rebaseable. The latter turns out to not always be the case, but even than resolving the conflicts is minutes work, not days. Hope that helps! (git is awesome for s*** like this). And here too goes: ask for help, please don't nag everything sucks (in which-ever wording). I have been helping people migrate left and right from what-ever VCS source they came all weekend; not planning to stop now. Just drop by on IRC (sorry ...... ) and ask where you are stuck!

And just to be perfectly clear about the subject: subversion was (still is, up till a few commits ago) the immutable public history, the only truth and source. All others were mirrors with similar data. We always said that, that was always published. But, same as with ic111, I get where you coming from. Just the wording ....




This brings me to the last part of my reply: TRUST! I see in this topic alone (and I know of others) a total lack of trust all sides and directions. This holds as true here as it does in real life: if there is no trust, you fail to communicate with each other. Again, both ways, not pointing any fingers (well, a lot of fingers? ). And sadly, this means some form of politics. If you are lucky at work, you have a manager who takes that all away from you. In a lot (most?) Open Source projects .. this isn't the case. So it takes effort. Some patches in OpenTTD took me weeks of sweettalking to people, doing that one thing they really wanted you to do (even if you think it is complete bulls***), etc. And I fully respect people who say: I don't want to do that! But expect some difficulty in that case. That is the flipside of that coin. And if you are lucky someone likes you enough to take that away from you. But than he is doing the politics for you. As politics is always there. If we want to or not.

I hate politics. People tell me I am good at it, but f*** that s***. Just let me program. Sadly .. that also meant I didn't get any patch in .. first I had to do politics, gain trust, trust the other, communicate, grow, and then .... pffft .. too much work

(..)
So, I might feel as being someone trying to improve this project by developing things, but if you only take people seriously that invest some hours every day in the chat then you simply kick people like me out of the project.

Things haven't changed in 14 years in relation to this. So clearly it doesn't kick people out (and that is a good thing ), but this closes my post perfectly. If you want to be a dev, yes, you have to invest time to chat to people, show them who you are and what you can do. If you want your patches to be accepted .. well .. in the old workflow that was just f*** up annoying. Hopefully a review system like GitHub helps with that. But I am not ready to jump to that conclusion .. please submit some PRs, and we will see how that goes.
To be perfectly clear, I fully accept and respect you don't want to be that involved (politics-wise). That sadly does mean you need to create the trust in another way, as we (as OpenTTD) have to get your trust in another way. OpenTTD is not used to that .. maybe this is the change? Maybe it isn't ...


The thing I have the most difficulty with: the ease people forget you are doing this in your free time. And I am sure that goes for you too (as you feel that time is now robbed away doing some migration you never asked for). I work in IT my whole life, and I speak people mostly when s*** goes wrong (goes for other industries too, but that doesn't change the point). This is very demotivating, and honestly the solo reason I haven't done anything OpenTTD related in a long time. I just get spit out over and over by remarks left and right of not doing "my job" and s*** like that. Please always remember we all do this unpaid. And the older we get, the more those unpaid minutes cost. So please don't forget a compliment left and right; it goes a LONG LONG way Hence I now repeat: I really really like what you all do with your patches and patchpacks And I truly mean that. And I am sure many others with me do. So if there is anything I can do to help with the migration, ask.
(on a sub-note, do you also have this issue with so many people just talking, so little people just doing? Pfffffftt .. let's be the change, instead of planning for it ).


To finish it off, really: any movement is good; you (the reader of this post; any reader) might not like it or agree with it, but standing still is far worse. Please don't be the guy/girl who sits on the sideline saying: NO NO NO, this sucks! Please join the party and try it out! If you tried it and still don't like it: speak up! (please, do speak up). But PLEASE give it a chance.

In Agile Scrum we have this rule: for every new change, give it at least 6 weeks of dedication before saying it sucks. It is an eye-opener. At least, it was for me


Anyway, tl;dr: TrueBrain rambles. Tnx for reading.

DISCLAIMER: all opinions are those of my own, and might or might not be the opinion of others; I am not speaking on behalf of OpenTTD.

[Wolf01]

/me claps

I really welcome the new changes, I see them as a way to open new doors to OpenTTD development.
Even with the first steps on the NRT project (which was the first fork "officially" supported from the devs), seeing it grow and then automatically compiled from the OpenTTD compile farm I was so excited, now with the new changes I think more patchpacks and forks could have their opportunity to be special, maybe in future we could automate it at a level that you just have a fork on github and an account on OTTD system and start to CI, compile, and release your fork without passing from a forum post!

Really, really many thanks to TrueBrain and every one else which work on this!

[ic111]

Hello,

ok, seems that you guys are reinventing your version control systems, don´t you?
(..)

Just so you know, I cannot read on the forums if this is meant aggressive or not. That makes it a bit hard for me to understand where you at. Remember we are all people here, so if it was meant aggressively, please don't. It sets a mood. If you meant it as a honest question: good! I can deal with that

No, this wasn't meant aggressive. It was more like: For the first time after some break, I felt I might have some time for OpenTTD. The obvious first step was: Fetch current source code. Reapply patches. Fix any conflicts. Go ahead with some plans to improve things.

But somehow, the source for "fetch current source code" had disappeared, hg pull only answered with an unintelligible error message. So I started to search what has happened. I found some hints, but no clear statement about what happens (and with which goal!) at all.

And here, that sentence of course contained some criticism: I simply wouldn't have imagined, that a project like OpenTTD starts a replacement of its basic infrastructure without any public notice. In the ideal case, a story like you wrote it in the post I currently answer, what you do and why, if you feel you don't have the time etc. for it, then at least some out-of-order notice (in job, you probably sometimes get messages like "Server X is not available tomorrow due to...").

So, for example, not just replace the repository link by the github link on the development page, but write some sentences about the migration, something like that. Something which someone who doesn't have the time to participate in chat regularily sees at first glance, when visiting homepage, forum, etc.

Yes, if I would read the last 100 to 150 chat protocols, I would probably get the whole picture. But honestly, a chat is for discussion, extracting the decisions from that protocols is probably very difficult (beside the fact that it costs me much more time than just read a summary of the decision), simply because there are 50 percent other topics, 30 percent completely off-topic, and among the remaining 20 percent, there are five different opinions, where one of them won in the end.

I especially understand frustration if you are finally used to mercurial. Sadly, it seems that mercurial is losing the race. Github is for sure a big part of that reason. Knowing git these days is very very valuable in Open Source. But .. everything new is difficult. The good thing is: this time OpenTTD didn't invent their own thing, but used something MANY projects are using. So there is a big help page called the internet (and of course, feel free to ask anything you like to ask .. but sadly, IRC is the best medium for that).

The good thing is, you used Mercurial Queues. Those are easy to port to Git. It is basically just committing them one by one in a branch, and you are done.

Ok, given how important you regarded the small chunks in the last 10 years, at first I feared that it will become some setup with multiple / many branches and difficult relationships / merging between them.

The other cool thing about GitHub: you WILL be setting up your own repository (by forking ours).

And what will (conceptionally) people do, who want to combine multiple of those forks?

BTW, the thing I personally don't regard so nice about Github is that a lot of projects depend on the same, private, company. If Github ever closes down or gets bankrupt, a lot of projects get lost.

To be perfectly clear, I fully accept and respect you don't want to be that involved (politics-wise).

BTW, my argument was more about time, less about politics.

[HackaLittleBit]

I think you have it easiest. It is just the learning of a new VCS

I just downloaded trunk with TortoiseHG
You need to use Hg-Git to do so.

Nice report TrueBrain

later I will comment more.

Thanks.

[TrueBrain]

No, this wasn't meant aggressive. (..)

Cool And fully agree that communication could have been better; guess nobody excepted people to notice this quickly (this is not an excuse)

(..)
And what will (conceptionally) people do, who want to combine multiple of those forks?

BTW, the thing I personally don't regard so nice about Github is that a lot of projects depend on the same, private, company. If Github ever closes down or gets bankrupt, a lot of projects get lost.
(..)

You can make a branch where you merge several other branches. Of course someone needs to resolve any conflicts, but otherwise pretty easy. Branches are just a queue of patches, so .. yeah.

And as we did with our subversion, of course the data on GitHub will be backed up; so in case they go bye-bye, we can always recover But yeah, it is a fair fear

(..)
You need to use Hg-Git to do so.
(..)

Smart!

[HackaLittleBit]

I like the silence.
I quite sure most of the readers would like to say a milion things.
Sometimes however it is better to stay put.
Just read this small article and reflect.(Think!)
It all stands or falls with organisation.

Edit:
Agile Scrum

[peter1138]

BTW, the thing I personally don't regard so nice about Github is that a lot of projects depend on the same, private, company. If Github ever closes down or gets bankrupt, a lot of projects get lost.

This is an advantage of switching to git (not github) as authoritative. Anybody who's cloned the repo has the fully history. Github allows central organisation but doesn't hinder distributed development at all. With the move to github we are no longer reliant on Truebrain to maintain the servers, software, backups, security updates required for the old bug tracker and SVN and other vcs systems. We lost the SVN history once long ago. That's very unlikely to happen with git.

I think you have it easiest. It is just the learning of a new VCS

I just downloaded trunk with TortoiseHG
You need to use Hg-Git to do so.

This works but I think it would be wise to take the time to switch to git instead, it will be easier in the long run.

[HackaLittleBit]

Thanks peter1138