Scribblings and Geekery

Note: The official version of this post can be found at https://www.atlassian.com/git/articles/migrating-svn-git-branching-workflow.

We're moving to Git, and we figured out how to use git-flow and Maven together in an efficient devleopment workflow. Before I get into what our workflow looks like now, it's important to know where we came from.

Back In the Olden Days...

In the our previous world, before migrating from SVN to Git, all of our version management was manual. Development took place on trunk simultaneously across all active features for the team. When a developer committed their changes to SVN, Bamboo would kick off a snapshot build (1.0.0-SNAPSHOT). If their work passed integration testing, they'd run the release plan after manually verifying nobody else had run a subsequent snapshot¹. That release build (1.0.0-1)² would be quickly smoke tested and then handed to QA for functional validation.

We were "releasing" all the time; every build that went to QA was from a Maven release goal without incrementing the minor or patch number. We had Bamboo tack on that -buildnumber to each release so that we could track specific releases to QA,.

Then, once QA blessed the "last one" for that release, we'd increment the minor version using mvn version:set. This meant that if yours truly wasn't completely on top of his game, we ran the risk of forgetting to increase that version number and building a "release" of 1.0.0-x that was after what we had released to production. Big mess. Big pain. But it meant that every build out of development had a clear, trackable, permanent number.

That was "good", but it was a tracking nightmare.

We didn't want to do that anymore. We only wanted to release when something was going to go out the door, customer-ready. Also, no more build numbers in production—major.minor.patch only.

However, we also wanted to ensure QA had a way to track a specific delivery to a specific release. QA couldn't be testing SNAPSHOT releases all the time. Thankfully, JIRA Software makes some of this easy by showing links between JIRA issues, Bitbucket repositories and build plans containing builds for that issue.

In the end, we decided that we would adhere to a few rules:

1. Developers perform integration testing from their feature branch, which they are keeping up to date with the develop branch.
2. Once integration testing is complete and passing, they issue a pull-request and have their code pulled into the develop branch.
3. They promote a QA candidate cut of our common project, and then trigger a QA candidate cut of develop for their product project. Since we need to use a specific common project release at build time³, we prompt a developer for the aforementioned common QA candidate release version.
4. QA tests QA candidate builds only. Never snapshots.

We repeat steps 1-5 for all features, bugs found in QA, etc. Then, once we're all done with a version's efforts, we follow git-flow and branch to a release branch by promoting a specific QA candidate build in Bamboo. This creates our first release candidate set, and QA performs final testing, regression testing, etc.

Building It Right

Getting your builds just right can be tricky. Before we moved everybody off of SVN and our original Bamboo build plans, we set up a POC project and then worked out all of our Bamboo plans using those. In the end, we found the following plans met our needs:

1. develop: Triggered from Bitbucket Server, it creates SNAPSHOT builds with every change pulled in from pull requests. The build plan has three stages:
   1. Snapshot - Builds snapshots and doesn't touch versions.
   2. QA Build - Builds a "QA Candidate" release numbered X.Y.Z-qa{buildnunber} from the same git commit used to create that snapshot.
   3. Promote To Release - Creates the release branch and increments develop's minor version number, all from the same git commit used to create the QA build.
2. feature: Set up with a default build plan, but mostly is watching for new branches created with feature/* in the name, using Bamboo's plan branches feature, and creates a branch plan for that branch whenever it does. It has only one stage, which builds SNAPSHOTs.⁴ Also, because we don't want feature to build by itself, we point its repository to a branch called "fake-branch" so that it never triggers.
3. release and hotfix: These have the same set of build and release steps so they get to share a plan. The plan also points to a "fake-branch" and only cuts branch plan-based releases with the following two stages:
   1. Build Candidate Release - This creates a release numbered X.Y.Z-rc{buildnumber}.
   2. Finalize Release - This sets the final release number, merges up to master, and creates a tag. Then it removes the release branch which disables the plan.
4. support: This is nearly identical to release and hotfix, but it never merges to master. Instead, we increament the minor version at the end of the Finalize Release stage.

Because feature, release_hotfix, and support plans are all running branch plans, when you view them in Bamboo, you see “Never built” for each plan. The first time this happened everyone had a blank look on their face... where are my builds?? but then we realized this made sense.

We have so many build plans⁵ that if bamboo displayed all of the plans inline including branch plans, you'd be scrolling forever and a day. The information overload would actually be less helpful. So we click once more to see our branch plan status. It's not a big deal, but it would be nice to find an easy way to see them all.⁶

Migration Process

Once we were satisfied that the POC project worked the way we wanted it to, and the developer workflow was consistent and reliable, we ran an import of our SVN codebase into Git and ran through several more iterations of testing each possible workflow to iron out build issues, workflow oddities, etc.

We found some things like my note above about cleaning the Maven workspace every time you run a feature branch build in Bamboo, and certain times we needed various flags for a specific maven lifecycle. Generally, these were easy to figure out, but once in awhile there was much shaking of fists and gnashing of teeth.⁷

After all of that, we decided it was actually now or never, and announced our migration date. As a part of the migration, we decide to do a bunch of cleanup so that our Git repo had a nice starting place. We did things like:

• Bulk code re-formatting so that we could enable some stricter style checks as a part of the build process
• Converted all line endings to UNIX
• Squashed three top-level dependency projects into one and refactored all dependent code as a result

These things out of the way, we kicked everybody out of SVN, made it read-only and did a final pull into Git. We conducted a training session with all of our developers to go through our workflow one more time and show it in action with the actual codebase.

It was the smoothest migration I've ever experienced. We froze SVN around 5pm on a Monday and by 10pm we were done with all of our initial build issues worked out. There were no major issues; some things required a lot of waiting. Training was the next morning and we were doing feature work by lunch.

New Developer Flow

Once through the migration, we were able to see how this workflow would work in the real world. When a developer starts work on a feature (ABC-4321), they need to do a few things to get started:

1. From JIRA Software, in the Development area of the issue, click on Create Branch.
2. This opens a screen within Bitbucket Server that lets them select the branch type (usually feature), repository, and branch name. Because of our Maven extension I mentioned in the previous post, the branch name is always the JIRA issue key, no description.
3. Repeat steps 1-2 for each of the associated projects for that feature, always using the single, same issue key.
4. git pull && git checkout origin/feature/ABC-4321 feature/ABC-4321⁸

This workflow is straightforward, repeatable and reliable. Developers can work in isolation and pull in contributed changes from develop as they move forward. The branching action can feel a bit repetitive if, say, a user story has work in all four of our product verticals and the common project. We've been thinking about automating this with some sort of JIRA workflow post-function to call the Bitbucket Server REST API, but that might be overkill for something that isn't costing us too much developer time.

Lessons Learned

This process to get us from SVN to git with a shiny new workflow was a long one — from kickoff to migration we took almost seven months. A vast majority of that time was spent wrestling with maven.

I'll admit that we had some staffing concerns along the way; in parallel to this work the same group of engineers working on the mgiration were supporting a department of 800 people on the Atlassian toolset, providing production support for our platform applications, and working on other operational R&D projects. Once we finally put three people on the migration nearly full-time, we were done in about a month.

Despite all of it, we learned a lot:

1. No amount of preparation truly prepares you for the real thing. For instance, each release build type we tried to perform failed the first time we tried for one reason or another. Once we fixed one project's build for that type, we copied the config to the others and haven't had a single repeat.
2. This workflow generates a lot of builds. So. Many. Builds. We needed to double our Bamboo agents to keep up.
3. This workflow generates a lot of build artifacts. Within the first week we ran out of disk space on our Artifactory instance and had to spend a solid day manually purging old release candidates and QA builds no longer needed. Then we needed to think of a way to ensure that when feature branches are removed, we also remove all of their branch-specific snapshot artifacts.
4. The team doesn't really like needing to pick between a hotfix or having a support branch. It makes sense to be able to cut a hotfix, but most of the time they want a support branch. We might decide to only use hotfixes on special occaisions when the merge would truly be straightforward.
5. The combination of JIRA Software, Bitbucket Server and Bamboo are seriously killer. Watching someone start work in JIRA Software, create a branch and immediately have a branch plan built and ready to validate their work is beautiful.
6. Pull Requests in Bitbucket Server are the greatest thing since sliced bread. Between keeping a push-happy engineer at bay or making sure we're ready for an offshore team's contributions, we couldn't be happier with the pull-request process. Given we perform code inspection in Crucible rather than at pull-request, we're able to use it for quick sanity checks as well.
7. Our previous SVN-based tags had been tagged by a service account that was performing the build. Since that user wasn't real, when we tried to create branches from tags, our git-hook to validate the user was valid for a given commit failed. I wrote an article on my personal blog on how to change the author of a single commit in Git, which came in handy the first time we needed to create a support branch from an old SVN-based tag... which was the day after the migration!

All in all, our migration was a great success. It didn't solve every problem my team has had, but it certainly solved many and gave us a more stable footing to move forward.