Open Feedback

A few weeks ago, Becky Hansmeyer wrote a post about how some of her "favorite people on Twitter seem to be…well, avoiding Twitter…and how that made [me] kinda sad". I wholeheartedly agreed with her original post and was both fascinated and excited (for her, this person I don't know personally) to hear this post be the topic of discussion on this past week's Analog(ue).

I've been ruminating on the concepts that Becky, Mike, and Casey all touched on for a few days. I wanted to share it with them, but I thought... is it my place to do so? I enjoy reading what Becky writes about app development1, and enjoy listening to Mike and Casey talk about their lives. Does that give me any sort of right to tell them what I think about what they wrote/said? I know the latter pair receive a fair amount of feedback. I've sent Casey some regarding his other podcast (ATP), but why was it ok for me to do that?

We live in both a highly-connected, and highly-insular world. The capability of our words to impact people we don't even know is more a reality now than ever before. There can be many positives to this, though it also leads to a logical fallacy: because we can say it, and we live in a place where "Congress shall make no law abridging" us from having our God-given right to an opinion, we therefore believe that we have the right to broadcast any opinion, no matter how completely uninformed, disrespectful, unkind, etc. We think that our opinion about a topic we just got all rage-y on today is as valid to be consumed by the masses as the critic, lawmaker, and victim who have first-hand experience with that issue and may have for far longer than we could imagine. Our perceived right broadcast our indignance leads to a world full of angry people, many of whom don't have a clue from where their rage truly originates.

And the culture uses this to its benefit all the while creating victims of groupthink rage. The mob mentality takes over before anyone can reasonably ask themselves "is it right for me to be angry?"2 .

Sometimes it is right. Most of the time, though, we're spectators in a globally-viewable personal conversation, weighing in on someone else's business or throwing trash on their lawn.

(And some of the time it's just the election season.)

Becky wrote a follow-up post and hit the content I wanted to touch on squarely on the head. It made me feel like it was 2004 all over again, where conversations between personal content creators (cough bloggers cough) was managed through trackback links and comment threads, pingomatic and technorati. Content was harder to find, and it was harder to randomly stumble into rage -- you either had to explicitly leave a comment with your email address attached, or wrote your own darn blog post to respond to someone else's ideas.

Now all you have to do is fire off a potentially-anonymous tweet, and that's why I don't blame folks who want to take their content into walled gardens instead of the public square.

But what about me and my thoughts? Should anyone listen to what I have to say in response to their opinions which they shared publicly? Nope. I mean... they can if they want, but something has changed in our culture that we expect they should, and so we send unfiltered responses without questioning if it's our place to do so.

And if I'm completely honest, some part of me wants to anyway despite knowing all of this. I want to be able to send criticism and be told I'm right because on one level I'm self-centered and consider myself important.

Instead I'm going to throw back to 2004 and be perfectly content with what I have here.


  1. And, let’s be honest: her dog is adorable. More corgis! Corgis for everyone!

  2. Jonah 4:9

Mar 15th, 2016

Frederick Douglass Daguerreotype

My father-in-law sent us a video about one of his long-ongoing research projects, the study of an early Daguerreotype of Frederick Douglass.1 For a nerd with any sort of science leaning, this is a pretty interesting topic. Every time we visit my wife's home, he has some new story about this project with twists, turns, "plot thickens" moments, etc. It's great to see the attention start to ramp up a bit!

I'm really quite proud to be related to this guy. He does some pretty cool work in the nano scale up at the UofR.


  1. In the video, he’s the guy doing all of the work on the microscope, pointing out findings, etc.

Jan 7th, 2016

Creative Branching Models for Multiple Release Streams

A couple of weeks ago, I had the incredible opportunity to speak at the 2015 Atlassian Summit in San Francisco, CA. The conference lasts about three days and covers a wide range of topics from Software Engineering to Process Improvement to Team Dynamics to Enhancing Communication and so on. Most of the sessions are focused on using Atlassian's tools to accomplish a given goal, but many are generally applicable to anyone in a Software Engineering field.

My presentation covered my team's migration from Subversion to Git with a long time spent talking about the work we needed to do to keep our multi-module build setup in Maven whilst using git-flow as a branching model and making our engineers do as few manual steps as possible. I was limited to about 30 minutes, so I wrote a brief series of posts to cover everything I couldn't say on stage:

  1. Git or SVN? How Nuance Healthcare chose a Git branching model)
  2. Dealing with Maven dependencies when switching to Git
  3. Pulling the Trigger: Migrating from SVN to Git

Video

Slides

This was my first time presenting at a technical conference. It was a great experience that I hope to repeat in the future.

Nov 17th, 2015

Pulling the Trigger: Migrating from SVN to Git

Note: The official version of this post can be found at https://www.atlassian.com/git/articles/migrating-svn-git-branching-workflow.

We're moving to Git, and we figured out how to use git-flow and Maven together in an efficient devleopment workflow. Before I get into what our workflow looks like now, it's important to know where we came from.

Back In the Olden Days...

In the our previous world, before migrating from SVN to Git, all of our version management was manual. Development took place on trunk simultaneously across all active features for the team. When a developer committed their changes to SVN, Bamboo would kick off a snapshot build (1.0.0-SNAPSHOT). If their work passed integration testing, they'd run the release plan after manually verifying nobody else had run a subsequent snapshot1. That release build (1.0.0-1)2 would be quickly smoke tested and then handed to QA for functional validation.

We were "releasing" all the time; every build that went to QA was from a Maven release goal without incrementing the minor or patch number. We had Bamboo tack on that -buildnumber to each release so that we could track specific releases to QA,.

Then, once QA blessed the "last one" for that release, we'd increment the minor version using mvn version:set. This meant that if yours truly wasn't completely on top of his game, we ran the risk of forgetting to increase that version number and building a "release" of 1.0.0-x that was after what we had released to production. Big mess. Big pain. But it meant that every build out of development had a clear, trackable, permanent number.

That was "good", but it was a tracking nightmare.

We didn't want to do that anymore. We only wanted to release when something was going to go out the door, customer-ready. Also, no more build numbers in production—major.minor.patch only.

However, we also wanted to ensure QA had a way to track a specific delivery to a specific release. QA couldn't be testing SNAPSHOT releases all the time. Thankfully, JIRA Software makes some of this easy by showing links between JIRA issues, Bitbucket repositories and build plans containing builds for that issue.

In the end, we decided that we would adhere to a few rules:

  1. Developers perform integration testing from their feature branch, which they are keeping up to date with the develop branch.
  2. Once integration testing is complete and passing, they issue a pull-request and have their code pulled into the develop branch.
  3. They promote a QA candidate cut of our common project, and then trigger a QA candidate cut of develop for their product project. Since we need to use a specific common project release at build time3, we prompt a developer for the aforementioned common QA candidate release version.
  4. QA tests QA candidate builds only. Never snapshots.

We repeat steps 1-5 for all features, bugs found in QA, etc. Then, once we're all done with a version's efforts, we follow git-flow and branch to a release branch by promoting a specific QA candidate build in Bamboo. This creates our first release candidate set, and QA performs final testing, regression testing, etc.

Building It Right

Getting your builds just right can be tricky. Before we moved everybody off of SVN and our original Bamboo build plans, we set up a POC project and then worked out all of our Bamboo plans using those. In the end, we found the following plans met our needs:

  1. develop: Triggered from Bitbucket Server, it creates SNAPSHOT builds with every change pulled in from pull requests. The build plan has three stages:
    1. Snapshot - Builds snapshots and doesn't touch versions.
    2. QA Build - Builds a "QA Candidate" release numbered X.Y.Z-qa{buildnunber} from the same git commit used to create that snapshot.
    3. Promote To Release - Creates the release branch and increments develop's minor version number, all from the same git commit used to create the QA build.
  2. feature: Set up with a default build plan, but mostly is watching for new branches created with feature/* in the name, using Bamboo's plan branches feature, and creates a branch plan for that branch whenever it does. It has only one stage, which builds SNAPSHOTs.4 Also, because we don't want feature to build by itself, we point its repository to a branch called "fake-branch" so that it never triggers.
  3. release and hotfix: These have the same set of build and release steps so they get to share a plan. The plan also points to a "fake-branch" and only cuts branch plan-based releases with the following two stages:
    1. Build Candidate Release - This creates a release numbered X.Y.Z-rc{buildnumber}.
    2. Finalize Release - This sets the final release number, merges up to master, and creates a tag. Then it removes the release branch which disables the plan.
  4. support: This is nearly identical to release and hotfix, but it never merges to master. Instead, we increament the minor version at the end of the Finalize Release stage.

Because feature, release_hotfix, and support plans are all running branch plans, when you view them in Bamboo, you see “Never built” for each plan. The first time this happened everyone had a blank look on their face... where are my builds?? but then we realized this made sense.

We have so many build plans5 that if bamboo displayed all of the plans inline including branch plans, you'd be scrolling forever and a day. The information overload would actually be less helpful. So we click once more to see our branch plan status. It's not a big deal, but it would be nice to find an easy way to see them all.6

Migration Process

Once we were satisfied that the POC project worked the way we wanted it to, and the developer workflow was consistent and reliable, we ran an import of our SVN codebase into Git and ran through several more iterations of testing each possible workflow to iron out build issues, workflow oddities, etc.

We found some things like my note above about cleaning the Maven workspace every time you run a feature branch build in Bamboo, and certain times we needed various flags for a specific maven lifecycle. Generally, these were easy to figure out, but once in awhile there was much shaking of fists and gnashing of teeth.7

After all of that, we decided it was actually now or never, and announced our migration date. As a part of the migration, we decide to do a bunch of cleanup so that our Git repo had a nice starting place. We did things like:

  • Bulk code re-formatting so that we could enable some stricter style checks as a part of the build process
  • Converted all line endings to UNIX
  • Squashed three top-level dependency projects into one and refactored all dependent code as a result

These things out of the way, we kicked everybody out of SVN, made it read-only and did a final pull into Git. We conducted a training session with all of our developers to go through our workflow one more time and show it in action with the actual codebase.

It was the smoothest migration I've ever experienced. We froze SVN around 5pm on a Monday and by 10pm we were done with all of our initial build issues worked out. There were no major issues; some things required a lot of waiting. Training was the next morning and we were doing feature work by lunch.

New Developer Flow

Once through the migration, we were able to see how this workflow would work in the real world. When a developer starts work on a feature (ABC-4321), they need to do a few things to get started:

  1. From JIRA Software, in the Development area of the issue, click on Create Branch.
  2. This opens a screen within Bitbucket Server that lets them select the branch type (usually feature), repository, and branch name. Because of our Maven extension I mentioned in the previous post, the branch name is always the JIRA issue key, no description.
  3. Repeat steps 1-2 for each of the associated projects for that feature, always using the single, same issue key.
  4. git pull && git checkout origin/feature/ABC-4321 feature/ABC-43218

This workflow is straightforward, repeatable and reliable. Developers can work in isolation and pull in contributed changes from develop as they move forward. The branching action can feel a bit repetitive if, say, a user story has work in all four of our product verticals and the common project. We've been thinking about automating this with some sort of JIRA workflow post-function to call the Bitbucket Server REST API, but that might be overkill for something that isn't costing us too much developer time.

Lessons Learned

This process to get us from SVN to git with a shiny new workflow was a long one — from kickoff to migration we took almost seven months. A vast majority of that time was spent wrestling with maven.

I'll admit that we had some staffing concerns along the way; in parallel to this work the same group of engineers working on the mgiration were supporting a department of 800 people on the Atlassian toolset, providing production support for our platform applications, and working on other operational R&D projects. Once we finally put three people on the migration nearly full-time, we were done in about a month.

Despite all of it, we learned a lot:

  1. No amount of preparation truly prepares you for the real thing. For instance, each release build type we tried to perform failed the first time we tried for one reason or another. Once we fixed one project's build for that type, we copied the config to the others and haven't had a single repeat.
  2. This workflow generates a lot of builds. So. Many. Builds. We needed to double our Bamboo agents to keep up.
  3. This workflow generates a lot of build artifacts. Within the first week we ran out of disk space on our Artifactory instance and had to spend a solid day manually purging old release candidates and QA builds no longer needed. Then we needed to think of a way to ensure that when feature branches are removed, we also remove all of their branch-specific snapshot artifacts.
  4. The team doesn't really like needing to pick between a hotfix or having a support branch. It makes sense to be able to cut a hotfix, but most of the time they want a support branch. We might decide to only use hotfixes on special occaisions when the merge would truly be straightforward.
  5. The combination of JIRA Software, Bitbucket Server and Bamboo are seriously killer. Watching someone start work in JIRA Software, create a branch and immediately have a branch plan built and ready to validate their work is beautiful.
  6. Pull Requests in Bitbucket Server are the greatest thing since sliced bread. Between keeping a push-happy engineer at bay or making sure we're ready for an offshore team's contributions, we couldn't be happier with the pull-request process. Given we perform code inspection in Crucible rather than at pull-request, we're able to use it for quick sanity checks as well.
  7. Our previous SVN-based tags had been tagged by a service account that was performing the build. Since that user wasn't real, when we tried to create branches from tags, our git-hook to validate the user was valid for a given commit failed. I wrote an article on my personal blog on how to change the author of a single commit in Git, which came in handy the first time we needed to create a support branch from an old SVN-based tag... which was the day after the migration!

All in all, our migration was a great success. It didn't solve every problem my team has had, but it certainly solved many and gave us a more stable footing to move forward.


  1. An uncomfortable amount of HipChat conversations in our Developers room went to asking if anyone needed to commit any changes before a release build was made. Prior to HipChat: Lync, email, or yelling over a cube wall.

  2. The -1 is an incrementing build number for the plan that never resets to 1. It didn’t take too long for release builds to have numbers in the high 100s.

  3. The POM still refers to a -SNAPSHOT release of common here and can’t risk the build pulling the wrong SNAPSHOT.

  4. It also forcibly cleans up the workspace every time it builds. YMMV, but we found this to be necessary.

  5. Across the org we have something like 120 plans in this particular Bamboo instance, growing all the time. Filtered just to my team’s plans, we’re roughly half of that list. With all of our branches we’re pushing 200.

  6. One of my engineers wrote a greasemonkey script that lets him see all branch plans. It only works if you aren’t a Bamboo admin due to the number of visible plans. I’m working on whipping up a dedicated AtlasBoard for myself.

  7. We only burned maven in effigy once or twice.

  8. The exact steps here vary depending on if the developer is using the command line or using Eclipse to switch branches.

Nov 1st, 2015

Dealing with Maven dependencies when switching to Git

Note: The official version of this post can be found at https://www.atlassian.com/git/articles/maven-dependencies-versions-merging.

So we're moving to Git and we like git-flow. Now what? Let's test it all out!

My team is great. They threw together a hit list of developer workflows in Confluence, all based on what we had been doing as a team and all of the weird things they thought we might have to do in the future. Then, in a project structure mirroring our own (but with no code in it—just a pom.xml), tried every workflow.

Maven dependencies were about to prove themselves to be our biggest problem in all of this.

Maven Build Numbering

Maven produces 1.0.0-SNAPSHOT builds until you release. When you release, -SNAPSHOT is removed and your version is 1.0.0. Your build process needs to be able to support incrementing your minor version up after the fact so that subsequent work on your next effort will produce builds like 1.1.0-SNAPSHOT. You're not tied to three digits -- you can define this when you start a project, but we use three. Anyway, the -SNAPSHOT part is really important to understand. This is always going to represent the latest pre-release cut of a project.

Artifacts

See, our big concern in all of these workflows was how we were going to ensure that our project versions and inter-project dependencies were properly managed.

Each time Maven dependencies are retrieved for a build, it will, by default, pull those down from Ye Olde Internet(e)™. Those artifacts are stored locally so that subsequent builds can be performed more quickly. One solution to make this a bit less painful is to use an artifact repository on your local network for acting as a local cache for those external dependencies. LAN retrieval is almost always going to be faster than downloading even from the quickest of CDNs. We use Artifactory Pro as our artifact repository. In addition, since we have a multi-module structure, we store our own build artifacts in Artifactory as well. When we build one of our common packages, we can pull that specific version in via maven dependency resolution and retrieve the artifact right out of the artifact repository.

This works swimmingly. Artifactory also lets you synchronize your artifacts between instances so if you wanted to, say, use it to replicate your release repository to your data centers for production deployments, you could do this without needing to build a separate process.

Maven dependencies, feature branches and pull requests

All of our builds go into Artifactory. With SVN, we had been using a snapshot repository for keeping the latest 2 snapshot builds, a staging repostory for any release builds not yet approved, and a release repository only for the builds blessed to go into production.1 These builds are numbered like I described earlier, and are retrievable by a predictable URL pattern based on repository and version.

The primary workflow for each developer was to create a feature branch from the develop branch for their work, complete it, and place a pull request to have that work merged back into the develop branch. For a single project, this works mostly without issue, but let me paint a picture for you of the first problem we ran into head-first, and one that had us seriously reconsidering the entire migration:

As I said before we have multiple layers of dependency between our projects. There's a very good reason for this—both historically and strategically—for our products. We've considered alternate architectures that would eliminate this problem, but they'd introduce others. We can make our lives easier (and we did, but that's for a later post), but for now it's strategic for us to keep our structure as it is.

So developer A, let's call her Angela, starts work on a feature in JIRA. This requires two branches: one from our common project and one from product X. The version for common is 2.1.0-SNAPSHOT. The version for productX is 2.4.0-SNAPSHOT. She works locally for a while and then finally pushes back up to Bitbucket Server. Bamboo picks up these changes, builds the common package and uploads common-2.1.0-SNAPSHOT to Artifactory, then builds productX with a dependency on common-2.1.0-SNAPSHOT, uploading productX-2.4.0-SNAPSHOT as well. Unit tests pass!

Developer B, let's call him Bruce, starts work on another feature in JIRA, for a different product: productY. This also requires two branches: one from our common project and one from productY. The version for common is, as above, 2.1.0-SNAPSHOT. The version of product Y is 2.7.0-SNAPSHOT. He works locally for a while and then finally pushes his changes up to Bitbucket Server. Bamboo picks up these changes, builds the common package and uploads common-2.1.0-SNAPSHOT to Artifactory, then builds productX with a dependency on common-2.1.0-SNAPSHOT, uploading productX-2.4.0-SNAPSHOT as well. Unit tests pass!

Angela, meanwhile, finds a small bug in her productX code and writes a unit test to validate her fix. She runs it locally and it passes. She pushes her changes to Bitbucket Server, and Bamboo picks up the change and builds productX. The build succeeds, but some of her unit tests fail. It's not the new ones she wrote, but the first ones from her initial changes to the feature. Somehow the Bamboo build has found a regression that her local build didn't? How is that possible?

Because her common dependency, the one Bamboo pulled in when it built productX, was no longer her copy. Bruce over-wrote common-2.1.0-SNAPSHOT in artifactory when his feature build completed. There was no source code conflict—both developers were working in isolation on their own branches, but the source of truth for Maven's artifact retrieval was corrupted.

Head. Meet desk.

For about a month after we discovered this problem we tried everything to get around this. Through our TAM2, we talked to people on the Bamboo team who use git-flow, and we talked to the developer who maintains jgit-flow, a java implementation of git-flow. They were all super helpful, but short of a process that required a list of manual steps for each developer every time they worked on a feature, we couldn't find a resolution that was tolerable.

If you're curious what we considered, here's everything we tried:

  1. Modify the version number on branch creation, or immediately thereafter.
    • We can do this with mvn jgitflow:feature-start to create the branch.
    • We can use a Bitbucket Server hook or a local githook.
    • We can manually set with mvn version:set-version after we create the branch.
    • We can automate the change with the [mavent-external-version] plugin.
  2. Modify the version number when finishing the branch and merging back to develop.
    • We can do this with mvn jgitflow:feature-finish to finish the branch.
    • Use a git merge driver to handle pom conflicts.
    • Use an asynchronous post-receive hook in Bitbucket Server
  3. Do it all manually. (Just kidding. We didn’t consider this option very long.)

Each one of these options had some sort of negative side-effect. Chiefly, manual steps for a developer each and every time they needed a feature branch. And we wanted them to create feature branches all the time. Also in most cases we could not effectively use pull requests, which was a deal-breaker.

This consumed 1-2 people for almost two months until we had a (mindblown) revelation as to why we were approaching this problem from the wrong direction.

One version to rule them all

Hindsight being 20/20, I can clearly see that our biggest mistake was that we were focusing our attention on the git-flow tools rather than using the tools we already had to implement the workflow we wanted. We had:

  • JIRA Software
  • Bamboo Server
  • Maven
  • Artifactory Pro

Turns out, those were all of the tools we needed.

One of our engineers got the very bright idea that since the problem wasn't the build management itself but rather the artifacts being over-written, that we should fix Artifactory instead. His idea was to use a Maven property to set the snapshot repository URL to a custom URL which included the JIRA issue ID, and then write out its artifacts to a dynamically-created repository in Artifactory with a custom template. Maven’s dependency resolver will find artifacts in the develop snapshot repository if we haven’t needed to branch them, for instance if we’re only working on a product and not also common.

We set that handy little property variable in our build settings file, and wrote a Maven plugin to populate it during the earliest part of maven’s build lifecycle. On paper, this sounded incredible and re-invigorated the team to work harder to solve this problem. Trouble was that we couldn't actually do this. The earliest stage of the maven lifecycle is 'validate'. By the time plugins bound to validate have been run, the repository URLs were already resolved. Because of this, our variable never populated and the URL is not branch-named after all. Even though we had been using a layout in a separate repository from our develop snapshots, it wouldn’t be isolated for parallel development.

Head, meet desk again, your BEST FRIEND.

After a beer, the aforementioned engineer did some more digging and research into another way to add functionality to maven: extensions.

“Here’s to beer: the cause of, and solution to, all of life’s problems.” —Homer Simpson

Extensions, like plugins, give you a whole host of power to enhance your Maven workflow, however they are executed before lifecycle goals, and have greater access to the Maven internals. By utilizing the RepositoryUtils package, we forced Maven to re-evaluate its URLs using a custom parser and then re-set them using our updated values.3

Extension in place and tested, we started knocking off pre-migration tasks one after another, going from "this is never going to happen" to "this IS going to happen Monday... so now I need to write ten pages of documentation by tomorrow". I'll write more soon about how the tools work together to achieve our new development workflow, and some of the lessons we learned about the process.


  1. One downside here was that I had to use a script I wrote to hit the Artifactory REST API to “promote” builds from staging to release. It’s fast enough, but begging for more automation.

  2. Technical Account Manager. More information here.

  3. After the initial development efforts, we found that we had to do even more to make this work 100% of the time, like when a snapshot is newer in Artifactory (from another engineer) than your local snapshot, Maven grabs the remote artifact because hey, it’s NEWER, so it must be BETTER, right?

Oct 31st, 2015

Git or SVN? How Nuance Healthcare chose a Git branching model

This is the first post in a series about my engineering team moving from Subversion to Git, why we did it, and what we encountered along the way. I am speaking on this topic at Atlassian Summit 2015, and since I'm limited to 30 minutes, I wanted to provide more context and some pieces I'll be leaving out of the presentation for anyone interested.

Note: The official version of this post can be found at https://www.atlassian.com/git/articles/git-or-svn-git-branching-model.

Background

My team is in the healthcare division at Nuance. We're geographically-distributed between a couple of offices and homes on the East Cast of the US, and in an office in Pune. We develop Java web services to deliver NLP1 solutions to the healthcare market.

For the most part, our service consumers are other healthcare software companies (including ourselves) such as EHR vendors and healthcare analytics companies. We do directly sell some products to hospitals, and the end-users of the applications range from physicians to medical billing staff. "Normal" people like you and me don't ever touch the software my team builds.

Our team has been around the block a few times with Application Lifecycle Management product combinations. We started life with a mix of Rally Enterprise and Seapine TestTrack Pro, did about 14 months of hard labor with Rational Team Concert, and eventually migrated fully to the Atlassian stack (JIRA Software, Confluence, Bamboo, Crucible, Bitbucket and HipChat). Historically we used Subversion 1.4/1.5 an our SCM with a quasi-normal trunk/branches/tags structure. We have been using maven since forever manage our build projects and dependencies, and switched from Jenkins to Bamboo for continuous integration (CI) a while ago in order to make use of tighter integrations with JIRA and its flexible build and deploy agent capabilities. Everything we use (now) is behind the firewall for reasons2.

Git or SVN?

We support roughly ten individual products across four product families, and the owners of these products are always battling for prioritization and timing. It’s nice to have our work be in high demand, and this is by no means a complaint, but it also necessitates cutting releases at a weird cadence and needing to change directions in the middle of a sprint3.

Our development process really felt prohibitive at times. There was a conversation that my team was having on a regular basis that went something like this:

Me: We need to release 1.8.0 to QA now for regression testing so that Customer foo can go to beta next week.

Dev: I'm still working on ABC-123 which is in trunk. It's not done yet.

Me: Foo doesn't need ABC-123. We could put it in the next release.

Dev: But I've been working on it for weeks. There's no clear spot to branch from to cut a release.

Me: Well, you'll need to pull out all of your changes by hand. You have about two hours or QA can't finish in time.

I know, I sound like a jerk! I never meant to be, and of course I'm exaggerating a bit to make a point, but we really did have to figure out how to get code that was in one place out of that place temporarily so that we could cut a release, and then put it right back for the next release4. And this happened all the time.

Now, I know some of you are thinking "Subversion supports branches, Matt...". It absolutely does, and we used them on occasion with SVN 1.4 and 1.5. Branching is a fine operation in SVN; merging can be a pain in the ass. As SVN has matured, it has gotten better, for sure. But we knew there were better options out there for us, so when the question of SCN or git arose, we set out to get Git.

A side note: We briefly looked at the latest SVN (1.8 at the time) to see if it was strong enough to solve our problems, but weren't completely satisfied. One of our peer groups has a large Perforce setup and it had a lot of what we needed, but I simply couldn't stomach the licensing costs. We also looked at Mercurial for a moment, but in the end, the existing team's exposure to Git was enough to tell us that it was the right direction.

I won't sugar-coat this: Atlassian's tools really favor teams who use git. Other SCMs work fine; our SVN integration was sufficient in that it linked us to where a given user story's changes were made. The integration capabilities for teams who use Bitbucket Server5 instead, however, are both stronger and more natural-feeling in the JIRA Software interface and development experience — ditto with Bamboo.

Knowing this, and having seen some very stellar demos at Summit 2013, I strongly encouraged the team to go for it. Nobody objected, and we already had the licenses in place to make change.

Choosing a Git Branching Model

After deciding to make this change, the first challenge we had was deciding what Git branching model to implement for our team. Atlassian's Git microsite as well as this great presentation from Summit 2013 explain in greater detail what a branching model is. The short version is that it describes how you will use branches in git to power your development workflow.

In SVN, we had a model for branching I'll call "make one when you realize you — OMG! — need one":

  • The newest code is in trunk. Releases from trunk will be numbered A.B.0-{build}.
  • If a fix is required to a trunk-based release (e.g. we have a bug in 1.2.0-64), a branch is created and from there we will release A.B.C-{build} releases, where C increments after every release that goes out the door. These branches may never exist for a given A.B and we could even have more than one.
  • We also tag every release in a tags directory.

An Aside About Versions Many years ago, when I was just cutting my teath on managing a development team, our release engineer had a system of versining that was... how shall I say?... really unintuitive. Essentially, every release was a patch on the previous one (A.B.n), with no respect for the place from which the patch originated. Figuring out where something came from and, in almost all cases, the release order, required you to look at svn log. We printed the tree on a wall for reference.

In addition, our public-facing release numbers tend to be things like 3.0, 3.1, 3.5, 4.0, or essentially something a customer might expect. Remember, though that my team builds web services not a boxed product. Our APIs are a contract. A few years ago I made the executive that my team's builds, and therefore its releases, would adhere to Semantic Versioning rules. I've' had to stand my ground a few times with upper management, but now it is understood why the rules are what they are, and we haven't looked back. Partners appreciate that sort of clarity.

I mentioned a problem earlier wherein we'd be working on a release (let's say 1.2.0) and we'd have a feature still in progress as we approached a release date. We would need to pull that code out, cut our release, branch to branches/1.2.1 and then merge that code back in, hoping nobody had a hard drive crash in the meantime6.

Removing a whole feature by itself from a shared trunk is a pain. Everyone hated life when they had to do that. svn blame can be useful, as can a strong diff tool, but it’s still annoying to work with. I often took it personally, feeling that my bad planning had led to us not having all of our ducks in a row before it was time to be done with a release7. My team dealt with this for long enough.

Sometimes we'd over-correct to avoid the pain and would ask developers to sit on their hands for a couple of days (a virtual code freeze, if you will), just so we didn't pollute trunk before a release.

So we knew we needed, at least, feature branches. There's a simple Git branching model that is applicable: a master branch for what's in prod, and using feature branches for every feature, bug, etc. Team's have to manage merge order to ensure that what ships out in master is what is supposed to ship out for the release. This is, essentially, the same thing we had before, with some better feature isolation, but we wanted freedom with our power.

In our environment, we often need to keep a few versions in production, and may need to fix defects in a release that is 2-3 minor revisions older than what we are working on right now. So, in addition to feature branches, we also needed some sort of release branch or similar that would let us fix issues from previous releases. The Atlassian Bitbucket Server team does this. They make fixes in long-running support branches, and then merge them up the branch stream so that a fix makes it in to all of the support streams.

Their model looked really good, and we ran a few prototype interactions with this model to see if it would suit our needs. The "killer app" for them is their rolling merge of a fix up to their develop branch. While we liked this concept, every time we tried it, we ran into one issue or another with our maven dependencies. Also, as a rule, we couldn't guarantee we wanted a straight merge of the work from one version into another. In some cases we needed to implement the same fix in slightly different ways between versions, so a direct merge wasn't possible.

A few of the members of the team strongly favored a variation of this model known as "git-flow". Git-flow is a set of branch naming conventions and merge guidelines, authored by Vincent Driessen. This felt very natural to the team, and we liked the structure since it eliminated many of the questions around "what do I do when we need to do x?". The answers were generally very obvious. Rather than explaining what git-flow is, you can read more about it in Atlassian's tutorial.

The only gap left for us with git-flow was what to do about those long-running releases in production. Since master keeps moving forward, we couldn't use the git-flow hotfix workflow for a bug fix from a previous release. On the other hand, we didn't always want a support branch.

Most of the time a hotfix, only patching the latest release in production, should be sufficient; support is only there when we need to go back further, or when we need to maintain compatibility for one reason or another. That latter use case we dissected further and came up with criteria for choosing to use a support branch rather than a hotfix and minor version upgrade:

  1. This code cannot be trivially merged back into develop.
  2. The partner/customer cannot handle an interface change that comes with the latest release.
  3. There is an internal dependency which cannot be changed.8

Both git-flow extension packages9 provide support for the support branch concept, which isn't part of the original draft of git-flow, but has become popular enough to warrant inclusion.

Git-flow offered a workflow we liked, with the tooling support we needed. In the next post I'll go into what happened when we actually tried using it in a POC project we used to represent our development process. It was... a learning experience!


  1. Natural Language Processing. WE CAN READ YOUR THOUGHTS. (No. Not really.)

  2. There is a lot that is attractive about Atlassian’s cloud offerings, but we need to keep our fingers wrapped tightly around our servers and data for the time being. While we don’t personally need to do much with PHI data, our software does and it’s important to keep it as secure as possible.

  3. Shhhh… don’t tell Ken Schwaber.

  4. Which might have only been a few days later anyway.

  5. Formerly known as Stash. Hello, Atlassian Fall Rebranding!

  6. I know we could always pull it out of the previous commit. I was kidding.

  7. This wasn’t usually the case—generally it was because someone else’s timeframe moved up and we had to react quickly.

  8. This is one of those things I can’t get into on my own blog. Just trust me. “Reasons”.

  9. The original package by Vincent Driessen isn’t being maintained any longer. A new fork , however, is regularly updated.

Oct 30th, 2015

Canonical Post Urls in Jekyll

In the next couple of days I'll be contributing a series of blog posts to the Atlassian Developers blog to serve as a backdrop for my presentation at Atlassian Summit 2015, about a week and a half from now1. I wanted to syndicate that content here, but for SEO reasons I was asked to mark the Atlassian URL as the canonical one for each post. That isn't possible out of the box, so here's how I modified my Jekyll template to accommodate this change:

First, I add a canonical-url tag to my YAML front-matter:

1
canonical-url: http://www.yahoo.com

And second, I modified my header template to set the canonical variable and populate the proper <link ...> tag:

1
2
3
4
5
6
{% if page.canonical-url == nil %}
{% capture canonical %}{{ site.url }}{% if site.permalink contains '.html' %}{{ page.url }}{% else %}{{ page.url | remove:'index.html' | strip_slash }}{% endif %}{% endcapture %}
{% else %}
{% assign canonical = page.canonical-url %}
{% endif %}
<link rel="canonical" href="{{ canonical }}">

The documentation out there for liquid is somewhat less helpful than I'd prefer, so the proper way to test if a variable was truly unset took a few stabs, and getting the proper context for the page variable, despite pages really being posts, was also unintuitive. One of these days, I might have to give up on Jekyll based purely on the "clear as mud" factor.

NB: Getting the above Liquid code to actually display was a royal pain in the butt. This site was helpful, and I appreciated his closing sentence:

Displaying liquid code in jekyll is really an annoying task. So try to avoid it as much as you can.


  1. Holy crap.

Oct 27th, 2015

Selfish Ambition

He who attempts to act and do things for others or for the world without deepening his own self-understanding, freedom, integrity and capacity to love, will not have anything to give others. He will communicate to them nothing but the contagion of his own obsessions, his aggressiveness, his ego-centered ambitions, his delusions about ends and means, his doctrinaire prejudices and ideas.

Thomas Merton Contemplation in a World of Action
Oct 8th, 2015

Harder Better Faster Stronger Cover

Oct 5th, 2015

Ranked List of Star Trek Films

On a recent Mac Power Users episode, David Sparks and Katie Floyd were trading their most and least favorite Star Trek movies1. I haven't given the series much thought in a while, and had some different opinions.2

  1. Star Trek VI: The Undiscovered Country (1991)
  2. Star Trek (2009)
  3. Star Trek II: The Wrath of Khan (1982)
  4. Star Trek: First Contact (1996)
  5. Star Trek Into Darkness (2013)
  6. Star Trek IV: The Voyage Home (1986)
  7. Star Trek Generations (1994)
  8. Star Trek III: The Search for Spock (1984)
  9. Star Trek V: The Final Frontier (1989)
  10. Star Trek: The Motion Picture (1979)
  11. Star Trek: Insurrection (1998)
  12. Star Trek: Nemesis (2002)

Most people rank The Final Frontier as their least favorite of the set, but I don't mind it as a movie. It's my least favorite of the original series crew, but definitely not my least favorite overall.

I really like The Undiscovered Country. It was the first one I saw in theatres, having only recently gotten into Star Trek TNG around the same time, and it showed off the growing special effects capabilities of the industry with the older cast.

I also am a big fan of the reboot films. I think Chris Pine is a better Kirk than Shatner3, and the casting overall is spectacular. The writing is great and the integration of Leonard Nimoy as Spock' is a bonus. I get all of the timeline weirdness, and truthfully I don't care how/why they are where they are and how it's not the same as the original timeline. I'm not a purist.

Overall, the TNG cast films aren't my favorites. Once Frakes got ahold of the director's chair, the films became too much inside baseball and less about advancing the universe. Generations was good, mostly because of the Kirk. First Contact was really good. Insurrection was ok, but not good. The ending to Nemesis made me furious.4 It might be one of the only times I didn't enjoy Tom Hardy in a film.


  1. though not very seriously

  2. To be clear, I own all of them and would watch any of it were offered.

  3. sacrilege, I know

  4. That whole movie made me furious. It should have been called Star Trek: Narcissist for how Picard-focused it was. I wanted so badly to like it, but ended up hating nearly every plot direction.

Sep 30th, 2015