Publishing This Blog With Bitbucket Pipelines

Earlier this year, Atlassian released Bitbucket Pipelines, it's Cloud CI offering, as a beta product coupled with their hosted Git/Hg source control service. I like to know as much as possible about my products, but I'm not much of a software developer1, so finding a use case to try out Pipelines was puzzling.

Then I remembered that I build this very blog. (Oops) It's not much of a job to run an octopres/jekyll build, but my current setup was highly-dependent on a single machine at my house always being on with Dropbox working. It was a little too fragile. I host this blog on nearlyfreespeech.net, and though I know there are other hosting options out there that might do the whole build and deploy process natively for me2, I (also) try to never shy away from a challenge.

At Summit last month, Atlassian released Pipelines as GA, and set an intro price of FREE for the remainder of the year.3 I was out of excuses, so in my spare minutes over the past month I have tried to get a simple jekyll build of the site to work in pipelines.

And, of course, ran into one problem after another.

First, pipelines runs its build in a Docker container. This isn't a negative, actually, but it added a complexity with which I had relatively little experience.. I figured the simplest way to get started was to use an existing ruby image that was the same version on my Mac and just install dependencies as part of the build. Jekyll, though, requires a JavaScript compiler and no matter what I tried to do in the build to install oneall of them, the best option was to get node working since that's how my current build works. At that point, though, my build script was tens of steps each run, meaning my build-minute use was going to be super high each time I wanted to publish something.

Rather than build my own Dockerfile, which was really tempting, I deciced to use another image in the Docker Hub that has both ruby and node already set up. It's not far from what I was about to do myself, so no use re-inventing...

...which was good, because second, I really didn't want to have to install Docker. Every time I have previously tried to install Docker and have it work reliably, the VirtualBox piece just dies on me at some point. (More on this in a moment, though, because I'm rarely this lucky.)

My blog is already a private bitbucket repository. I was able to skip a few parts of the setup and just enable Pipelines on my existing repo, though I chose to create a branch for this work which I merged in once I was happy with the results. I also wanted my repo to be as simple as possible, so I spent some time adding items to my .gitignore and expanding the rake task for cleanup to remove the generated site.

An aside: Now that I work for Atlassian, I have a parallel set of accounts to the ones I had before as a customer. This means I have two bitbucket accounts, and as a result my normal method of keeping my bitbucket ssh key in my local keyring failed to choose the right key when working with my blog repo locally. Enter [git aliasing](https://developer.atlassian.com/blog/2016/04/different-ssh-keys-multiple-bitbucket-accounts/), which is super handy.

That fixed, I looked at how my current site generation task is run4 and tried to replicate that via the bitbucket-pipelines.yml:

1
2
3
4
5
6
7
8
9
image: starefossen/ruby-node:latest

pipelines:
  default:
    - step:
        script:
          - bundle install
          - rake generate
          - rake deploy

After that, I knew I'd need to get key-based SSH set up to actually deploy the content:5

1
2
3
4
5
6
7
8
9
10
11
12
image: starefossen/ruby-node:latest

pipelines:
  default:
    - step:
        script:
          - mkdir -p ~/.ssh
          - cat my_known_hosts >> ~/.ssh/known_hosts
          - (umask 077 ; echo $SSH_KEY_VAR | base64 --decode > ~/.ssh/id_rsa)
          - bundle install
          - rake generate
          - rake deploy

And while I expected that to work, I quickly found out that rsync wasn't part of the ruby-node docker image:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
image: starefossen/ruby-node:latest

pipelines:
  default:
    - step:
        script:
          - apt-get update
          - apt-get install -y rsync
          - mkdir -p ~/.ssh
          - cat my_known_hosts >> ~/.ssh/known_hosts
          - (umask 077 ; echo $SSH_KEY_VAR | base64 --decode > ~/.ssh/id_rsa)
          - bundle install
          - rake generate
          - rake deploy

This worked, nearly flawlessly, with one major problem: the results of rake generate did not match what I had at home, and for a little while I had a very ugly, completely empty, web site. (Oops)

After a respite and far too many (frustrating) re-runs, I decided I had no choice but to bite the bullet and get Docker installed locally. In the last 8-ish months, though, Docker has fixed the problem that had been bugging me for ever and ever, and removed the need to have a separate VM application running on the machine. Replicating my pipeline environment was therefore trivial6 and I ran through the build steps without any issues.

Which was infuriating, because it worked perfectly fine.

So, you know, it has to be the environment.

Turns out I was over-zealous in my cleanup efforts and had removed the Gemfile.lock from the repo, meaning that it was grabbing all of the latest dependency versions from rubygems. Somewhere in there was an update that broke generation altogether. The lock file was still there locally (because I had run bundle install outside of Docker at some point in the troubleshooting process), so once it was pushed back to the remote, the pipeline build completed flawlessly. And deployed. And I was happy.

Each time my site builds, I will be consuming about 2.5 build-minutes. I can publish 20ish posts per month (HA!) for free for the foreseeable future.

This was a great learning experience. Prior to this a majority of my CI experience had been using Bamboo to kick off mvn commands. I'd highly recommend using pipelines for simple build tasks, even if it's for things like unit tests, content generation, publishing to a remote, whatever.


  1. My weak-arse perl skills don’t count, nor do I write unit tests for my hack-jobs, alleviating me of any sort of CI benefits…

  2. I had been using the free aerobatic.io hosting in bitbucket for a bit, and I could move to Github and hsot there with a CNAME, but I’m happy with my setup.

  3. Starting in January the cost goes up to whopping $0 for 50 build-minutes, $10/mo for 1000 if for some reason I need that much.

  4. Via hazel, as a simple shell script.

  5. I apprecate that Atlassian made it trivial to store a local key securely via a protected environment variable.

  6. No, seriously. Maybe 5 minutes, while the kids ran around screaming and hitting each other with light sabers.

Nov 10th, 2016

Comments