Building a Blog Part 5: Continuous integration with CircleCI

January 22, 2019

In part 5 of Building a Blog I will talk about how I automate deployments of new blog posts using CircleCI. Every time I push to the master branch of my GitHub repository, a web hook is triggered and CircleCI checks out the latest code, runs a few tests, and finally deploys it. This makes it really easy for me to add new posts. However, the process isn’t that simple under the hood, and I want to explain in this post how it works.

Why CircleCI

If you want to build high quality software you need a continuous integration system. This is specially true if at some point you want have multiple collaborators. Almost every successful project or library hosted on GitHub uses either Travis CI or CircleCI to verify that new contributions don’t break the code. For web applications, the CI system can also deploy the app after successfully building it. If you want to automate your deployments and make your life easier, you need a CI system.

There are few options on the internet for CI, a lot of them are even free (as in free beer). Pretty much all of them integrate nicely with GitHub, requiring little or no configuration. I have used both Travis CI and CircleCI in the past, and they are both great services. I prefer CircleCI for mostly subjective reasons. What I like the most is that every build runs inside a container, and you can pick any image you want for that matter. CircleCI has huge variety of pre-built images that should cover basic needs. If you need something more complex you can also choose any image uploaded to DockerHub, including your own. This is essential for projects like mine that require multiple unrelated compilers or runtimes for building.

Preparing the build environment

Since this blog uses mostly Haskell and a bit of Node.js to generate the HTML, CSS and JS files, I need an image that has both Stack and Node.js installed in order to build the blog in a CircleCI job. It’s not a common combination so there is no pre-built image that I can use. The good thing is that I can create my own.

I need an image with multiple tools so I should pick a Linux distro that can make it easy to install them. I have used Arch Linux on all my personal computers for years. I think it’s a great minimalistic distro so I went with that and installed all dependencies via Pacman. This way I can build the blog in the cloud with same environment of my development machine.

Since a base Arch Linux install is very bare bones, containing only the absolutely indispensable for the OS to function there were a few gotchas like having to explicitly install gcc and make so that Stack can compile and install GHC; and configuring UTF-8 support so that Hakyll can compile files with Unicode characters. Once the image is uploaded to DockerHub I can reference it in my CircleCI config file and all future builds will run in that container.

Running Haskell code in CircleCI

Unlike JavaScript, Haskell code needs to be compiled before it can be executed. This process can be very intensive and unfit for constrained cloud environments. CircleCI builds run in virtualized machines with 2 CPUs and 4 GB RAM. If your build needs more memory, it will fail.

In my case, I want to run Hakyll in the cloud to generate the static pages and deploy them. So I need to run Stack to compile my static site generator. At first, I used this configuration:

This would fail after 30 minutes or so. It turns out that Hakyll pulls a lot of dependencies, generating a dependency tree of a size comparable to the one generated by your typical npm project. Since all this dependencies have to be compiled, the process takes significantly longer than running npm install.

Probably the biggest dependency here is is Pandoc. The build would always fail while compiling Pandoc. Every single time. When I ran stack build on my local machine I noticed that RAM usage peaked at just over 4 GB while compiling Pandoc, just over the limit. I was about to give up on this and just generate the pages on my local machine instead of the cloud, but then I found out that stack build supports a few flags that can reduce RAM usage:

Here is the updated configuration:

This finally works. The build succeeds but there is one little problem here, though. It takes almost a full hour to compile the whole thing! Waiting an hour to have a new post published sounds like bad automation. Luckily, CircleCI supports caching dependencies and generated binaries to drastically speed up builds like this one.

Caching for faster builds

To avoid wasting time compiling Pandoc every single time, I can declare in my CircleCI config which paths on the file system should be cached for the next builds. Here’s the configuration:

The steps can be summarized like this:

  1. Checkout code from Git.
  2. Restore the cache.
  3. Run steps that generate things that should be cached. In this case the steps are:
    • Download & compile the static site generator’s dependencies.
    • Download & compile the Scotty server’s dependencies.
    • Pull npm dependencies used in the frontend.
  4. Store outputs in the cache. I just have to specify the paths that should be saved and on the next build they will be added to the container when restoring the cache.
  5. After caching, I can finally run tests, build and deploy the site.

You might be wondering why I chose these paths for caching, so I’ll explain:

Now that everything is cached, pushing new posts can take less than 2 minutes to deploy. Nice!

Deploying with webhooks

After running tests and building the site, the final step is to deploy the thing. How do we achieve that? There are many approaches.

Probably the easiest one is to and copy files to the server using scp. This is a very solid approach, but I personally dislike it because I have to manage SSH keys on the server, which means extra configuration that is not part of my repository. Also it would be nice to have version control of the files that get deployed. Instead of blindly copying files to the server I’d prefer to commit them in a new new orphan branch and have the server pull from that branch during every deploy. This way, if something goes wrong I can use Git to quickly rollback to a previous version.

What I ended up doing is having a server branch on GitHub with the blog’s generated static files and the sources for the backend. This branch is synced with the server using GitHub webhooks. Everytime a new post is pushed to the server branch, GitHub sends an HTTP POST request to the blog’s server so that it can update itself.

Listening for GitHub events on the server

In order to use GitHub webhooks, I need a program on the server that listens for POST requests and pulls the latest code whenever it gets updated. I already have a Scotty server for my blog’s backend so I can add a new endpoint that runs a bash script that runs a git pull every time it receives a POST request. Here’s the code that runs the script:

It expects an update.sh script to exist in the specified working directory. It spawns a new child process that runs this script, ignoring stdout and stderr and waits for its completion returning the exit code. Here is the Scotty handler that runs this code:

This handler runs whenever a POST /ci-hook request is received. It runs the update script at the configured working directory and returns status code 200 if it succeeds, otherwise it returns status code 500.

This works pretty well. The only problem is that anyone could trigger a git pull just by sending a POST request if it was a public endpoint. Fortunately my Scotty server only listens to requests forwarded by my Nginx instance. In order to “hide” the endpoint, I use this configuration:

location = /ci/CI_SECRET {
        rewrite ^ /ci-hook break;

        proxy_pass http://localhost:API_PORT;
}

The = means that the path has to be exactly the same as /ci/CI_SECRET where CI_SECRET is a random string like “e7093f2c-4b04-421c-8c41-d81dba20aa0a”. The random string is only kept by GitHub to send the requests so it is secure for potential attackers.

Triggering the webhooks

The server is ready to receive payloads from GitHub so now we need the code that triggers the GitHub webhooks. I just use a bash script that pushes the updated pages to the server branch. Note that to push to GitHub CircleCI needs write permissions to repo. By default it only has read permissions because that’s enough for building the project. This is easily configured from settings though.

If we go back to the CircleCI configuration, the final two steps are to generate the static pages, and commit & push the updated pages to GitHub, triggering the webhooks. Here’s the result:

If you are curious about push_to_server.sh, the bash script that does all the Git work, here it is:

And that is all! Here’s the recap of the whole deploy process:

  1. CircleCI runs tests and builds the site.
  2. If new pages have to be deployed, they are pushed to GitHub.
  3. Github sends a POST request to the Scotty server after the repo is updated.
  4. The server pulls the latest code from the server branch and Ngninx automatically servers the updated pages.

That’s all I can say about my blog. I think it was pretty fun to do it. Some parts are a bit too convoluted and/or over-engineered for such a small site, but I did it because I don’t really get to play with these tools on my day job. Hope you enjoyed reading this. See you around!