Recently we were having a debate about release processes, and I wrote that deployments are not always equal to releases. also deploy != release —Andy, baiting discussion in Slack This turned out to be somewhat controversial until we discussed what I specifically meant by deploy and release. As with all things, agreeing on definitions or understanding what someone means when they use a specific term is essential, so I thought I would write down a short blog post on it....

On several occasions when building complex projects, I have been tempted to set up Bazel to help speed up the build process; after all, it has a lot to offer: only building what has changed, caching built artifacts, and sharing that cache between machines for even more speed. TLDR We can use Make and a couple of short shell scripts to implement file content-based caching and read/write that cache to remote storage, such as S3....

I write a lot of blog posts which contain code snippets. Like most people, I have done this using a fenced codeblock in markdown which is fine for short blocks of code. The problem occours when I am embedding code from another repository; often there will be tweaks or bug fixes, and keeping the code in the blog post in sync with the code in the other repo is annoying and manual....

This is in response to the recent posts about the death of UML; while I think some parts of UML have fallen ill, the remaining parts are still alive, and useful to this day. TLDR Out of 14 types of diagram there are 3 that I use on a regular basis: Activity Diagram, State Machine Diagram, and Sequence Diagram. I think the Timing Diagram is borderline, but I can only think of a couple of occasions when it has been useful....

One of the reasons I prefer Pulumi over Terraform is the additional control I have over my processes due to the fact that it’s a programming language. For example, I have a CLI, that creates a cluster of machines for a user; the machines use IAM Authentication with Vault so that they can request certificates on boot. The trouble with this application is that it is slow; it takes 175 seconds on average to provision the machines, write the IAM information to Vault, and then re-run the cloud-init script on all the machines in the cluster (as when they first booted, the configuration hadn’t been written to Vault yet....

I recently encountered a behaviour in Nginx that I didn’t expect and caused a production outage in the process. While I would love to blame DNS for this, as it’s usually the cause of most network-related issues, in this case, the fault lies with Nginx. I was running a very simple Nginx proxy, relaying an internal service to the outside world. The internal service is behind an AWS ALB, and the Nginx configuration was proxying to the ALB’s FQDN:...

The Operator Pattern from Kubernetes is an excellent way of handling tasks in a cluster in an automated way, for example, provisioning applications, running backups, requesting certificates, and injecting chaos testing. As a Nomad user, I wanted to do something similar for my clusters, so I set about seeing how it would be possible. It turns out; it is much easier than I expected! While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API....

An interesting question came up at work today: how do you tag your Docker images? In previous projects, I’ve always used a short git sha, or sometimes a semver, but with no great consistency. As luck would have it, I had pushed for a change in tagging format at a client not so long ago as the method we were using didn’t make a lot of sense and, worst of all, it was a manual process....

Key Takeaway: os .cpus() returns the number of cores on a Kubernetes host, not the number of cores assigned to a pod. Investigating excessive memory usage Recently, when I was looking through a cluster health dashboard for a Kubernetes cluster, I noticed that one of the applications deployed was using a considerable amount of RAM - way more than I thought could be reasonable. Each instance (pod) of the application used approximately 8 GB of RAM, which was definitely excessive for a reasonably simple NodeJS webserver....

One of the things I like to do when setting up a Vault cluster is to visualise all the operations Vault is performing, which helps see usage patterns changing, whether there are lots of failed requests coming in, and what endpoints are receiving the most traffic. While Vault has a lot of data available in Prometheus telemetry, the kind of information I am after is best taken from the Audit backend....