consul

When it comes to implementing a new feature in an application’s ecosystem, I don’t like spending my innovation tokens unless I have to, so I try not to add new tools to my infrastructure unless I really need them. This same approach comes when I either want, need, or have been told, to implement a Service Mesh. This means I don’t instantly setup Istio. Not because it’s bad - far from it - but because it’s extra complexity I would rather avoid, unless I need it....

I noticed when running an Alpine based virtual machine with Consul DNS forwarding set up, that sometimes the machine couldn’t resolve *.consul domains, but not in a consistent manner. Inspecting the logs looked like the request was being made and responded to successfully, but the result was being ignored. After a lot of googling and frustration, I was able to track down that it’s down to a difference (or optimisation) in musl libc, which glibc doesn’t do....

So we want to set up a Vault instance, and have it’s storage be a TLS based Consul cluster. The problem is that the Consul cluster needs Vault to create the certificates for TLS, which is quite the catch-22. Luckily for us, quite easy to solve: Start a temporary Vault instance as an intermediate ca Launch Consul cluster, using Vault to generate certificates Destroy temporary Vault instance Start a permanent Vault instance, with Consul as the store Reprovision the Consul cluster with certificates from the new Vault instance There is a repository on Github with all the scripts used, and a few more details on some options....

I was recently using my Hashibox for a test, and I noticed the DNS resolution didn’t seem to work. This was a bit worrying, as I have written about how to do DNS resolution with Consul forwarding in Ubuntu, and apparently something is wrong with how I do it. Interestingly, the Alpine version works fine, so it appears there is something not quite working with how I am configuring Systemd-resolved....

This post is going to go through how to set up a Consul cluster to communicate over TLS. I will be using Vagrant to create three machines locally, which will form my cluster, and in the provisioning step will use Vault to generate the certificates needed. How to securely communicate with Vault to get the TLS certificates is out of scope for this post. Host Configuration Unless you already have Vault running somewhere on your network, or have another mechanism to generate TLS certificates for each machine, you’ll need to start and configure Vault on the Host machine....

I wanted to implement canary routing for some HTTP services deployed via Nomad the other day, but rather than having the traffic split by weighting to the containers, I wanted to direct the traffic based on a header. My first choice of tech was to use Fabio, but it only supports routing by URL prefix, and additionally with a route weight. While I was at JustDevOps in Poland, I heard about another router/loadbalancer which worked in a similar way to Fabio: Traefik....

DEPRECATED - This has a race condition! Please see this post for an updated version which works! Following on from the post the other day on setting up DNS forwarding to Consul with SystemD, I wanted also to show how to get Consul up and running under Alpine Linux, as it’s a little more awkward in some respects. To start with, I am going to setup Consul as a service - I didn’t do this in the Ubuntu version, as there are plenty of useful articles about that already, but that is not the case with Alpine....

DEPRECATED - This doesn’t work properly Please see this post for an updated version which works! One of the advantages of using Consul for service discovery is that besides an HTTP API, you can also query it by DNS. The DNS server is listening on port 8600 by default, and you can query both A records or SRV records from it. SRV records are useful as they contain additional properties (priority, weight and port), and you can get multiple records back from a single query, letting you do load balancing client side:...

Last time I wrote about running a RabbitMQ cluster in Nomad, one of the main pieces of feedback I received was about the (lack) of security of the setup, so I decided to revisit this, and write about how to launch as secure RabbitMQ node in Nomad. The things I want to cover are: Username and Password for the management UI Secure value for the Erlang Cookie SSL for Management and AMQP As usual, the demo repository with all the code is available if you’d rather just jump into that....

Update If you want a secure version of this cluster, see Running a Secure RabbitMQ Cluster in Nomad. RabbitMQ is the centre of a lot of micros service architectures, and while you can cluster it manually, it is a lot easier to use some of the auto clustering plugins, such as AWS (EC2), Consul, Etcd, or Kubernetes. As I like to use Nomad for container orchestration, I thought it would be a good idea to show how to cluster RabbitMQ when it is running in a Docker container, on an unknown host (i....