Creating a TLS enabled Consul cluster

14 Sep 2019

This post is going to go through how to set up a Consul cluster to communicate over TLS. I will be using Vagrant to create three machines locally, which will form my cluster, and in the provisioning step will use Vault to generate the certificates needed.

How to securely communicate with Vault to get the TLS certificates is out of scope for this post.

Host Configuration

Unless you already have Vault running somewhere on your network, or have another mechanism to generate TLS certificates for each machine, you’ll need to start and configure Vault on the Host machine. I am using my Vault Dev Intermediate CA script from my previous post.

To set this up, all I need to do is run this on the host machine, which starts Vault in a docker container, and configures it as an intermediate certificate authority:

./run_int.sh

I also have DNS on my network setup for the tecra.xyz domain so will be using that to test with.

Consul Machine Configuration

The Vagrantfile is very minimal - I am using my Hashibox (be aware the libvirt provider for this might not work, for some reason vagrant package with libvirt produces a non-bootable box).

Vagrant.configure(2) do |config|
  config.vm.box = "pondidum/hashibox"
  config.vm.provision "consul", type: "shell", path: "./provision.sh"

  config.vm.define "c1" do |c1|
    c1.vm.hostname = "consul1"
  end

  config.vm.define "c2" do |c2|
    c2.vm.hostname = "consul2"
  end

  config.vm.define "c3" do |c3|
    c3.vm.hostname = "consul3"
  end
end

The hashibox script already has all the tools we’ll need installed already: Consul, Vault, and jq.

First up, we request a certificate from Vault to use for Consul - How you get this certificate in a secure manner in a production environment is up to you. There is a catch-22 here for me, in that in a production environment I use Vault with Consul as it’s backing store…but Consul needs Vault to start! I’ll go over how I get around this in a future post.

export VAULT_ADDR="http://vault.tecra.xyz:8200"
export VAULT_TOKEN="vault"

response=$(vault write pki/issue/cert -format=json common_name=$HOSTNAME.tecra.xyz alt_names="server.dc1.consul")
config_dir="/etc/consul.d"

The first thing to note is that we have specified an alt_names for the certificate - you must have a SAN of server.$DC.$DOMAIN so either server.dc1.consul or server.euwest1.tecra.xyz, and the server prefix is required!.

Next, we need to take all the certificates from the response and write them to the filesystem.

mkdir -p "$config_dir/ca"

for (( i=0; i<$(echo "$response" | jq '.data.ca_chain | length'); i++ )); do
  cert=$(echo "$response" | jq -r ".data.ca_chain[$i]")
  name=$(echo "$cert" | openssl x509 -noout -subject -nameopt multiline | sed -n 's/ *commonName *= //p' | sed 's/\s//g')

  echo "$cert" > "$config_dir/ca/$name.pem"
done

echo "$response" | jq -r .data.private_key > $config_dir/consul.key
echo "$response" | jq -r .data.certificate > $config_dir/consul.crt
echo "$response" | jq -r .data.issuing_ca >> $config_dir/consul.crt

The for loop iterates through all of the certificates returned in the ca_chain and writes them into a ca directory. We use openssl to get the name of the certificate, so the files are named nicely!

Finally, it writes the private_key for the node’s certificate to consul.key, and both the certificate and issuing_ca to the consul.crt file.

Now for the consul.json. To setup a secure cluster, first of all we need to add the certificate configuration, pointing to the files we wrote earlier:

"ca_path": "$config_dir/ca/",
"cert_file": "$config_dir/consul.crt",
"key_file": "$config_dir/consul.key",

We will also disable the HTTP port, and enable the HTTPS port:

"ports": {
    "http": -1,
    "https": 8501
}

Finally, we need to add some security settings. First is encrypt, which sets that the key that all Consul nodes will use to encrypt their communications. It must match on all nodes. The easiest way to generate this is just run consul keygen and use the value that produces.

  • "encrypt": "oNMJiPZRlaP8RnQiQo9p8MMK5RSJ+dXA2u+GjFm1qx8=":

    The key the cluster will use to encrypt all it’s traffic. It must be the same on all nodes, and the easiest way to generate the value is to use the output of consul keygen.

  • "verify_outgoing": true:

    All the traffic leaving this node will be encrypted with the TLS certificates. However, the node will still accept non-TLS traffic.

  • "verify_incoming_rpc": true:

    All the gossip traffic arriving at this node must be signed with an authority in the ca_path.

  • "verify_incoming_https": false:

    We are going to use the Consul Web UI, so we want to allow traffic to hit the API without a client certificate. If you are using the UI from a non-server node, you can set this to true.

  • "verify_server_hostname": true:

    Set Consul to verify outgoing connections have a hostname in the format of server.<datacenter>.<domain>. From the docs: “This setting is critical to prevent a compromised client from being restarted as a server and having all cluster state including all ACL tokens and Connect CA root keys replicated to it”

The complete config we will use is listed here:

(
cat <<-EOF
{
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "data_dir": "/var/consul",
    "leave_on_terminate": true,
    "rejoin_after_leave": true,
    "retry_join": ["consul1", "consul2", "consul3"],
    "server": true,
    "ui": true,
    "encrypt": "oNMJiPZRlaP8RnQiQo9p8MMK5RSJ+dXA2u+GjFm1qx8=",
    "verify_incoming_rpc": true,
    "verify_incoming_https": false,
    "verify_outgoing": true,
    "verify_server_hostname": true,
    "ca_file": "$config_dir/issuer.crt",
    "cert_file": "$config_dir/consul.crt",
    "key_file": "$config_dir/consul.key",
    "ports": {
        "http": -1,
        "https": 8501
    }
}
EOF
) | sudo tee $config_dir/consul.json

Lastly, we’ll make a systemd service unit to start consul:

(
cat <<-EOF
[Unit]
Description=consul agent
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/bin/consul agent -config-file=$config_dir/consul.json -bind ''
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target
EOF
) | sudo tee /etc/systemd/system/consul.service

sudo systemctl daemon-reload
sudo systemctl enable consul.service
sudo systemctl start consul

As the machines we are starting also have docker networks (and potentially others), our startup line specifies to bind to the eth0 network, using a Consul Template.

Running

First, we need to run our intermediate CA, then provision our three machines:

./run_int.sh
vagrant up

After a few moments, you should be able to curl the consul ui (curl https://consul1.tecra.xyz:8501) or open https://consul1.tecra.xyz:8501 in your browser.

Note, however, the if your root CA is self-signed, like mine is, some browsers (such as FireFox) won’t trust it, as they won’t use your machine’s Trusted Certificate Store, but their own in built store. You can either accept the warning or add your root certificate to the browser’s store.

Testing

Now that we have our cluster seemingly running with TLS, what happens if we try to connect a Consul client without TLS to it? On the host machine, I just run a single node, and tell it to connect to one of the cluster nodes:

consul agent \
  -join consul1.tecra.xyz \
  -bind '' \
  -data-dir /tmp/consul

The result of this is a refusal to connect, as the cluster has TLS configured, but this instance does not:

==> Starting Consul agent...
==> Log data will now stream in as it occurs:
==> Joining cluster...
==> 1 error occurred:
  * Failed to join 192.168.121.231: Remote state is encrypted and encryption is not configured

Success!

In the next post, I’ll go through how we can set up a Vault cluster which stores its data in Consul, but also provision that same Consul cluster with certificates from the Vault instance!

vault, security, tls, consul

---

Using Vault as a Development CA

25 Aug 2019

Often when developing or testing some code, I need (or want) to use SSL, and one of the easiest ways to do that is to use Vault. However, it gets pretty annoying having to generate a new CA for each project, and import the CA cert into windows (less painful in Linux, but still annoying), especially as I forget which cert is in use, and accidentally clean up the wrong ones.

My solution has been to generate a single CA certificate and PrivateKey, import this into my Trusted Root Certificate Store, and then whenever I need a Vault instance, I just setup Vault to use the existing certificate and private key. The documentation for how to do this seems somewhat lacking, so here’s how I do it.

Things you’ll need:

  • Docker
  • Vault cli
  • JQ

Generating the Root Certificate

First we need to create a Certificate, which we will do using the Vault docker container, and our local Vault CLI. We start the docker container in the background, and mark it for deletion when it stops (--rm):

container=$(docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest)

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

certs_dir="./ca"
max_ttl="87600h" # 10 years why not

mkdir -p $certs_dir
rm -rf $certs_dir/*.*

vault secrets enable pki
vault secrets tune -max-lease-ttl=$max_ttl pki

Finally, we generate a certificate by writing to the pki/root/generate/exported path. If the path ends with exported the Private Key is returned too. If you specify /internal then the Private Key is stored internally to Vault, and never accessible.

result=$(vault write -format "json" \
  pki/root/generate/exported \
  common_name="Local Dev CA" \
  alt_names="localhost,mshome.net" \
  ttl=$max_ttl)

echo "$result" > $certs_dir/response.json
echo "$result" | jq -r .data.certificate > $certs_dir/ca.crt
echo "$result" | jq -r .data.private_key > $certs_dir/private.key

docker stop $container

We put the entire response into a json file just incase there is something interesting we want out of it later, and store the certificate and private key into the same directory too. Note for the certificate’s alt_names I have specified both localhost and mshome.net, which is the domain that Hyper-V machines use.

Lastly, we can now import the root CA into our machine/user’s Trusted Root Certification Authorities store, meaning our later uses of this certificate will be trusted by our local machine.

Creating a Vault CA

As before, we use a Docker container to run the Vault instance, except this time we import the existing CA certificate into the PKI backend. The first half of the script (run_ca.sh) is pretty much the same as before, except we don’t delete the contents of the ./ca directory, and our certificate max_ttl is much lower:

docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

certs_dir="./ca"
max_ttl="72h"

vault secrets enable pki
vault secrets tune -max-lease-ttl=$max_ttl pki

The last part is to read in the certificate and private key, bundle them together, and configure the pki backend to use them, and add a single role to use for issuing certificates:

pem=$(cat $certs_dir/ca.crt $certs_dir/private.key)

vault write pki/config/ca pem_bundle="$pem"

vault write pki/roles/cert \
  allowed_domains=localhost,mshome.net \
  allow_subdomains=true \
  max_ttl=$max_ttl

Also note how we don’t stop the docker container either. Wouldn’t be much of a CA if it stopped the second it was configured…

Creating a Vault Intermediate CA

Sometimes, I want to test that a piece of software works when I have issued certificates from an Intermediate CA, rather than directly from the root. We can configure Vault to do this too, with a modified script which this time we start two PKI secret backends, one to act as the root, and onc as the intermediate:

#!/bin/bash

set -e

docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

# create root ca
certs_dir="./ca"
pem=$(cat $certs_dir/ca.crt $certs_dir/private.key)

vault secrets enable -path=pki_root pki
vault secrets tune -max-lease-ttl=87600h pki_root
vault write pki_root/config/ca pem_bundle="$pem"

# create the intermediate
vault secrets enable pki
vault secrets tune -max-lease-ttl=43800h pki

csr=$(vault write pki/intermediate/generate/internal \
  -format=json common_name="Spectre Dev Intermdiate CA" \
  | jq -r .data.csr)

intermediate=$(vault write pki_root/root/sign-intermediate \
  -format=json csr="$csr" format=pem_bundle ttl=43800h \
  | jq -r .data.certificate)

chained=$(echo -e "$intermediate\n$(cat $certs_dir/ca.crt)")

vault write pki/intermediate/set-signed certificate="$chained"

echo "$intermediate" > intermediate.crt

vault write pki/roles/cert \
  allowed_domains=localhost,mshome.net \
  allow_subdomains=true \
  max_ttl=43800h

# destroy the temp root
vault secrets disable pki_root

We use the pki_root backend to sign a CSR from the pki (intermediate) backend, and once the signed response is stored in pki, we delete the pki_root backend, as it is no longer needed for our Development Intermediate CA.

Issuing Certificates

We can now use the cert role to issue certificates for our applications, which I have in a script called issue.sh:

#!/bin/bash

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

vault write \
  -format=json \
  pki/issue/cert \
  common_name=$1.mshome.net

This script I usually use with jq to do something useful with:

response=$(./issue.sh consul)

cert=$(echo "$response" | jq -r .data.certificate)
key=$(echo "$response" | jq -r .data.private_key)

Cleaning Up

When I have finished with an application or demo, I can just stop the Vault container, and run the run_ca.sh script again if I need Vault for another project.

vault, security, tls

---

Architecture Decision Records

29 Jun 2019

This is a text version of a short talk (affectionately known as a “Coffee Bag”) I gave at work this week, on Architecture Design Records. You can see the slides here, but there isn’t a recording available, unfortunately.

It should be noted; these are not to replace full architecture diagrams; you should definitely still write C4 Models to cover the overall architecture. ADRs are for the details, such as serializer formats, convention-over-configuration details, number precisions for timings, or which metrics library is used and why.

What?

Architecture Design Records are there to solve the main question people repeatedly ask when they view a new codebase or look at an older part of their current codebase:

Why on earth was it done like this?!

Generally speaking, architectural decisions have been made in good faith at the time, but as time marches on, things change, and the reasoning gets lost. The reasoning might be discoverable through the commit history, or some comments in a type somewhere, and every once in a while, people remember the Wiki exists, and hope that someone else remembered and put some docs there. They didn’t by the way.

Architecture Design Records are aiming to solve all of this, with three straightforward attributes: Easy to Write, Easy to Read, and Easy to Find. Let’s look at these on their own, and then have a look at an example.

Easy to Find

As I alluded to earlier, “easy to find” doesn’t mean “hidden in confluence” (or any other wiki, for that matter.) The best place to put records of architecture decisions is in the repository. If you want them elsewhere, that’s fine, but the copy in the repository should be the source of truth.

As long as the location is consistent (and somewhat reasonable), it doesn’t matter too much where they go. I like to put them in the docs/arch path, but a common option is docs/adr too:

$ tree ~/dev/projects/awesome-api
|-- docs
|   `-- arch
|       |-- api-error-codes.md
|       |-- controller-convention.md
|       `-- serialization-format.md
|-- src
|-- test
`-- readme.md

The file names for each architecture decision are imperative - e.g. “serialization format”, rather than “figure out what format to use”, much like your commit messages are (right?) You might also note that the files are Markdown. Because what else would they be really?

Easy to Write

As just mentioned, I usually use Markdown for writing all documents, but as long as you are consistent (notice a pattern here?) and that it is plain-text viewable (i.e. in a terminal), it doesn’t matter too much. Try and pick a format that doesn’t add much mental overhead to writing the documents, and if it can be processed by tools easily, that’s a bonus, as we will look into later.

Easy to Read

There are two components to this: Rendering and Format.

Rendering is covering how we actually read it - plain text in a terminal, syntax highlighting in an editor, or rendered into a web page. Good ADRs can handle all three, and Markdown is a good fit for all of them! By using Markdown, not only can we render to HTML, we can even use Confluences’s questionable “Insert Markdown Markup” support to write them into a wiki location if desired.

Format is covering what the content of the document is. There are many different templates you can use, which have different levels of detail, and are aimed at different levels of decisions. I like to use a template based off Michael Nygard’s, which I modified a little bit to have the following sections:

  • Title
  • Status
  • Context
  • Considered Options
  • Chosen Decision
  • Consequences

Let’s have a look at these in an example.

Example

We have a new API we are developing, and we need to figure out which serialization format we should use for all the requests and responses it will handle.

We’ll start off with our empty document and add in the Title, and Status:

# Serialization Format

## Status

In Progress

The Title is usually the same as the file name, but not necessarily. The Status indicates where the document is in its lifespan. What statuses you choose is up to you, but I usually have:

  • In Progress
  • Accepted
  • Rejected
  • Superseded
  • Deprecated

Once an ADR is Accepted (or Rejected), the content won’t change again. Any subsequent changes will be a new ADR, and the previous one will be marked as either Deprecated or Superseded, along with a link to the ADR which replaces it, for example:

## Status

Superseded by [Api Transport Mechanisms](api-transport-mechanisms.md)

Next, we need to add some context for the decision being made. In our serialization example, this will cover what area of the codebase we are covering (the API, rather than storage), and any key points, such as message volume, compatibilities etc.

## Context

We need to have a consistent serialization scheme for the API.  It needs to be backwards and forwards compatible, as we don't control all of the clients.  Messages will be fairly high volume and don't *need* to be human readable.

Now that we have some context, we need to explain what choices we have available. This will help when reading past decisions, as it will let us answer the question “was xxxx or yyyy considered?”. In our example, we consider JSON, Apache Avro, the inbuilt binary serializer, and a custom built serializer (and others, such as Thrift, ProtoBufs, etc.)

## Considered Options

1. **Json**: Very portable, and with serializers available for all languages.  We need to agree on a date format, and numeric precision, however.  The serialization should not include white space to save payload size.  Forwards and Backwards compatibility exists but is the developer's responsibility.

2. **Apache Avro**: Binary format which includes the schema with the data, meaning no need for schema distribution.  No code generator to run, and libraries are available for most languages.

3. **Inbuilt Binary**: The API is awkward to use, and its output is not portable to other programming languages, so wouldn't be easy to consume for other teams, as well as some of our internal services.

4. **Custom Built**: A lot of overhead for little to no benefit over Avro/gRPC etc.

5. **Thrift**: ...

The second to last section is our Chosen Decision, which will not only list which one we picked (Avro, in this case) but also why it was chosen over other options. All this helps reading older decisions, as it lets you know what was known at the time the decision was made - and you will always know less at the time of the decision than you do now.

## Chosen Decision

**2. Apache Avro**

Avro was chosen because it has the best combination of message size and schema definition.  No need to have a central schema repository set up is also a huge benefit.

In this example, we have selected Avro and listed that our main reasons were message size, and the fact that Avro includes the schema with each message, meaning we don’t need a central (or distributed) schema repository to be able to read messages.

The final section is for Consequences of the decision. This is not to list reasons that we could have picked other decisions, but to explain things that we need to start doing or stop doing because of this decision. Let’s see what our example has:

## Consequences

As the messages are binary format, we cannot directly view them on the wire.  However, a small CLI will be built to take a message and pretty print it to aid debugging.

As we have selected a binary message format, the messages can’t be easily viewed any more, so we will build a small CLI which when given a message (which as noted, contains the schema), renders a human-readable version of the message.

Dates

You might notice that the record doesn’t contain any dates so far. That is because it’s tracked in source control, which means we can pull all the relevant information from the commit history. For example, a full list of changes to any ADR could be fetched from Git with this command:

git log --format='%ci %s' -- docs/arch/

Likewise, when you’re running your build process, you could extract the commit history which effects a single ADR:

git log --reverse --format='%ci %s' -- docs/arch/serialization-format.md

And then take that list and insert it into the rendered output so people can see what changed, and when:

<div style="float: right">
<h2>History</h2>
    <ul>
        <li><strong>2018-09-26</strong> start serialization format docs</li>
        <li><strong>2018-09-26</strong> consider json</li>
        <li><strong>2018-09-26</strong> consider avro, inbuilt binary and custom binary</li>
        <li><strong>2018-09-27</strong> should consider thrift too</li>
        <li><strong>2018-09-28</strong> select Avro</li>
        <li><strong>2018-09-28</strong> accepted :)</li>
        <li><strong>2019-03-12</strong> accept api transport mechanisms</li>
    </ul>
</div>

Note how that last log entry is the deprecation of this ADR. You can, of course, expand your log parsing only to detect Status changes etc.

End

Hopefully, this gives you a taste of how easily useful documentation can be written, read and found. I’m interested to hear anyone else’s thoughts on whether they find this useful, or any other alternatives.

architecture, process, design

---

Canary Routing with Traefik in Nomad

23 Jun 2019

I wanted to implement canary routing for some HTTP services deployed via Nomad the other day, but rather than having the traffic split by weighting to the containers, I wanted to direct the traffic based on a header.

My first choice of tech was to use Fabio, but it only supports routing by URL prefix, and additionally with a route weight. While I was at JustDevOps in Poland, I heard about another router/loadbalancer which worked in a similar way to Fabio: Traefik.

While Traefik also doesn’t directly support canary routing, it is much more flexible than Fabio, also allowing request filtering based on HTTP headers. Traefik integrates with a number of container schedulers directly, but Nomad is not one of them. It does however also support using the Consul Service Catalog so that you can use it as an almost drop-in replacement for Fabio.

So let’s get to the setup. As usual, there is a complete repository on GitHub: Nomad Traefik Canary Routing.

Nomad

As usual, I am using my Hashibox Vagrant base image, and provisioning it as a single Nomad server and client node, using this script. I won’t dig into all the setup in that, as I’ve written it a few times now.

Consul

Consul is already running on the Hashibox base, so we have no further configuration to do.

Traefik

Traefik can be deployed as a Docker container, and either configured through a TOML file (yay, not yaml!) or with command line switches. As we only need a minimal configuration, I opted to use the command line.

The container exposes two ports we need to care about: 80 for incoming traffic to be routed, and 8080 for the UI, which are statically allocated to the host as 8000 and 8080 for this demo.

The command line configuration used is as follows:

  • --api - enable the UI.
  • --consulcatalog - Traefik has two ways to use Consul - --consul uses the KV store for service definitions, and --consulcatalog makes use Consul’s service catalogue.
  • --consulcatalog.endpoint=consul.service.consul:8500 as Consul is not running in the same container as Traefik, we need to tell it where Consul is listening, and as we have DNS Forwarding for *.consul domains, we use the address consul.service.consul. If DNS forwarding was not available, you could use the Nomad variable ${attr.unique.network.ip-address} to get the current task’s host’s IP.
  • --consulcatalog.frontEndRule disable the default rule - each service needs to specify traefik.frontend.rule.
  • --consulcatalog.exposedByDefault=false - lastly, we stop Traefik showing all services registered into consul, the will need to have the traefik.enable=true tag to be processed.

The entire job file is listed below:

job "traefik" {
  datacenters = ["dc1"]
  type = "service"

  group "loadbalancers" {
    count = 1

    task "traefik" {
      driver = "docker"

      config {
        image = "traefik:1.7.12"

        args = [
          "--api",
          "--consulcatalog",
          "--consulcatalog.endpoint=consul.service.consul:8500",
          "--consulcatalog.frontEndRule=''",
          "--consulcatalog.exposedByDefault=false"
        ]

        port_map {
          http = 80
          ui = 8080
        }
      }

      resources {
        network {
          port "http" { static = 8000 }
          port "ui" { static = 8080 }
        }

        memory = 50
      }

    }
  }
}

We register the job into Nomad, and then start on the backend services we will route to:

nomad job run jobs/traefik.nomad

The Backend Services

To demonstrate the services can be routed to correctly, we can use the containersol/k8s-deployment-strategies docker container. This image exposes an HTTP service which responds with the container’s hostname and the content of the VERSION environment variable, something like this:

$ curl http://echo.service.consul:8080
# Host: 23351e48dc98, Version: 1.0.0

We’ll start by making a standard nomad job for this container, and then update it to support canarying. The entire job is listed below:

job "echo" {
  datacenters = ["dc1"]
  type = "service"

  group "apis" {
    count = 3

    task "echo" {
      driver = "docker"

      config {
        image = "containersol/k8s-deployment-strategies"

        port_map {
          http = 8080
        }
      }

      env {
        VERSION = "1.0.0"
      }

      resources {
        network {
          port "http" { }
        }
      }

      service {
        name = "echo"
        port = "http"

        tags = [
          "traefik.enable=true",
          "traefik.frontend.rule=Host:api.localhost"
        ]

        check {
          type = "http"
          path = "/"
          interval = "5s"
          timeout = "1s"
        }
      }
    }
  }
}

The only part of interest in this version of the job is the service stanza, which is registering our echo service into consul, with a few tags to control how it is routed by Traefik:

service {
  name = "echo"
  port = "http"

  tags = [
    "traefik.enable=true",
    "traefik.frontend.rule=Host:api.localhost"
  ]

  check {
    type = "http"
    path = "/"
    interval = "5s"
    timeout = "1s"
  }
}

The traefik.enabled=true tag allows this service to be handled by Traefik (as we set exposedByDefault=false in Traefik), and traefik.frontend.rule=Host:api.localhost the rule means that any traffic with the Host header set to api.localhost will be routed to the service.

Which we can now run the job in Nomad:

nomad job run jobs/echo.nomad

Once it is up and running, we’ll get 3 instances of echo running which will be round-robin routed by Traefik:

$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: 1ac8a49cbaee, Version: 1.0.0
$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: 23351e48dc98, Version: 1.0.0
$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: c2f8a9dcab95, Version: 1.0.0

Now that we have working routing for the Echo service let’s make it canaryable.

Canaries

To show canary routing, we will create a second version of the service to respond to HTTP traffic with a Canary header.

The first change to make is to add in the update stanza, which controls how the containers get updated when Nomad pushes a new version. The canary parameter controls how many instances of the task will be created for canary purposes (and must be less than the total number of containers). Likewise, the max_parallel parameter controls how many containers will be replaced at a time when a deployment happens.

group "apis" {
  count = 3

+  update {
+    max_parallel = 1
+    canary = 1
+  }

  task "echo" {

Next, we need to modify the service stanza to write different tags to Consul when a task is a canary instance so that it does not get included in the “normal” backend routing group.

If we don’t specify at least 1 value in canary_tags, Nomad will use the tags even in the canary version - an empty canary_tags = [] declaration is not enough!

service {
  name = "echo"
  port = "http"
  tags = [
    "traefik.enable=true",
    "traefik.frontend.rule=Host:api.localhost"
  ]
+  canary_tags = [
+    "traefik.enable=false"
+  ]
  check {

Finally, we need to add a separate service stanza to create a second backend group which will contain the canary versions. Note how this group has a different name, and has no tags, but does have a set of canary_tags.

service {
  name = "echo-canary"
  port = "http"
  tags = []
  canary_tags = [
    "traefik.enable=true",
    "traefik.frontend.rule=Host:api.localhost;Headers: Canary,true"
  ]
  check {
    type = "http"
    path = "/"
    interval = "5s"
    timeout = "1s"
  }
}

The reason we need two service stanzas is that Traefik can only create backends based on the name of the service registered to Consul and not from a tag in that registration. If we just used one service stanza, then the canary version of the container would be added to both the canary backend and standard backend. I was hoping for traefik.backend=echo-canary to work, but alas no.

The entire updated jobfile is as follows:

job "echo" {
  datacenters = ["dc1"]
  type = "service"

  group "apis" {
    count = 3

    update {
      max_parallel = 1
      canary = 1
    }

    task "echo" {
      driver = "docker"

      config {
        image = "containersol/k8s-deployment-strategies"

        port_map {
          http = 8080
        }
      }

      env {
        VERSION = "1.0.0"
      }

      resources {
        network {
          port "http" { }
        }

        memory = 50
      }

      service {
        name = "echo-canary"
        port = "http"

        tags = []
        canary_tags = [
          "traefik.enable=true",
          "traefik.frontend.rule=Host:api.localhost;Headers: Canary,true"
        ]

        check {
          type = "http"
          path = "/"
          interval = "5s"
          timeout = "1s"
        }
      }

      service {
        name = "echo"
        port = "http"

        tags = [
          "traefik.enable=true",
          "traefik.frontend.rule=Host:api.localhost"
        ]
        canary_tags = [
          "traefik.enable=false"
        ]

        check {
          type = "http"
          path = "/"
          interval = "5s"
          timeout = "1s"
        }
      }
    }
  }
}

Testing

First, we will change the VERSION environment variable so that Nomad sees the job as changed, and we get a different response from HTTP calls to the canary:

env {
-  VERSION = "1.0.0"
+  VERSION = "2.0.0"
}

Now we will update the job in Nomad:

nomad job run jobs/echo.nomad

If we run the status command, we can see that the deployment has started, and there is one canary instance running. Nothing further will happen until we promote it:

$ nomad status echo
ID            = echo
Status        = running

Latest Deployment
ID          = 330216b9
Status      = running
Description = Deployment is running but requires promotion

Deployed
Task Group  Promoted  Desired  Canaries  Placed  Healthy  Unhealthy  Progress Deadline
apis        false     3        1         1       1        0          2019-06-19T11:19:31Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
dcff2555  82f6ea8b  apis        1        run      running  18s ago    2s ago
5b2710ed  82f6ea8b  apis        0        run      running  6m52s ago  6m26s ago
698bd8a7  82f6ea8b  apis        0        run      running  6m52s ago  6m27s ago
b315bcd3  82f6ea8b  apis        0        run      running  6m52s ago  6m25s ago

We can now test that the original containers still work, and that the canary version works:

$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: 1ac8a49cbaee, Version: 1.0.0
$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: 23351e48dc98, Version: 1.0.0
$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost'
#Host: c2f8a9dcab95, Version: 1.0.0
$ curl http://traefik.service.consul:8080 -H 'Host: api.localhost' -H 'Canary: true'
#Host: 496840b438f2, Version: 2.0.0

Assuming we are happy with our new version, we can tell Nomad to promote the deployment, which will remove the canary and start a rolling update of the three tasks, one at a time:

nomad deployment promote 330216b9

End

My hope is that the next version of Traefik will have better support for canary by header, meaning I could simplify the Nomad jobs a little, but as it stands, this doesn’t add much complexity to the jobs, and can be easily put into an Architecture Decision Record (or documented in a wiki page, never to be seen or read from again!)

infrastructure, vagrant, nomad, consul, traefik

---

Feature Toggles: Reducing Coupling

11 Jun 2019

One of the points I make in my Feature Toggles talk is that you shouldn’t be querying a toggle’s status all over your codebase. Ideally, each toggle gets checked in as few places as possible - preferably only one place. The advantage of doing this is that very little of your codebase needs to be coupled to the toggles (either the toggle itself or the library/system for managing toggles itself).

This post will go over several situations when that seems hard to do, namely: multiple services, multiple distinct areas of a codebase, and multiple times in a complex class or method. As in the previous post on this, we will be using Branch By Abstraction to do most of the heavy lifting.

Multiple Services

Multiple services interacting with the same feature toggle is a problematic situation to deal with, especially if multiple teams own the different services.

One of the main issues with this is trying to coordinate the two (or more) services. For example, if one team needs to switch off their implementation due to a problem, should the other services also get turned off too? To compound on this problem, what happens if one system can react to the toggle change faster than the other?

Services changing configuration at different speeds can also cause issues with handling in-flight requests too: if the message format is different when the toggle is on, will the receiving system be able to process a message produced when the toggle was in one state but consumed in the other state?

We can solve some of this by using separate toggles for each service (and they are not allowed to query the other service’s toggle state), and by writing the services so that they can handle both old format and new format requests at the same time.

For example, if we had a sending system which when the toggle is off will send this DTO:

public class PurchaseOptions
{
    public Address Address { get; set; }
}

And when the toggle is enabled, it will send the following DTO instead:

public class PurchaseOptions
{
    public BillingAddress Address { get; set; }
    public DeliveryAddress Address { get; set; }
}

To make the receiving system handle this, we deserialize the request into a DTO which contains all possible versions of the address, and then use the best version based on our own toggle state:

public class PurchaseOptionsRequest
{
    public Address Address { get; set; }
    public BillingAddress Address { get; set; }
    public DeliveryAddress Address { get; set; }
}

public class PurchaseController
{
    public async Task<PurchaseOptionsResponse> Post(PurchaseOptionsRequest request)
    {
        if (separateAddresses.Enabled)
        {
            var deliveryAddress = request.DeliveryAddress ?? request.Address;
            var billingAddress = request.BillingAddress ?? request.Address;

            ConfigureDelivery(deliveryAddress);
            CreateInvoice(billingAddress, deliveryAddress);
        }
        else
        {
            var address = request.Address ?? request.DeliveryAddress ?? request.BillingAddress;

            ConfigureDelivery(address)
            CreateInvoice(address, address);
        }
    }
}

Note how both sides of the toggle check read all three possible address fields, but try to use different fields first. This means that no matter whether the sending service has it’s toggle on or not, we will use the correct address.

Multiple Areas of the Codebase

To continue using the address example, we might have a UI, Controller and Handler, which all need to act differently based on the same toggle:

  • The UI needs to display either one or two address editors
  • The controller needs to have different validation logic for multiple addresses
  • The Command Handler will need to dispatch different values

We can solve this all by utilising Branch By Abstraction and Dependency Injection to make most of the codebase unaware that a feature toggle exists. Even the implementations won’t need to know about the toggles.

public class Startup
{
    public void ConfigureContainer(ServiceRegistry services)
    {
        if (separateAddresses.Enabled) {
            services.Add<IAddressEditor, MultiAddressEditor>();
            services.Add<IRequestValidator, MultiAddressValidator>();
            services.Add<IDeliveryHandler, MultiAddressDeliveryHandler>();
        }
        else {
            services.Add<IAddressEditor, SingleAddressEditor>();
            services.Add<IRequestValidator, SingleAddressValidator>();
            services.Add<IDeliveryHandler, SingleAddressDeliveryHandler>();
        }
    }
}

Let’s look at how one of these might work. The IRequestValidator has a definition like so:

public interface IRequestValidator<TRequest>
{
    public IEnumerable<string> Validate(TRequest request);
}

There is a middleware in the API request pipeline which will pick the right validator out of the container, based on the request type being processed. We implement two validators, once for the single address, and one for multiaddress:

public class SingleAddressValidator : IRequestValidator<SingleAddressRequest>
{
    public IEnumerable<string> Validate(SingleAddressRequest request)
    {
        //complex validation logic..
        if (request.Address == null)
            yield return "No Address specified";

        if (PostCode.Validate(request.Address.PostCode) == false)
            yield return "Invalid Postcode";
    }
}

public class MultiAddressValidator : IRequestValidator<MultiAddressRequest>
{
    public IEnumerable<string> Validate(MultiAddressRequest request)
    {
        var billingMessages = ValidateAddress(request.BillingAddress);

        if (billingMessages.Any())
            return billingMessages;

        if (request.DifferentDeliveryAddress)
            return ValidateAddress(request.DeliveryAddress);
    }
}

The implementations themselves don’t need to know about the state of the toggle, as the container and middleware take care of picking the right implementation to use.

Multiple Places in a Class/Method

If you have a single method (or class) which needs to check the toggle state in multiple places, you can also use the same Branch by Abstraction technique as above, by creating a custom interface and pair of implementations, which contain all the functionality which changes.

For example, if we have a method for finding an offer for a customer’s basket, which has a few separate checks that the toggle is enabled in it:

public SuggestedBasket CreateOffer(CreateOfferCommand command)
{
    if (newFeature.Enabled) {
        ExtraPreValidation(command).Throw();
    } else {
        StandardPreValidation(command).Throw();
    }

    var offer = SelectBestOffer(command.Items);

    if (offer == null && newFeature.Enabled) {
        offer = FindAlternativeOffer(command.Customer, command.Items);
    }

    return SuggestedBasket
        .From(command)
        .With(offer);
}

We can extract an interface for this, and replace the toggle specific parts with calls to the interface instead:

public interface ICreateOfferStrategy
{
    IThrowable PreValidate(CreateOfferCommand command);
    Offer AlternativeOffer(CreateOfferCommand command, Offer existingOffer);
}

public class DefaultOfferStrategy : ICreateOfferStrategy
{
    public IThrowable PreValidate(CreateOfferCommand command)
    {
        return StandardPreValidation(command);
    }

    public Offer AlternativeOffer(CreateOfferCommand command, Offer existingOffer)
    {
        return existingOffer;
    }
}

public class DefaultOfferStrategy : ICreateOfferStrategy
{
    public IThrowable PreValidate(CreateOfferCommand command)
    {
        return ExtraPreValidation(command);
    }

    public Offer AlternativeOffer(CreateOfferCommand command, Offer existingOffer)
    {
        if (existingOffer != null)
            return existingOffer;

        return TryFindAlternativeOffer(command.Customer, command.Items, offer);
    }
}

public class OfferBuilder
{
    private readonly ICreateOfferStrategy _strategy;

    public OfferBuilder(ICreateOfferStrategy strategy)
    {
        _strategy = strategy;
    }

    public SuggestedBasket CreateOffer(CreateOfferCommand command)
    {
        _strategy.PreValidation(command).Throw();

        var offer = SelectBestOffer(command.Items);

        offer = _strategy.AlternativeOffer(command, offer);

        return SuggestedBasket
            .From(command)
            .With(offer);
    }
}

Now that we have done this, our CreateOffer method has shrunk dramatically and no longer needs to know about the toggle state, as like the rest of our DI examples, the toggle can be queried once in the startup of the service and the correct ICreateOfferStrategy implementation registered into the container.

End

Hopefully, this post will give a few insights into different ways of reducing the number of calls to your feature toggling library, and prevent you scattering lots of if statements around the codebase!

featuretoggles, c#, di, microservices

---