Nomad Good, Kubernetes Bad

21 Nov 2019

I will update this post as I learn more (both positive and negative), and is here to be linked to when people ask me why I don’t like Kubernetes, and why I would pick Nomad in most situations if I chose to use an orchestrator at all.

TLDR: I don’t like complexity, and Kubernetes has more complexity than benefits.

Operational Complexity

Operating Nomad is very straight forward. There are very few moving parts, so the number of things which can go wrong is significantly reduced. No external dependencies are required to run it, and there is only one binary to use. You run 3-5 copies in Server mode to manage the cluster and as many as you want running in Client mode to do the actual work. You can add Consul if you want service discovery, but it’s optional. More on that later.

Compare this to operating a Kubernetes cluster. There are multiple Kubernetes orchestration projects, tools, and companies to get clusters up and running, which should be an indication of the level of complexity involved. Once you have the cluster set up, you need to keep it running. There are so many moving parts (Controller Manager, Scheduler, API Server, Etcd, Kubelets) that it quickly becomes a full-time job to keep the cluster up and running. Use a cloud service to run Kubernetes, and if you must use your own infrastructure, pay someone else to manage it. It’s cheaper in the long run. Trust me.

Deployment

Nomad, being a single binary, is easy to deploy. If you want to use Terraform to create a cluster, Hashicorp provides modules for both AWS and Azure. Alternatively, you can do everything yourself, as it’s just keeping one binary running on hosts, and a bit of network/DNS config to get them talking to each other.

By comparison, Kubernetes has a multitude of tools to help you deploy a cluster. Still, while it gives you a lot of flexibility in choice, you also have to hope that the tool continues to exist and that there is enough community/company/documentation about that specific tool to help you when something goes wrong.

Upgrading The Cluster

Upgrading Nomad involves doing a rolling deployment of the Servers and Clients. If you are using the Hashicorp Terraform module, you re-apply the module with the new AMI ID to use, and then delete nodes (gracefully!) from the cluster and let the AutoScaleGroup take care of bringing new nodes up. If you need to revert to an older version of Nomad, you follow the same process.

When it comes to Kubernetes, please pay someone else to do it. It’s not a fun process. The process will differ depending on which cluster management tool you are using, and you also need to think about updates to etcd and managing state in the process. There is a nice long document on how to upgrade etcd.

Debugging a Cluster

As mentioned earlier, Nomad has a small number of moving parts. There are three ports involved (HTTP, RPC and Gossip), so as long as those ports are open and reachable, Nomad should be operable. Then you need to keep the Nomad agents alive. That’s pretty much it.

Where to start for Kubernetes? As many Kubernetes Failure Stories point out: it’s always DNS. Or etcd. Or Istio. Or networking. Or Kubelets. Or all of these.

Local Development

To run Nomad locally, you use the same binary as the production clusters, but in dev mode: nomad agent -dev. To get a local cluster, you can spin up some Vagrant boxes instead. I use my Hashibox Vagrant box to do this when I do conference talks and don’t trust the wifi to work.

To run Kubernetes locally to test things, you need to install/deploy MiniKube, K3S, etc. The downside to this approach is that the environment is significantly different to your real Kubernetes cluster, and you can end up where a deployment works in one, but not the other, which makes debugging issues much harder.

Features & Choice

Nomad is relatively light on built-in features, which allows you the choice of what features to add, and what implementations of the features to use. For example, it is pretty popular to use Consul for service discovery, but if you would rather use Eureka, or Zookeeper, or even etcd, that is fine, but you lose out on the seamless integration with Nomad that other Hashicorp tools have. Nomad also supports Plugins if you want to add support for your favourite tool.

By comparison, Kubernetes does everything, but like the phrase “Jack of all trades, master of none”, often you will have to supplement the inbuilt features. The downside to this is that you can’t switch off Kubernetes features you are not using, or don’t want. So if you add Vault for secret management, the Kubernetes Secrets are still available, and you have to be careful that people don’t use them accidentally. The same goes for all other features, such as Load Balancing, Feature Toggles, Service Discovery, DNS, etc.

Secret Management

Nomad doesn’t provide a Secret Management solution out of the box, but it does have seamless Vault integration, and you are also free to use any other Secrets As A Service tool you like. If you do choose Vault, you can either use it directly from your tasks or use Nomad’s integration to provide the secrets to your application. It can even send a signal (e.g. SIGINT etc.) to your process when the secrets need re-reading.

Kubernetes, on the other hand, provides “Secrets”. I put the word “secrets” in quotes because they are not secrets at all. The values are stored encoded in base64 in etcd, so anyone who has access to the etcd cluster has access to all the secrets. The official documentation suggests making sure only administrators have access to the etcd cluster to solve this. Oh, and if you can deploy a container to the same namespace as a secret, you can reveal it by writing it to stdout.

Kubernetes secrets are not secret, just “slightly obscured.”

If you want real Secrets, you will almost certainly use Vault. You can either run it inside or outside of Kubernetes, and either use it directly from containers via it’s HTTPS API or use it to populate Kubernetes Secrets. I’d avoid populating Kubernetes Secrets if I were you.

Support

If Nomad breaks, you can either use community support or if you are using the Enterprise version, you have Hashicorp’s support.

When Kubernetes breaks, you can either use community support or find and buy support from a Kubernetes management company.

The main difference here is “when Kubernetes breaks” vs “if Nomad breaks”. The level of complexity in Kubernetes makes it far more likely to break, and that much harder to debug.

nomad, infrastructure, kubernetes

---

Creating a Vault instance with a TLS Consul Cluster

06 Oct 2019

So we want to set up a Vault instance, and have it’s storage be a TLS based Consul cluster. The problem is that the Consul cluster needs Vault to create the certificates for TLS, which is quite the catch-22. Luckily for us, quite easy to solve:

  1. Start a temporary Vault instance as an intermediate ca
  2. Launch Consul cluster, using Vault to generate certificates
  3. Destroy temporary Vault instance
  4. Start a permanent Vault instance, with Consul as the store
  5. Reprovision the Consul cluster with certificates from the new Vault instance

Sequence diagram of the previous numbered list

There is a repository on Github with all the scripts used, and a few more details on some options.

Assumptions:

The Host machine needs the following software available in your PATH:

You have a TLS Certificate you can use to create an intermediate CA with. See this blog post for How to create a local CA

Running

The run.sh script will do all of this for you, but an explanation of the steps is below:

  1. Start a Temporary Vault instance

     echo '
     storage "inmem" {}
     listener "tcp" {
       address = "0.0.0.0:8200"
       tls_disable = 1
     }' > "vault/temp_vault.hcl"
    
     vault server -config="vault/temp_vault.hcl" &
     echo "$!" > vault.pid
    
     export VAULT_TOKEN=$(./configure_vault.sh | tail -n 1)
    
  2. Generate a Vault token for the Consul machines to use to authenticate with Vault

     export CONSUL_VAULT_TOKEN=$(vault write -field=token -force auth/token/create)
    
  3. Launch 3 Consul nodes (uses the CONSUL_VAULT_TOKEN variable)

     vagrant up
    

    The vagrantfile just declares 3 identical machines:

     Vagrant.configure(2) do |config|
       config.vm.box = "pondidum/hashibox"
    
       config.vm.provision "consul",
         type: "shell",
         path: "./provision.sh",
         env: {
             "VAULT_TOKEN" => ENV["CONSUL_VAULT_TOKEN"]
         }
    
       config.vm.define "c1" do |c1|
         c1.vm.hostname = "consul1"
       end
    
       config.vm.define "c2" do |c2|
         c2.vm.hostname = "consul2"
       end
    
       config.vm.define "c3" do |c3|
         c3.vm.hostname = "consul3"
       end
     end
    

    The provisioning script just reads a certificate from Vault, and writes out pretty much the same configuration as in the last post on creating a TLS enabled Consul Cluster, but you can view it in the repository for this demo too.

  4. Create a local Consul server to communicate with the cluster:

     ./local_consul.sh
    

    This is done so that the Vault instance can always communicate with the Consul cluster, no matter which Consul node we are reprovisioning later. In a production environment, you would have this Consul server running on each machine that Vault is running on.

  5. Stop the temporary Vault instance now that all nodes have a certificate

    kill $(cat vault.pid)
    
  6. Start the persistent Vault instance, using the local Consul agent

     echo '
     storage "consul" {
       address = "localhost:8501"
       scheme = "https"
     }
     listener "tcp" {
       address = "0.0.0.0:8200"
       tls_disable = 1
     }' > "$config_dir/persistent_vault.hcl"
    
     vault server -config="$config_dir/persistent_vault.hcl" > /dev/null &
     echo "$!" > vault.pid
    
     export VAULT_TOKEN=$(./configure_vault.sh | tail -n 1)
    
  7. Generate a new Vault token for the Consul machines to use to authenticate with Vault (same as step 2)

     export CONSUL_VAULT_TOKEN=$(vault write -field=token -force auth/token/create)
    
  8. Reprovision the Consul nodes with new certificates

     vagrant provision c1 --provision-with consul
     vagrant provision c2 --provision-with consul
     vagrant provision c3 --provision-with consul
    
  9. Profit

    To clean up the host’s copy of Vault and Consul, you can run this:

     kill $(cat vault.pid)
     kill $(cat consul.pid)
    

Summary & Further Actions

Luckily, this is the kind of thing that should only need doing once (or once per isolated environment). When running in a real environment, you will also want to set up:

  • ACL in Consul which locks down the KV storage Vault uses to only be visible/writeable by Vault
  • Provisioning the VAULT_TOKEN to the machines in a secure fashion
  • Periodic refresh of the Certificates uses in the Consul cluster

consul, vault, infrastructure, security, tls

---

Consul DNS Fowarding in Ubuntu, revisited

24 Sep 2019

I was recently using my Hashibox for a test, and I noticed the DNS resolution didn’t seem to work. This was a bit worrying, as I have written about how to do DNS resolution with Consul forwarding in Ubuntu, and apparently something is wrong with how I do it. Interestingly, the Alpine version works fine, so it appears there is something not quite working with how I am configuring Systemd-resolved.

So this post is how I figured out what was wrong, and how to do DNS resolution with Consul forwarding on Ubuntu properly!

The Problem

If Consul is running on the host, I can only resolve .consul domains, and if Consul is not running, I can resolve anything else. Clearly I have configured something wrong!

To summarise, I want to be able to resolve 3 kinds of address:

  • *.consul addresses should be handled by the local Consul instance
  • $HOSTNAME.mshome.net should be handled by the Hyper-V DNS server (running on the Host machine)
  • reddit.com public DNS should be resolved properly

Discovery

To make sure that hostname resolution even works by default, I create a blank Ubuntu box in Hyper-V, using Vagrant.

Vagrant.configure(2) do |config|
  config.vm.box = "bento/ubuntu-16.04"
  config.vm.hostname = "test"
end

I set the hostname so that I can test that dns resolution works from the host machine to the guest machines too. I next bring up the machine, SSH into it, and try to dig my hostmachine’s DNS name (spectre.mshome.net):

> vagrant up
> vagrant ssh
> dig spectre.mshome.net

; <<>> DiG 9.10.3-P4-Ubuntu <<>> spectre.mshome.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12333
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;spectre.mshome.net.            IN      A

;; ANSWER SECTION:
Spectre.mshome.net.     0       IN      A       192.168.181.161

;; Query time: 0 msec
;; SERVER: 192.168.181.161#53(192.168.181.161)
;; WHEN: Mon Sep 23 21:57:26 UTC 2019
;; MSG SIZE  rcvd: 70

> exit
> vagrant destroy -f

As you can see, the host machine’s DNS server responds with the right address. Now that I know that this should work, we can tweak the Vagrantfile to start an instance of my Hashibox:

Vagrant.configure(2) do |config|
  config.vm.box = "pondidum/hashibox"
  config.vm.hostname = "test"
end

When I run the same command sin this box, I get a slighty different response:

; <<>> DiG 9.10.3-P4-Ubuntu <<>> spectre.mshome.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 57216
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;spectre.mshome.net.            IN      A

;; AUTHORITY SECTION:
consul.                 0       IN      SOA     ns.consul. hostmaster.consul. 1569276784 3600 600 86400 0

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Sep 23 22:13:04 UTC 2019
;; MSG SIZE  rcvd: 103

As intended, the DNS server on localhost responded…but it looks like Consul answered, not the inbuilt dns server (systemd-resolved), as I intended.

The reason for this is that I am running Consul’s DNS endpoint on 8600, and Systemd-Resolved cannot send requests to anything other than port 53, so I use iptables to redirect the traffic from port 53 to 8600, which means any local use of DNS will always be sent to Consul.

The reason it works when Consul is not running is that we have both 127.0.0.1 specified as a nameserver, and a fallback set to be the eth0’s Gateway, so when Consul doesn’t respond, the request hits the default DNS instead.

The Solution: Dnsmasq.

Basically, stop using systemd-resolved and use something that has a more flexible configuration. Enter Dnsmasq.

Starting from the blank Ubuntu box, I install dnsmasq, and disable systemd-resolved. Doing this might prevent any DNS resolutio working for a while…

sudo apt-get install -yq dnsmasq
sudo systemctl disable systemd-resolved.service

If you would rather not disable systemd-resolved entirely, you can use these two lines instead to just switch off the local DNS stub:

echo "DNSStubListener=no" | sudo tee --append /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved

Next I update /etc/resolv.conf to not be managed by Systemd, and point to where dnsmasq will be running:

sudo rm /etc/resolv.conf
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

The reason for deleting the file is that it was symlinked to the Systemd-Resolved managed file, so that link needed to be broken first to prevent Systemd interfering.

Lastly a minimal configuration for dnsmasq:

echo '
port=53
resolv-file=/var/run/dnsmasq/resolv.conf
bind-interfaces
listen-address=127.0.0.1
server=/consul/127.0.0.1#8600
' | sudo tee /etc/dnsmasq.d/default

sudo systemctl restart dnsmasq

This config does a few things, the two most important lines are:

  • resolv-file=/var/run/dnsmasq/resolv.conf which is pointing to the default resolv.conf written by dnsmasq. This file contains the default nameserver supplied by the default network connection, and I want to use this as a fallback for anything dnsmasq cannot resolve directly (which will be everything, except .consul). In my case, the content of this file is just nameserver 192.168.181.161.

  • server=/consul/127.0.0.1#8600 specifies that any address ending in .consul should be forwarded to Consul, running at 127.0.0.1 on port 8600. No more iptables rules!

Testing

Now that I have a (probably) working DNS system, let’s look at testing it properly this time. There are 3 kinds of address I want to test:

  • Consul resolution, e.g. consul.service.consul should return the current Consul instance address.
  • Hostname resolution, e.g. spectre.mshome.net should resolve to the machine hosting the VM.
  • Public resolution, e.g. reddit.com should resolve to…reddit.

I also want to test that the latter two cases work when Consul is not running too.

So let’s write a simple script to make sure these all work. This way I can reuse the same script on other machines, and also with other VM providers to check DNS works as it should. The entire script is here:

local_domain=${1:-mshome.net}
host_machine=${2:-spectre}

consul agent -dev -client 0.0.0.0 -bind '{{ GetInterfaceIP "eth0" }}' > /dev/null &
sleep 1

consul_ip=$(dig consul.service.consul +short)
self_ip=$(dig $HOSTNAME.$local_domain +short | tail -n 1)
host_ip=$(dig $host_machine.$local_domain +short | tail -n 1)
reddit_ip=$(dig reddit.com +short | tail -n 1)

kill %1

[ "$consul_ip" == "" ] && echo "Didn't get consul ip" >&2 && exit 1
[ "$self_ip" == "" ] && echo "Didn't get self ip" >&2 && exit 1
[ "$host_ip" == "" ] && echo "Didn't get host ip" >&2 && exit 1
[ "$reddit_ip" == "" ] && echo "Didn't get reddit ip" >&2 && exit 1

echo "==> Consul Running: Success!"

consul_ip=$(dig consul.service.consul +short | tail -n 1)
self_ip=$(dig $HOSTNAME.$local_domain +short | tail -n 1)
host_ip=$(dig $host_machine.$local_domain +short | tail -n 1)
reddit_ip=$(dig reddit.com +short | tail -n 1)

[[ "$consul_ip" != *";; connection timed out;"* ]] && echo "Got a consul ip ($consul_ip)" >&2 && exit 1
[ "$self_ip" == "" ] && echo "Didn't get self ip" >&2 && exit 1
[ "$host_ip" == "" ] && echo "Didn't get host ip" >&2 && exit 1
[ "$reddit_ip" == "" ] && echo "Didn't get reddit ip" >&2 && exit 1

echo "==> Consul Stopped: Success!"

exit 0

What this does is:

  1. Read two command line arguments, or use defaults if not specified
  2. Start Consul as a background job
  3. Query 4 domains, storing the results
  4. Stop Consul (kill %1)
  5. Check an IP address came back for each domain
  6. Query the same 4 domains, storing the results
  7. Check that a timeout was received for consul.service.consul
  8. Check an IP address came back for the other domains

To further prove that dnsmasq is forwarding requests correctly, I can include two more lines to /etc/dnsmasq.d/default to enable logging, and restart dnsmasq

echo "log-queries" | sudo tee /etc/dnsmasq.d/default
echo "log-facility=/var/log/dnsmasq.log" | sudo tee /etc/dnsmasq.d/default
sudo systemctl restart dnsmasq
dig consul.service.consul

Now I can view the log file and check that it received the DNS query and did the right thing. In this case, it recieved the consul.service.consul query, and forwarded it to the local Consul instance:

Sep 24 06:30:50 dnsmasq[13635]: query[A] consul.service.consul from 127.0.0.1
Sep 24 06:30:50 dnsmasq[13635]: forwarded consul.service.consul to 127.0.0.1
Sep 24 06:30:50 dnsmasq[13635]: reply consul.service.consul is 192.168.181.172

I don’t tend to keep DNS logging on in my Hashibox as the log files can grow very quickly.

Wrapping Up

Now that I have proven my DNS resolution works (I think), I have rolled it back into my Hashibox, and can now use machine names for setting up clusters, rather than having to specify IP addresses initially.

consul, dns, infrastructure

---

Creating a TLS enabled Consul cluster

14 Sep 2019

This post is going to go through how to set up a Consul cluster to communicate over TLS. I will be using Vagrant to create three machines locally, which will form my cluster, and in the provisioning step will use Vault to generate the certificates needed.

How to securely communicate with Vault to get the TLS certificates is out of scope for this post.

Host Configuration

Unless you already have Vault running somewhere on your network, or have another mechanism to generate TLS certificates for each machine, you’ll need to start and configure Vault on the Host machine. I am using my Vault Dev Intermediate CA script from my previous post.

To set this up, all I need to do is run this on the host machine, which starts Vault in a docker container, and configures it as an intermediate certificate authority:

./run_int.sh

I also have DNS on my network setup for the tecra.xyz domain so will be using that to test with.

Consul Machine Configuration

The Vagrantfile is very minimal - I am using my Hashibox (be aware the libvirt provider for this might not work, for some reason vagrant package with libvirt produces a non-bootable box).

Vagrant.configure(2) do |config|
  config.vm.box = "pondidum/hashibox"
  config.vm.provision "consul", type: "shell", path: "./provision.sh"

  config.vm.define "c1" do |c1|
    c1.vm.hostname = "consul1"
  end

  config.vm.define "c2" do |c2|
    c2.vm.hostname = "consul2"
  end

  config.vm.define "c3" do |c3|
    c3.vm.hostname = "consul3"
  end
end

The hashibox script already has all the tools we’ll need installed already: Consul, Vault, and jq.

First up, we request a certificate from Vault to use for Consul - How you get this certificate in a secure manner in a production environment is up to you. There is a catch-22 here for me, in that in a production environment I use Vault with Consul as it’s backing store…but Consul needs Vault to start! I’ll go over how I get around this in a future post.

export VAULT_ADDR="http://vault.tecra.xyz:8200"
export VAULT_TOKEN="vault"

response=$(vault write pki/issue/cert -format=json common_name=$HOSTNAME.tecra.xyz alt_names="server.dc1.consul")
config_dir="/etc/consul.d"

The first thing to note is that we have specified an alt_names for the certificate - you must have a SAN of server.$DC.$DOMAIN so either server.dc1.consul or server.euwest1.tecra.xyz, and the server prefix is required!.

Next, we need to take all the certificates from the response and write them to the filesystem.

mkdir -p "$config_dir/ca"

for (( i=0; i<$(echo "$response" | jq '.data.ca_chain | length'); i++ )); do
  cert=$(echo "$response" | jq -r ".data.ca_chain[$i]")
  name=$(echo "$cert" | openssl x509 -noout -subject -nameopt multiline | sed -n 's/ *commonName *= //p' | sed 's/\s//g')

  echo "$cert" > "$config_dir/ca/$name.pem"
done

echo "$response" | jq -r .data.private_key > $config_dir/consul.key
echo "$response" | jq -r .data.certificate > $config_dir/consul.crt
echo "$response" | jq -r .data.issuing_ca >> $config_dir/consul.crt

The for loop iterates through all of the certificates returned in the ca_chain and writes them into a ca directory. We use openssl to get the name of the certificate, so the files are named nicely!

Finally, it writes the private_key for the node’s certificate to consul.key, and both the certificate and issuing_ca to the consul.crt file.

Now for the consul.json. To setup a secure cluster, first of all we need to add the certificate configuration, pointing to the files we wrote earlier:

"ca_path": "$config_dir/ca/",
"cert_file": "$config_dir/consul.crt",
"key_file": "$config_dir/consul.key",

We will also disable the HTTP port, and enable the HTTPS port:

"ports": {
    "http": -1,
    "https": 8501
}

Finally, we need to add some security settings. First is encrypt, which sets that the key that all Consul nodes will use to encrypt their communications. It must match on all nodes. The easiest way to generate this is just run consul keygen and use the value that produces.

  • "encrypt": "oNMJiPZRlaP8RnQiQo9p8MMK5RSJ+dXA2u+GjFm1qx8=":

    The key the cluster will use to encrypt all it’s traffic. It must be the same on all nodes, and the easiest way to generate the value is to use the output of consul keygen.

  • "verify_outgoing": true:

    All the traffic leaving this node will be encrypted with the TLS certificates. However, the node will still accept non-TLS traffic.

  • "verify_incoming_rpc": true:

    All the gossip traffic arriving at this node must be signed with an authority in the ca_path.

  • "verify_incoming_https": false:

    We are going to use the Consul Web UI, so we want to allow traffic to hit the API without a client certificate. If you are using the UI from a non-server node, you can set this to true.

  • "verify_server_hostname": true:

    Set Consul to verify outgoing connections have a hostname in the format of server.<datacenter>.<domain>. From the docs: “This setting is critical to prevent a compromised client from being restarted as a server and having all cluster state including all ACL tokens and Connect CA root keys replicated to it”

The complete config we will use is listed here:

(
cat <<-EOF
{
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "data_dir": "/var/consul",
    "leave_on_terminate": true,
    "rejoin_after_leave": true,
    "retry_join": ["consul1", "consul2", "consul3"],
    "server": true,
    "ui": true,
    "encrypt": "oNMJiPZRlaP8RnQiQo9p8MMK5RSJ+dXA2u+GjFm1qx8=",
    "verify_incoming_rpc": true,
    "verify_incoming_https": false,
    "verify_outgoing": true,
    "verify_server_hostname": true,
    "ca_path": "$config_dir/ca/",
    "cert_file": "$config_dir/consul.crt",
    "key_file": "$config_dir/consul.key",
    "ports": {
        "http": -1,
        "https": 8501
    }
}
EOF
) | sudo tee $config_dir/consul.json

Lastly, we’ll make a systemd service unit to start consul:

(
cat <<-EOF
[Unit]
Description=consul agent
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/bin/consul agent -config-file=$config_dir/consul.json -bind '{{ GetInterfaceIP "eth0" }}'
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target
EOF
) | sudo tee /etc/systemd/system/consul.service

sudo systemctl daemon-reload
sudo systemctl enable consul.service
sudo systemctl start consul

As the machines we are starting also have docker networks (and potentially others), our startup line specifies to bind to the eth0 network, using a Consul Template.

Running

First, we need to run our intermediate CA, then provision our three machines:

./run_int.sh
vagrant up

After a few moments, you should be able to curl the consul ui (curl https://consul1.tecra.xyz:8501) or open https://consul1.tecra.xyz:8501 in your browser.

Note, however, the if your root CA is self-signed, like mine is, some browsers (such as FireFox) won’t trust it, as they won’t use your machine’s Trusted Certificate Store, but their own in built store. You can either accept the warning or add your root certificate to the browser’s store.

Testing

Now that we have our cluster seemingly running with TLS, what happens if we try to connect a Consul client without TLS to it? On the host machine, I just run a single node, and tell it to connect to one of the cluster nodes:

consul agent \
  -join consul1.tecra.xyz \
  -bind '{{ GetInterfaceIP "eth0" }}' \
  -data-dir /tmp/consul

The result of this is a refusal to connect, as the cluster has TLS configured, but this instance does not:

==> Starting Consul agent...
==> Log data will now stream in as it occurs:
==> Joining cluster...
==> 1 error occurred:
  * Failed to join 192.168.121.231: Remote state is encrypted and encryption is not configured

Success!

In the next post, I’ll go through how we can set up a Vault cluster which stores its data in Consul, but also provision that same Consul cluster with certificates from the Vault instance!

vault, security, tls, consul

---

Using Vault as a Development CA

25 Aug 2019

Often when developing or testing some code, I need (or want) to use SSL, and one of the easiest ways to do that is to use Vault. However, it gets pretty annoying having to generate a new CA for each project, and import the CA cert into windows (less painful in Linux, but still annoying), especially as I forget which cert is in use, and accidentally clean up the wrong ones.

My solution has been to generate a single CA certificate and PrivateKey, import this into my Trusted Root Certificate Store, and then whenever I need a Vault instance, I just setup Vault to use the existing certificate and private key. The documentation for how to do this seems somewhat lacking, so here’s how I do it.

Things you’ll need:

  • Docker
  • Vault cli
  • JQ

Generating the Root Certificate

First we need to create a Certificate, which we will do using the Vault docker container, and our local Vault CLI. We start the docker container in the background, and mark it for deletion when it stops (--rm):

container=$(docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest)

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

certs_dir="./ca"
max_ttl="87600h" # 10 years why not

mkdir -p $certs_dir
rm -rf $certs_dir/*.*

vault secrets enable pki
vault secrets tune -max-lease-ttl=$max_ttl pki

Finally, we generate a certificate by writing to the pki/root/generate/exported path. If the path ends with exported the Private Key is returned too. If you specify /internal then the Private Key is stored internally to Vault, and never accessible.

result=$(vault write -format "json" \
  pki/root/generate/exported \
  common_name="Local Dev CA" \
  alt_names="localhost,mshome.net" \
  ttl=$max_ttl)

echo "$result" > $certs_dir/response.json
echo "$result" | jq -r .data.certificate > $certs_dir/ca.crt
echo "$result" | jq -r .data.private_key > $certs_dir/private.key

docker stop $container

We put the entire response into a json file just incase there is something interesting we want out of it later, and store the certificate and private key into the same directory too. Note for the certificate’s alt_names I have specified both localhost and mshome.net, which is the domain that Hyper-V machines use.

Lastly, we can now import the root CA into our machine/user’s Trusted Root Certification Authorities store, meaning our later uses of this certificate will be trusted by our local machine.

Creating a Vault CA

As before, we use a Docker container to run the Vault instance, except this time we import the existing CA certificate into the PKI backend. The first half of the script (run_ca.sh) is pretty much the same as before, except we don’t delete the contents of the ./ca directory, and our certificate max_ttl is much lower:

docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

certs_dir="./ca"
max_ttl="72h"

vault secrets enable pki
vault secrets tune -max-lease-ttl=$max_ttl pki

The last part is to read in the certificate and private key, bundle them together, and configure the pki backend to use them, and add a single role to use for issuing certificates:

pem=$(cat $certs_dir/ca.crt $certs_dir/private.key)

vault write pki/config/ca pem_bundle="$pem"

vault write pki/roles/cert \
  allowed_domains=localhost,mshome.net \
  allow_subdomains=true \
  max_ttl=$max_ttl

Also note how we don’t stop the docker container either. Wouldn’t be much of a CA if it stopped the second it was configured…

Creating a Vault Intermediate CA

Sometimes, I want to test that a piece of software works when I have issued certificates from an Intermediate CA, rather than directly from the root. We can configure Vault to do this too, with a modified script which this time we start two PKI secret backends, one to act as the root, and onc as the intermediate:

#!/bin/bash

set -e

docker run -d --rm  --cap-add=IPC_LOCK -p 8200:8200 -e "VAULT_DEV_ROOT_TOKEN_ID=vault" vault:latest

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

# create root ca
certs_dir="./ca"
pem=$(cat $certs_dir/ca.crt $certs_dir/private.key)

vault secrets enable -path=pki_root pki
vault secrets tune -max-lease-ttl=87600h pki_root
vault write pki_root/config/ca pem_bundle="$pem"

# create the intermediate
vault secrets enable pki
vault secrets tune -max-lease-ttl=43800h pki

csr=$(vault write pki/intermediate/generate/internal \
  -format=json common_name="Spectre Dev Intermdiate CA" \
  | jq -r .data.csr)

intermediate=$(vault write pki_root/root/sign-intermediate \
  -format=json csr="$csr" format=pem_bundle ttl=43800h \
  | jq -r .data.certificate)

chained=$(echo -e "$intermediate\n$(cat $certs_dir/ca.crt)")

vault write pki/intermediate/set-signed certificate="$chained"

echo "$intermediate" > intermediate.crt

vault write pki/roles/cert \
  allowed_domains=localhost,mshome.net \
  allow_subdomains=true \
  max_ttl=43800h

# destroy the temp root
vault secrets disable pki_root

We use the pki_root backend to sign a CSR from the pki (intermediate) backend, and once the signed response is stored in pki, we delete the pki_root backend, as it is no longer needed for our Development Intermediate CA.

Issuing Certificates

We can now use the cert role to issue certificates for our applications, which I have in a script called issue.sh:

#!/bin/bash

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="vault"

vault write \
  -format=json \
  pki/issue/cert \
  common_name=$1.mshome.net

This script I usually use with jq to do something useful with:

response=$(./issue.sh consul)

cert=$(echo "$response" | jq -r .data.certificate)
key=$(echo "$response" | jq -r .data.private_key)

Cleaning Up

When I have finished with an application or demo, I can just stop the Vault container, and run the run_ca.sh script again if I need Vault for another project.

vault, security, tls

---