An NGINX and DNS based outage

I recently encountered a behaviour in Nginx that I didn’t expect and caused a production outage in the process. While I would love to blame DNS for this, as it’s usually the cause of most network-related issues, in this case, the fault lies with Nginx. I was running a very simple Nginx proxy, relaying an internal service to the outside world. The internal service is behind an AWS ALB, and the Nginx configuration was proxying to the ALB’s FQDN:...

April 23, 2022 · 3 min

Observability with Infrastructure as Code

This article was originally published on the Pulumi blog. When using the Pulumi Automation API to create applications which can provision infrastructure, it is very handy to be able to use observability techniques to ensure the application functions correctly and to help see where performance bottlenecks are. One of the applications I work on creates a VPC and Bastion host and then stores the credentials into a Vault instance....

March 1, 2021 · 4 min

Testing Immutable Infrastructure

In my previous post, I glossed over one of the most important and useful parts of Immutable Infrastructure: Testability. There are many kinds of tests we can write for our infrastructure, but they should all be focused on the machine/service and maybe it’s nearest dependencies, not the entire system. While this post focuses on testing a full machine (both locally in a VM, and remotely as an Amazon EC2 instance), it is also possible to do most of the same kind of tests against a Docker container....

January 1, 2019 · 17 min

Terraform, Kinesis Streams, Lambda and IAM problems

I hit an problem the recently with Terraform, when I was trying to hook up a Lambda Trigger to a Kinesis stream. Both the lambda itself, and the stream creation succeeded within Terraform, but the trigger would just stay stuck on “creating…” for at least 5 minutes, before I got bored of waiting and killed the process. Several attempts at doing this had the same issue. The code looked something along the lines of this:...

July 12, 2017 · 2 min

S3 Multi-File upload with Terraform

Hosting a static website with S3 is really easy, especially from terraform: First off, we want a public readable S3 bucket policy, but we want to apply this only to one specific bucket. To achive that we can use Terraform’s template_file data block to merge in a value: { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::${bucket_name}/*" ] } ] } As you can see the interpolation syntax is pretty much the same as how you use variables in terraform itself....

April 23, 2017 · 3 min

Using Terraform to setup AWS API-Gateway and Lambda

I have been writing simple webhook type applications using Claudiajs, which in behind the scenes is using Aws’s Lambda and Api Gateway to make things happen, but I really wanted to understand what exactly it was doing for me, and how I could achieve the same results using Terraform. The Lambda Function I started off with a simple NodeJS function, in a file called index.js exports.handler = function(event, context, callback) { callback(null, { statusCode: '200', body: JSON....

March 17, 2017 · 4 min