ASP.NET made heavy usage of Annotations, where you add decorators to methods which at runtime are interpreted by a framework to add functionality, or configure how a function is used. It wasn’t uncommon to see a method with 5 or more Annotations:

HttpRoute("/user")
public class UserAddressesController : Controller {

  HttpPost()
  HttpPut()
  HttpRoute("/addresses")
  Authenticate()
  Accept("application/json")
  Response("application/json")
  public function AddAddress() {
    var body = HttpContext.Body;
    //...
  }
}

Other tools in the .NET world used the concept of Marker interfaces to remove some of the inheritance. A marker interface doesn’t have any methods, but by implementing it the framework finds the class at runtime to use.

FubuMVC (“For Us, By Us”) was heavily inspired by Ruby, and more specifically Rails. It favoured Convention over Configuration heavily. Rather than Annotations or Marker Interfaces, it used naming conventions. Name your class ThingController? it became a controller. Name it OtherEndpoint and it became an Endpoint. I don’t currently remember what the difference was. Name your method GetUserAddress and it would handle GET requests to /user/address

This was fine when you were doing the obvious parts of the application, like Controllers, and Views and Models etc, but when you started to need to do other things, it got a little harder. How do you do some error handling decoration to all routes? Add a class called ErrorFilter, with a method with the right signature, and it just works. As the author wrote in a retrospective about the project, the documentation was…limited to put it lightly. A lot of questions and answers were in a chatroom on Gitter (I think) which was hard to search.

This experience wasn’t all negative. Far from it in fact; it had amazing testability, worked with a decent dependency injection container, and didn’t use Annotations at all, which was fantastic.

The system worked well once you knew how it worked, but new developers on the system? They had better hope there was a more experienced developer around to show them how to add routes, handlers, controllers, and whatnot.

Over time, I have drifted further and further away from Convention over Configuration in this sense. Conventions in a codebase are still important; things like “we name all identifiers ThingId”, or “cli actions are referred to as Commands” give a lot of consistency, without an extra mental hurdle.

Being the New Guy

At work recently, my friend and I have been given a service to take ownership of, modernise, and start implementing new features in. The codebase could be generously described as “awaiting care”, but, it does work. The issue we are having however is that it is heavily Convention over Configuration based, highly abstracted, and very un-explicit everywhere. It is borderline magical how the system actually works.

The project is Java based which in itself is not a bad thing, but it leans heavily on Annotations (or Attributes), and two gigantic “common” libraries which both can do so much based solely off of configuration values.

For example, the codebase uses some kind of role based authentication via an Annotation on the class, and then further annotations on the methods. I want to know what is actually doing the authentication, but in the .properties files there are several oauth clients configured, all with fairly generic names like “oauth”, “clients”, “accounts”, kind of names. None of them are referenced in application code, and it took a lot of digging through shared libraries to figure out what was going on.

Another example are the many HTTP route handlers, which are also configured with Annotations, and @bean magic. I want to find the handler for some long path, ending with /bulk, which is quite hard to do when the path is split across multiple Annotations (class level, method level), and half the time (yay for inconsistencies!) the route is defined on an abstract class then I have to find the (only) ClassWhateverImpl to see the handler.

These kind of things are problematic for several reasons:

1. I am owning the service. If auth breaks, I should be able to debug it
2. We want to refactor and remove unused dependencies. Are these properties used?
3. We might replace the service, can we rely on a common auth module for another language?

What I would do instead

Be explicit. Figure out how the team wants to define routes, then do that one thing everywhere. I love opening a server.go file and finding a method that looks like this:

func NewHttpAPI(...) {
  server, err := createServer(...)
  // imagine error handling here

  signals := withOsSignals(ctx)

  withGracefulShutdown(ctx, server, signals)
  withTracing(ctx, server)
  withMetrics(ctx, server)
  withPanicRecovery(ctx, server)

  server.Handle("GET /_info", infoHandler(config, deploymentInfo))
  server.Handle("GET /_info/ready", readyHandler(ctx, db, cache, signals))
  server.Handle("GET /_info/live", liveHandler(ctx))

  auth, err := authMiddleware(db)

  server.Handle("GET /users", getUsersHandler(cache))
  server.Handle("POST /users", auth("user:create"), creatUserHandler(db, cache))
  server.Handler("DELETE /users/{userid}", auth("user:delete"), deleteUserHandler(db, cache))
  //...
}

This code tells me so many things all in once place:

• all routes are handled by functions called ...Handler
• the server is listening to some OS signals
• the signals are used for graceful shutdown
• there are tracing, and metrics globally
• there is a panic handler to prevent crashes
• there is a db and a cache
• listing users comes from cache, NOT db
• listing users is anonymous allowed
• creating a user requires user:create permission, deleting requires user:delete

This code is possibly longer than a convention based system, but typing speed is not the part of development that slows people down (thinking is). Code is read far more than written, and having all this information explicitly defined means any new person to the project has a reasonable chance of getting started.

There are also good abstractions hinted at: I don’t need to know how graceful shutdown works, or how OS signals are handled. If graceful shutdown stops working, the first place I am checking is withGracefulShutdown, followed by withOsSignals.

For the future

Be explicit, reduce magic, and remember code is read more than written.

Be kind to the new people joining your codebase. It might just be you in the future!

For example, a person table would have the prefix PEO, and the columns would be PEO_PersonID, PEO_FirstName, PEO_DateOfBirth, etc. When you wanted to create a new table, you opened the shared Excel sheet, added your table to the bottom, and made up a prefix that wasn’t already in the sheet. Even link tables (for many-to-many relationships) were not immune to this rule.

With over 100 tables in the database, finding a prefix which was vaguely related to the table’s purpose became harder and harder, especially for common letters, such as C which off the top of my head had tables like Companies, Candidates, Contacts, Categories,Contracts, ContractAttachments, ContractExceptions, and a bunch of link tables to go with them all.

When asked, the DBA said that the reason the convention existed was to prevent column name conflicts when joining tables; all columns would be globally unique! This made some level of sense for simple queries:

select  CAN_CandidateID,
        PEO_FirstName,
        PEO_LastName
from    candidates
join    people on CAN_PersonID = PEO_PersonID
where   CAN_CandidateID = @candidateID

The problem was that this didn’t really solve the issue of columns not being ambiguous; queries often needed to join to the person table mulitple times, often via another table:

select  CAN_CandidateID,
        peo.PEO_FirstName + ' ' + peo.PEO_LastName as 'name',
        creatorperson.PEO_FirstName + ' ' + creatorperson.PEO_LastName as 'creator',
        modifierperson.PEO_FirstName + ' ' + modifierperson.PEO_LastName as 'modifier'
from    candidates
join    people peo             on CAN_PersonID = peo.PEO_PersonID
join    users creator          on creator.USR_UserID = CAN_CreatedBy
join    people creatorperson   on creatorperson.PEO_PersonID = creator.USR_PersonID
join    users modifier         on modifier.USR_UserID = CAN_ModifiedBy
join    people modifierperson  on modifierperson.PEO_PersonID = creator.USR_PersonID
where   CAN_CandidateID = @candidateID

A thing of pure beauty, as you can see. Not only were all the sql statements far longer than they needed to be, and you often needed to figure out some obscure prefixes, you end up “stuttering” with things like person.PEO_PersonID - how many times do I need to know this is a PersonID in a single sentence?

The primary key of each table had to include the table name too. I’m not actually convinced this is a bad idea; having a bunch of columns called id doesn’t really make things clear when joining 6 tables.

The interesting part of this is that it’s all useless; the database server we used supported table aliases (as seen above), so we could use those and drop the prefixes entirely:

select  CandidateID,
        p.FirstName + ' ' + p.LastName as 'name',
        creatorperson.FirstName + ' ' + creatorperson.LastName as 'creator',
        modifierperson.FirstName + ' ' + modifierperson.LastName as 'modifier'
from    candidates c
join    people p               on c.PersonID = p.PersonID
join    users creator          on creator.UserID = c.CreatedBy
join    people creatorperson   on creatorperson.PersonID = creator.PersonID
join    users modifier         on modifier.UserID = c.ModifiedBy
join    people modifierperson  on modifierperson.PersonID = creator.PersonID
where   CandidateID = @candidateID

Shorter, at least.

The table relationships didn’t always help matters either; the idea of the Person table was that a person could exist as multiple entities in our system; they could be a user, a candidate, and a contact. Not that this actually happened; they had unique person records for each of their entities.

The table prefixes also meant that when we wanted to use a microORM (Dapper, which only maps queries into objects), we had to make every query alias every column, otherwise our property names would have to also include the prefixes, and we really didn’t want the column prefixes polluting the rest of the domain!

We never got rid of this scheme in the primary database; the change wasn’t worth doing. If we had started making new tables without prefixes, we would still have had 100+ old tables with the prefixes, and no chance of fixing them as usually it involved dropping and recreating the tables. Definitely not worth the hassle.

However, when we started creating separate databases for services which didn’t rely on any data in the main database, the column prefix was not used. In those services, everything felt a little smoother and a little less noisy.

json: cannot unmarshal array into Go struct field Thing.Parts of type Parts

The data structure it is referring to looks like this:

type Thing struct {
	Parts Parts
}

type Parts struct {
	Part []Part
}

Which represents the (slightly weird) json structure we receive in a message:

{
	"thing": {
		"parts": {
			"part": [
				{ "name": "one" },
				{ "name": "two" }
			]
		}
	}
}

However, rarely, we receive a json document which looks like this instead, where the parts struct has instead become an array, with one object containing the part array:

{
	"thing": {
		"parts": [	
			{
				"part": [
					{ "name": "one" },
					{ "name": "two" }
				]
			}
		]
	}
}

In the interest of keeping the software running while digging through for the root cause of this, I added an implementation of the json.Unmarshaler interface to the Parts struct to allow it to handle both forms of json:

// duplicate of the Parts type, to prevent recursive calls to the UnmarshalJSON method
type dto struct {
	Part []Part
}

func (i *Parts) UnmarshalJSON(b []byte) error {

	// this is the standard format that json arrives in.
	normal := dto{}

	err := json.Unmarshal(b, &normal)
	if err == nil {
		t.Part = normal.Part
		return nil
	}

	// sometimes, we get json with an extra array, so if we get an error about that,
	// try the alternative structure
	if jsonErr, ok := err.(*json.UnmarshalTypeError); ok && jsonErr.Value == "array" {

		weird := []dto{}
		if err := json.Unmarshal(b, &weird); err != nil {
			return err
		}

		if len(weird) > 0 {
			t.Part = weird[0].Part
		}
		return nil
	}

	return err
}

When I opened a pullrequest about this, one of my colleagues approved it, but also noted:

for once typescript would solve something more cleanly in my opinion

And I agree, after deserialising, doing something like this is much less code, and basically has the same result.

const thing = JSON.parse(message);

if (Array.isArray(thing.parts)) {
	thing.parts = thing.parts[0]
}

Down the Rabbit Hole

Tracing back through the system to figure out where the message came from lead me back to a system which parses an XML document and, after doing some work on the result, emits the json message we handle. The XML itself has a pretty reasonable structure (and far larger than I am showing here, with tens, if not hundreds of nodes):

<Thing>
	<Parts>
		<Part name="one" />
		<Part name="two" />
	</Parts>
</Thing>

Which it mangles into that weird json structure. It does, however, do some sanitisation to the Thing before writing it out, and I found one for dealing with the Parts property:

// if there is only one part, the parser doesn't emit an array, so force an array.
if (!Array.isArray(thing.parts.part)) {
	thing.parts.part = [ part ]
}

Interesting! but this is a different bug to the one we’ve just seen; in our case the Parts became an array…

Checking the original XML file which was processed, it looked entirely normal until I noticed that it has two Parts nodes:

<Thing>
	<Parts>
		<Part name="one" />
		<Part name="two" />
	</Parts>
	<!-- many nodes later -->
	<Parts>
		<Part name="three" />
		<Part name="four" />
	</Parts>
</Thing>

So the fix is to add another sanitisation to our parser:

if (Array.isArray(thing.parts)) {
	thing.parts = {
		part = flatMap(thing.parts)
	}
}

This fixes the data as soon as it appears in our system; however, searching our codebase revealed that, up until this fix, many places had been missing data, or incorrectly handling the data.

While the TypeScript types written for the Thing are correct, that doesn’t help when the data is supplied at runtime and apparently can have varying shapes.

The Tradeoff

The tradeoff between typescript/javascript and Go feels like this to me:

Go causes me to notice when something isn’t working, as errors start being returned about data not matching the shape it was expected to be in. Fixing the issues in general, require more code than the same fix in typescript would.

Typescript has short code, but as its only a compile-time type checking system, when weird data starts arriving, you don’t get any errors (directly, things later on can break however.)

For me, I would rather have slightly longer code which is more explicit, and tells me when something goes wrong, rather than silently continuing. The likelihood of a silent error in serialisation leading to data loss or corruption is just too high.

The problem showed up at random, and instead of a window, dialogue, or control being rendered, it would be replaced with a white box with a red outline and red diagonal cross, and I think some error text in one corner, saying something about GDI handles. The issue itself didn’t seem to be related to either time or memory usage; when the app crashed, we got an error report (usually), and that never suggested that the application had been open particularly long or that it was using an excessive amount of memory.

I had been given the job of fixing this problem (and others), so armed with a memory profiler (RedGate’s, if I recall correctly), I went to work. With no given reproduction, it was hard. Some searching had shown that GDI handles were created when doing custom control painting; usually, this would be a fairly strong indicator of where to look, but in the case of this application, nearly all controls were custom-drawn, and there were many.

After days of taking memory snapshots and running comparisons, the only thing I was really noticing was that the number of font instances in use seemed high and got higher whenever I opened dialogues; it never went down again.

So, something to do with fonts. Reading through our common control base code, I noticed this:

public class ControlBase : Control
{
  private Font _normal = new Font("some-font", 12);
  private Font _bold = new Font("some-font", 12, Options.Bold);
  private Font _italic = new Font("some-font", 12, Options.Italic);

  // ...
}

This snippet means that every single control had 3 instances of a Font, which was never disposed (no _normal.Dispose() in the ControlBase.Dispose() function.) My first reaction was to add the three .Dispose() calls, but realising the fonts were never modified after creation led me to make them static so that all control instances shared the same font instances.

A week or two of work, and the only output was adding 3x static words to the codebase - but the effect was that our application went from using 1000s of font instances to 3. Quite the saving - and we never had the red X problem again!

The first lesson I took from this was that memory profiling is hard - running the app, taking a snapshot, running a bit more, taking a snapshot, etc. was not a fun feedback loop, especially as running with the profile slowed everything down massively.

The second, and probably more important, lesson was that the number of lines changed doesn’t reflect the amount of effort that went into changing those lines.

Typically, my application startup looks something like this:

func main() {
  ctx, cancel := context.WithCancel(context.Background())
  handleSignals(cancel)

  tracerProvider := configureTelemetry(ctx)
  defer tracerProvider.Shutdown(ctx)

  tr = traceProvider.Tracer("cli")

  if err := runMain(ctx, os.Args[:]); err != nil {
    fmt.Fprintf(os.Stderr, err.Error())
    tracerProvider.Shutdown(ctx)
    os.Exit(1)
  }
}

func runMain(ctx context.Context, args []string) error {
  ctx, span := tr.Start(ctx, "main")
  defer span.End()

  // some kind of loop
  for _, message := range someProcess(ctx) {
    select {
      case <-ctx.Done:
        return ctx.Err()

      default:
        doMessageThings(ctx, message)
    }
  }

  return nil
}

The handleSignals method listens to things like sigint, and calls the cancel() function, and the app stops processing messages and exits gracefully.

When the application exited due to errors, I would see the whole trace from the application, but if the application stopped due to sigint or similar, the main span would never come through.

After a bit of reading, I realised the bug is here:

ctx, cancel := context.WithCancel(context.Background())
handleSignals(cancel)

tracerProvider := configureTelemetry(ctx)
- defer tracerProvider.Shutdown(ctx)
+ defer tracerProvider.Shutdown(context.Background())

  tr = traceProvider.Tracer("cli")

  if err := runMain(ctx, os.Args[:]); err != nil {
    fmt.Fprintf(os.Stderr, err.Error())
-    tracerProvider.Shutdown(ctx)
+    tracerProvider.Shutdown(context.Background())
    os.Exit(1)

The problem is that when the context has been cancelled, the tracerProvider skips doing any work, so never sends through the last spans!

This issue would have been more noticable if:

• I checked the err value from Shutdown(), which is easily missed in a defer call
• Setting OTEL_LOG_LEVEL to debug printed something useful!

Hopefully by writing this down I will remember or at least find the answer next time I manage to do this again!

I try and follow what I call “outside in design”; I try and make something that requires the bare minimum amount of configuration to cover the most common of use-cases. Once this functionality is working, further configuration can be added to cover the next most common use cases.

API Reduction As A Feature

The first example I want to go through is how I removed options from an HTTP rate limiter we use. There are many teams using rate limiters, and we have noticed that there are often similar mistakes made in how they work and duplication of domain-specific functionality.

In order to make life easier for most users, a new rate limiter was made which reduced the API surface area, only exposing the bare minimum options.

Algorithm. Instead of offering many different types of algorithm (for example Token Bucket, Leaky Bucket, Fixed Window Counter, and Sliding Window), the rate limiter only uses Sliding Window.

Sizes.: By forcing the use of a specific algorithm, we eliminate a lot of algorithm specific options, such as bucket capacity, refill/drain rate, and window overlap. We expose a single option of WindowSeconds with a default vault of 60.

Penalties. We also decided to not expose how long a ban is, and instead make it a multiple of the WindowSeconds option, in our case WindowSeconds * 3.

Selection Criteria. Rate limiters we observed could filter by many different properties, such as IP Address, Headers, Cookies, Path, Query, Status Code, etc. Our rate limiter has the following rules:

• Account and Path
• Account and (bad) Status
• IP Address and Path
• IP Address and (bad) Status
• Anonymous IP

The way the account is detected is the same for all services in our domain, so we can centralise the checking code to be identical in all instances.

Triggering. For triggering, we went with the simplest thing possible: if any counter’s value goes over a threshold, then a ban is issued for the Account or IP address triggering the ban. The threshold value is exposed as MaxRequests with a default value of 100

Finally, we expose one additional configuration: Storage. This is an optional field which you can set to a Valkey or Redis-compatible client so that if you have many instances of your application, the rate-limiting is shared amongst all instances.

For most of our teams, using a rate limiter is now this:

limiter := &org.NewRateLimiter(
  org.WithStorage(valkey),
  org.WithMaxRequests(80),
)

mux := http.NewServeMux()
mux.Handle("/", limiter(apiRootHandler))

For teams that need more customisation, we recommend they reach out to us to see what their needs are; the outcome would usually be embedding and customising the rate-limiter, forking the library, or using an off-the-shelf library directly. So far very few teams have needed extra customisation.

The downside to this approach is if we want to change some detail about how the rate limiter works; we now have to find all the teams using it to make sure we don’t break their workflow. For Go projects, we typically bump the major version of the library, and have explicit messaging about the differences in the readme.

Organisation Conventions

The next example is trying to show off what we can achive when using conventions; some of these conventions were in place before we wrote this code, and some have become conventions since.

When it comes to building docker containers, there are a few things that people, in general, want:

• the container to be built
• tests to be run, preventing publishing broken containers
• the (working) container to be published somewhere so it can be used
• extra artifacts from the build to be collected (test reports, coverage, etc.)
• it to be fast

The problem with all these things is in the details; building itself is fairly straightforward, but publishing requires knowing where to publish, any credentials required, and how to name and version the container. Likewise, artifact collection requires knowing where the artifacts are to be collected from, and where to publish them to (along with authentication etc.)

The even bigger issue is “to be fast”; people don’t care about how its fast, they just want fast. This means not only making a cacheable dockerfile but doing that caching somehow; with ephemeral build agents, that caching becomes harder.

We can go through our requirements and see what ones we know the answers to already and what we need to get from users:

Docker Registry. The internal Secret Management Service (SMS) has a convention for where your docker registry is, and what the credentials are: read from /teams/$team_name/docker/registry.

Container Path. We always publish to $registry/$team_name/$repo_name/$container_name.

Container Name. A repository can have multiple containers, or the name of the container can differ from the repository. So for this property, we need the users to supply something.

Container Version. We decided that a short git SHA is enough for versioning.

Caching. The registry has a second path convention for storing cache contents: $registry/cache/$team_name/$repo_name/$container_name.

Artifacts. Artifacts are published to the Github Actions artifacts, so no extra authentication or settings are needed. We decided that if the .artifacts folder exists and has contents, that is what will get stored.

Given the above analysis, we decided on 4 configuration options:

1. team_name: no default. We will use this value to find the registry information and build container and cache paths.
2. container_name: no default. You need to tell us what the name of your container should be.
3. build_args: default empty. Supply extra arguments to the docker build command. Some teams need to inject extra information from the host.
4. dockerfile: default ./Dockerfile. Some teams have multiple dockerfiles in their repository, or keep the files in subfolders.

By relying on the team_name parameter, so many other options can be eliminated, and it turns out most people don’t care what exact path their containers are uploaded to, as long as they are accessible when it comes to being used in a deployment environment. This is foreshadowing!

For most teams, their build workflow becomes just two steps: checkout sourcecode, and build the container:

steps:
- uses: actions/checkout@v4

- uses: org/docker-build@v1
  with:
    team_name: "team-one"
    container_name: api

Organisation Conventions Two

Now that we have a shared way to build docker containers with low configuration, the next logical step was figuring out if we could do the same for deployment. It turns out a lot of the conventions used to build the container can be applied to deployment: docker registry, container path, container name, and container version are all the same between the two. In addition, we need to add a few more: the name of the environment being deployed to and the path to your deployment definition file (for example, a Nomad job).

steps:
- uses: org/nomad-docker@v1
  with:
    team_name: "team-one"
    container_name: api
    environment: live

The Pit of Success

We also like to leverage The Pit of Success, which seems to originate from Rico Mariani; we want to make doing the easiest thing to be the correct thing.

To that end, we provide a library to populate an app’s secrets. This library handles multiple forms of authentication for different runtime locations (developer machine, nomad cluster, lambda, etc.), and handles where the secrets themselves are located.

The library’s usage boils down to two things. A single struct to represent their secrets:

type Secrets struct {
  ClientID string
  ClientSecret string
  ApiToken string
  // etc
}

And a single function call to populate it:

err := org.ReadSecrets(ctx, "management-api", secrets)

This function call does a lot behind the scenes:

Authentication. This varies based on where the app is running: on a developer machine, it uses the local cached secret manager credentials and triggers authentication flows if needed. When deployed, it uses the relevant secret authentication system for that environment (e.g. Nomad’s Vault integration or AWS Secret Manager in Lambda).

Secret Location. It reads all the secrets for from a conventional path: /teams/$team_name/apps/$app_name/$env/*, where the values come from different places:

• team_name comes from a common environment variable, and ReadSecrets errors if its not populated
• env comes from either an environment variable when the app is deployed somewhere or is set to local on a developer’s machine.
• app_name is supplied in code (management-api in this case)

While teams can roll their own secret management integration, our library is so easy to use that almost no teams choose to do anything different.

The Golden Path

Our tools form what we call our Golden Path, a term which seems to originate from spotify. We use it to define a way to develop and deploy software in a tried and tested manner. Teams are always free to choose their own path by changing what parts of the system they see fit.

The trade off teams are making is between maintenance burden and configuration; choose our tools, and you don’t need to worry about them working, but you need to follow our conventions and opinions.

How Do You Design Software?

While this is working really well for me and my teams, there has to be other opinions too; I’d be interested in hearing how people do this for their teams and projects.

TLDR

Print debugging is a tool, and it has its uses - however, if there are better tools available, maybe use those instead. For me, a better tool is OpenTelemetry tracing; it gives me high granularity, parent-child relationships between operations, timings, and is filterable and searchable. I can also use it to debug remote issues, as long as the user can send me a file.

Print Debugging

The good thing about print debugging is that it is always available; it is very rare a process doesn’t have a stdout (or stderr) available, or some kind of logging system. The downside with print debugging is that it is very basic; anything more you want to do with it, such as timing data, has to be implemented.

One of the comments I read was, in essence:

one place print debugging is best is dealing with threading/timing bugs, it lets you see what is happening when

I can understand this, but often print statements are buffered before hitting the terminal, so you might lose some order; you can solve that with a timestamp (assuming it’s accurate enough), but as I mentioned earlier, you need to do that yourself.

Use a Debugger?

The first debugger I used was in Visual Studio, and it was amazing. I learned how to use a lot of features in it, such as automatic break-on-exception, conditional breakpoints, automatic counters (“break after hitting this line x times”), watches, and computed values. This was at the beginning of my career, and neither I, nor the codebase, knew anything of automated testing.

Once I learned about automated testing, and then started writing tests, the amount of time I used the debugger reduced. It was still a great tool, but tests were sometimes the quicker way to verify some behaviour. Often times, once I had investigated a problem with the debugger, I would write a test to cover that situation - now the repeated debugger usage to know if the fix worked wasn’t needed.

I don’t often use a debugger these days. It still has it’s place in my toolkit, but in general, I lean more on OpenTelemetry and tracing in general.

Tracing

My current go-to tool for debugging and application development is OpenTelemetry tracing. I’ve written about why I prefer tracing to Structured Logging before, but this is a bit different.

By default, tracing lets me see the relationships between function calls, the timing durations of each span (a span is a unit of execution; it could be a function call or a call into a library), and many structured attributes (both automatic and manually added.)

The downside is that you need some way to view that tracing data; using the stdout trace exporter is generally not useful - its far too noisy. For local development I use Grafana Tempo all-in-one, or Otel-TUI. For deployed environments, I still think honeycomb is the best placec for traces.

Most projects I interact with have a docker-compose file for all their local dependencies; adding a tracing viewer to that compose file is pretty trival, so the barrier to entry is really not that high:

grafana:
  image: grafana/otel-lgtm
  ports:
  - "3000:3000"
  - "4317:4317"

Tracing for Remote Debugging

I maintain some internal CLI applications, which make extensive use of tracing. When a user has a problem, they can add a --store-traces flag to the CLI, and it will write all the tracing data for their run to a file on disk. They can then send me that file, I can load it into a trace viewer, and use all the high-cardinality attributes and timing data to figure out where their problem is exactly.

Do you ever use print debugging?

Not often, but it does happen sometimes. As I said at the outset; its just a tool, and I can and do use it. I just happen to prefer a different tool.

However, too much configuration is a bad thing in general. There are two ways I want to view configuration: internally, from the developer of the software’s perspective, and externally, from the user of the software’s persepective (who might also be writing software.)

Most of the categories I will cover exist in both internal and external sections, as the effects they have are bi-directional.

Internal Issues

While internal issues will be more felt by you, the developer of software, I think that they are actually less important than the external issues. How a user experiences your software is siginficantly important, and in general I would rather take on a little burden to allow my users to have a better experience - to a point though.

Cognitive Load

One of the most underlooked aspect of configuration is the cognative load it puts on developers. Every configuration value brings in additional questions, some of which we will touch on later. For now I want to look at how a value is actually used. Every piece of indirection in a codebase adds up over time - some indirection is useful and needed for sure, but having too much makes it harder to follow flow of the program.

Often times, configuration values interact with eachother in unexpected ways; a retry-count and a backoff-strategy when configured together can cause your program to retry forever very fast, or retry forever with days between attempts.

Changing Values

A configuration value is part of your API contract; changing anyhthing about it can break your users, and might keep you from making needed changes. For example, if a configuration has a default value, and you want to change that default, how do you handle all the usages of your software which are expecting the default to remain the same? will it impact their usage? The same goes for removing a default value, renaming a configuration setting, or changing its type.

Validation

Configuration values need validating, and not just checking they are the expected type (type validation) - they also need semantic validation; checking wether a value makes sense in the given context. For example, setting an HTTP timeout to 3 days is probably not a useful thing to do.

This adds to the complexity of your software: constants (or even magic values) don’t need this extra validation and the more code you have to maintain, the higher the burden of maintenance.

Complexity

This area in general applies to applications over libraries, but does apply to both, especially when libraries can read configuration from multiple places.

First off, how many configuration sources does your software have? For a CLI, there are usually at least three sources: Command arguments and flags, environment variables, and configuration files. These sources need to have their values merged somehow, and even if that strategy is just “cli arguments win”, you still have to implement a hierarchy and all the complexities that come with that. Boolean configurations are fairly straightforward, but how do arrays or nested objects work? Do you merge the array values? Do you replace the entire array? And is the answer the same for all arrays in your configuration, or does it differ?

If you are lucky, all the complexity is encapsulated into one place, such that the rest of you software sees a single Configuration object of some form, and doesn’t need to worry about how the values got into that structure.

However, even libraries end up doing things with the environment. For example, the OpenTelemetry libraries will read the environment for default values; this is nice to start with as you have to write less code to get things up and running, and (at least for OpenTelemetry) the environment variables are documented, and the same across languages and versions. It does mean, though, that the configuration for your OpenTelemetry setup is separate from the rest of your application. What if you also need to do something that depends on a value that is used for OpenTelemetry?

The final complexity to talk about is the extra code paths introduced by each configuration value. This applies to things like which storage method to use in the application, rather than only supporting one, or which transports to use. I have had experience with a vendor whose product supported multiple central logging backends, but it turned out they primarily used Firebase, and all other backends weren’t tested so thoroughly. We saw so many performance problems with the other “supported” backends. Which leads us nicely to Testing.

Testing

Testing configuration is tricky; generally its not really done, which leaves you open to problems like “did this value here really used to work?”

Frequently when developing software, I have multiple implementations of a cache in my programs: there is the real implementation (such as Redis or a filesystem), a testing cache (in memory), and a bunch of decorators (statistics about the cache, tracing of what is happening etc.) When testing most of my applications, I use the InMemoryCache as it is the fastest and eliminates shared state from my tests (or even between test runs), but I have to make sure there are also sufficient integration tests for the application using the real cache - not just testing its implementation.

Documentation

Configuration values need documenting: what each value does, whether it is required or not, what it’s default value is, and any interactions it has (i.e. configuration it cannot work with or requires to also be set.) Documentation is generally not the strongest of points for developers, and keeping that documentation up to date and accurate again adds to the maintenance burden.

External Issues

The issues we face internally are also faced externally by the users of our software. Not all of them are exactly the same, and the problems can be easier or harder than the internal problems.

Cognative Load

When you encounter a piece of software which has so many options, how do you go about figuring out what settings you need for your usage, and what properties need to be set? If you are lucky, the Documentation is up to date and easily discoverable.

One interesting problem occurs with default values; I have seen software where not specifying a property means something very different from specifying it as null/empty/etc.

The interaction of different properties can also be hard to follow; properties which only work when other properties are set (or not set) is one issue, and the other is how the values you specify interact with each other, such as with timeouts and retries.

If there are many configuration properties, trying to discern the right one to change is also a burden, and this gets even worse when there are many sources of configuration: where is the appropriate place to set the value? is something else overriding it? do you even know where all possible places the configuration can come from are?

Changing Values

This is pretty much the same as the Internal Issues Changing Values section; if I have configured something and it is working, but then at some point the default values of something change, and it stops working, I now have to spend a lot of effort figuring out which value change was the culprit, and possibly try and work out what the old value was so that I can put it back to how it was.

This can be especially difficult if the changelog only says “changed the default value of $thingy to 17” - what it was before is a mystery, but maybe you can figure it out by looking through the commits. Assuming its opensource (or source available.)

What happens when a new feature is added? Should it be disabled by default or enabled by default? What if it suddenly starts doing something you don’t want, or its configuration starts affecting your existing configuration?

Validation

Like the previous validation section, we are trying to think of not only what values can go here but also what format they should be in. Is the value specified in seconds? Milliseconds? Epoch? or is it a string of “3m20s”? Hopefully, the software will tell you the value is invalid, but it might just ignore invalid values, making you think you’re setting something, but nothing is actually happening.

An experience I had somewhat recently with the ALB Controller was related to a configuration value. We had the following set in an annotation, which worked fine:

alb.ingress.kubernetes.io/success-codes: 200, 204

But someone decided that 204 was no longer a valid success-code, so removed the , 204:

alb.ingress.kubernetes.io/success-codes: 200

…which broke silently when deployed - the problem was that the ALB controller is expecting a string here, when the value changed it suddenly parsed as an integer. What made this harder to pick up is that the deployment itself worked, but the ALB controller started throwing errors, and that wasn’t noticed for a while. The fix, by the way, was to add quotes around the value.

alb.ingress.kubernetes.io/success-codes: "200"

Testing

How often do you test that configuration of software is correct? I would hazzard a guess at “not often” - and when it is done, it tends to be just testing of the configuration values themselves; checking a timeout is set or not set for example.

What is harder to manage is to test the the given configuration has the desired effect on the software itself. How do you go about testing that timeouts and retries work as expected? or authentication parameters are having the desired effect?

This kind of testing is important too, especially when updating software; minor, or even patch versions of software can have breaking changes, and if you’re not testing it still does what you want, how will you know that everything is configured correctly?

Suggestions for better configuration

1. Don’t make something configurable unless it needs to be
2. Clearly specify what the default values are, what format values are in, and what the property actually does
3. Ideally, use a system that keeps the documentation for the property with the property in code. It might stay in sync then.
4. If there are conventions, follow them. This applies to all aspects: source of configuration, format, property names, and values.

Wrapping Up

We haven’t even gotten into the debate about the right configuration file language!

In a future post, I want to go over how these thoughts have affected how I design software, and the configuration for it, but this post is quite long enough already so I will wait for another time.

Can I have multiple errors in a Span?

Yes! An error is stored as an event attached to the span, so calling span.recordError(err) can be called multiple times. However, a span can only have one status: either Unset, Ok, or Error, so while many errors can be recorded, the status of the span can only be set to one value.

Multiple unrelated errors

The problem with wanting to store multiple errors in one trace is that while you can have multiple error events, if you are also using the span’s attributes for the error details (i.e. setting err.message and err.details etc.) then whichever was your last error will overwrite previous values.

One solution offered is to combine all errors into one, and then record that in the attributes. This feels like a bad idea to me due to how you consume the errors in your OTEL service of choice; generally, you filter by attributes without any kind of free text search or, at best, prefix matching.

This, however, means that you lose a lot of filtering ability. Rather than being able to do direct matching on the err.message attribute, you need to do wildcard searching, which is slower, and some providers don’t even support it.

Furthermore, you lose the ability to group by error type; you can no longer write group by err.message or group by err.type as they all have the same type (composite), or differing messages (especially if the order of the grouped errors is not deterministic.) You also loose out on being able to track how often a specific kind of error is occurring, making alerting on changes in error rates much harder to implement.

For these reasons, I would recommend refactoring your code to have a span per operation (or function, whichever provides the granularity you need), such that the main method emits a single span with several child spans, which each can have their own error details, and the parent span can then contain an error, and the overall status of the operation.

func handleMessage(ctx context.Context, m Message) error {
  ctx, span := tr.StartSpan(ctx, "handle_message")
  defer span.End()

  oneSuccess := childOperationOne(ctx, m)
  twoSuccess := childOperationTwo(ctx, m)
  threeSuccess := childOperationThree(ctx, m)

  childOperationFour(ctx, m) //optional

  if oneSuccess && twoSuccess && threeSuccess {
    span.SetStatus(codes.Ok)
  } else {
    span.SetStatus(codes.Error, "mandatory operations failed")
  }
}

Multiple related errors

A good use case for storing multiple errors is when the operation can emit multiple of the same error before succeeding. For example, an HTTP request retry middleware might emit an error for each attempt it makes but only set the span status to error if there wasn’t a successful call after several tries.

func Retry(ctx context.Context, req http.Request) (http.Response, error) {
  ctx, span := tr.StartSpan("retry", ctx)
  defer span.End()

  maxRetries := 5
  span.SetAttributes(attribute.Int("retries.max", maxRetries))

  for i := range maxRetries {
    span.SetAttributes(attribute.Int("retries.current", i))

    res, err := client.Do(req)
    if err == nil {
      span.SetStatus(codes.Ok, "")
      return res, nil
    }

    s.RecordError(err)
    backOffSleep(i)
  }

  retryError := fmt.Errorf("failed after %v tries", maxRetries)
  span.SetStatus(codes.Error, retryError.Error())

  return nil, retryError
}

Tracing Providers

A lot of how you deal with this is still going to come down to which tracing provider/software you use to actually ingest your traces. While OpenTelemetry has a specification of the protocol and fields, it doesn’t specify how a UI has to render that information, so it is possible that you have to adjust your usage to match your provider.

or example, if there are many child operations, then a span per operation with its own possibility of error would make sense; the parent span could then decide based on the child operations whether the whole operation was successful or not. You could also add attributes such as operation_count, operation_failures, etc.

Next Steps

For the codebase in question, based on what little I know of it, I think adding a few more Spans around the problem areas is the right thing to do; but depending on how many places this occurs, it could be more hassle than its worth.

I’d stick with the same approach for adding tracing to a codebase: add it where it hurts, as that’s where you’ll see the most value.

Being able to save a file in my editor and see the changes instantly in a web browser is an amazing developer experience, and I want to recreate that for htmx. I realised the steps to build my own hot reload were actually pretty small.

On the server side:

• generate a guid on startup
• expose this to the client somehow

On the client side:

• fetch the guid
• if the guid doesn’t match what we have seen before, refresh the page

Despite my preference for HTMX and html/template in Go, neither the Client nor Server implementations a framework specific. The server utilises Fiber as its host, but it is not a hard requirement.

The Client

I decided to use a websocket for the transport, as if I decide later to make the server notify the client of changes also. For the client side, I have a single script that I include in the html template, which connects a websocket, and handles all messages received. It also handles reconnection if the server disconnects, along with a simple backoff mechanism.

(function () {
  var lastUuid = "";
  var timeout;

  const resetBackoff = () => {
    timeout = 1000;
  };

  const backOff = () => {
    if (timeout > 10 * 1000) {
      return;
    }

    timeout = timeout * 2;
  };

  const hotReloadUrl = () => {
    const hostAndPort =
      location.hostname + (location.port ? ":" + location.port : "");

    if (location.protocol === "https:") {
      return "wss://" + hostAndPort + "/ws/hotreload";
    } else if (location.protocol === "http:") {
      return "ws://" + hostAndPort + "/ws/hotreload";
    }
  };

  function connectHotReload() {
    const socket = new WebSocket(hotReloadUrl());

    socket.onmessage = (event) => {
      if (lastUuid === "") {
        lastUuid = event.data;
      }

      if (lastUuid !== event.data) {
        console.log("[Hot Reloader] Server Changed, reloading");
        location.reload();
      }
    };

    socket.onopen = () => {
      resetBackoff();
      socket.send("Hello");
    };

    socket.onclose = () => {
      const timeoutId = setTimeout(function () {
        clearTimeout(timeoutId);
        backOff();

        connectHotReload();
      }, timeout);
    };
  }

  resetBackoff();
  connectHotReload();
})();

Note this is a pretty dumb hot reload - it just refreshes the current page.

The Server

The entire implementation is about 20 lines of go, utilising the websocket package for fiber, my web server framework of choice. There is not a lot to it; just create a UUID, and send that value to any client which connects to the websocket, and sends any message to us.

import (
  "github.com/gofiber/contrib/websocket"
  "github.com/gofiber/fiber/v2"
  "github.com/google/uuid"
)

func WithHotReload(app *fiber.App) {
  id := []byte(uuid.New().String())

  app.Use("/ws", func(c *fiber.Ctx) error {
    if websocket.IsWebSocketUpgrade(c) {
      return c.Next()
    }
    return fiber.ErrUpgradeRequired
  })

  app.Get("/ws/hotreload", websocket.New(func(c *websocket.Conn) {
    for {
      if _, _, err := c.ReadMessage(); err != nil {
        break
      }

      if err := c.WriteMessage(websocket.TextMessage, id); err != nil {
        break
      }
    }
  }))

}

How it works

I use the modd tool to restart my go applications when I am developing them: any time I save a file, the app restarts.

When the app restarts, all websocket connections are aborted. The client then tries to reconnect, and when it does, it receives a new UUID from the server, causing the entire page to refresh. As all my apps are serverside rendered, there is usually little, if any, state to keep, so a full page refresh is fine for my development needs.

In this implementation, the socket is not needed; it would be just as easy to poll an API every x seconds to see if the UUID has changed, but having the client react to the server breaking the connection on restart seems better; there is less random HTTP noise in the network tab too.

Future modifications

I think I could make htmx do the hard work of switching out the page dom when the socket indicates the page has changed, and while that would be cool, it does mean that the client part would become htmx specific, so I probably won’t do this.

If you’re writing log statements, you’re doing it wrong.

This is a pretty incendiary statement, and while there has been some good discussion after, I figured it was time to write down why I think logs are bad, why tracing should be used instead, and how we get from one to the other.

Hopefully, with less clickbait. Step 3 will shock you, though.

Logs vs Traces

First, lets breakdown what I see as the key differences between logging and tracing code. If you want the practical example and want to skip this wall of text, click here. There is also a short Question and Answer at the end.

Log Levels

Log Levels are meaningless. Is a log line debug, info, warning, error, fatal, or some other shade in between?

The only time I have seen this well managed was when we had 3 descriptions:

• Info (everything)
• Create A Task for Later (e.g. this can wait for working hours)
• Wake Someone Up (this is on fire).

However, this has issues; a timeout once is “for later” or just info, but many timeouts might be “wake up”, how do you encode that logic somewhere?

There is no equivalent of this in a trace, so there’s one less thing to worry about.

Messages

Something like found ${count} users in the cache looks like a good log message. It is clear and has the information you care about. However, when you come to querying your log aggregation system, you need to do a free-text search on a substring of the message or use a wildcard search. Both of these operations are slow, and worse still cannot help you with the negative version of the question “find me operations where the cache wasn’t checked”.

Assuming structured logs are being used, we can at least add the count as a property to the log message so it can now be filtered on by the log aggregator. At which point, why do you need the log message at all? Why not just have log({ users_in_cache: count })?

The logging libraries also put more importance on the message than the properties by having it be the first (and only required) argument to the logging function. From a querying and analysis perspective, this is wrong: messages force you into free text searching and reduce the likelihood of fast queries on attributes.

Given this logline:

logger.info("found the user in the cache", userId);

You could reconstruct this as a statement that can be filtered on:

span.addAttributes({
  user_in_cache: true,
  user_id: userId,
})

Mixed Outputs / Semantics

Nowadays, people configure their HTTP server to write logs to stdout rather than to a log file. This makes a great deal of sense, as now a separate piece of software can tail the logs and send them off to the aggregator. Your application’s output, however, will likely end up with mixed plaintext and structured logs, mostly down to libraries and frameworks doing their own thing with logging, which is probably not the same as what you are doing (or other libraries are doing.)

The second problem with writing logs to stdout is that it mixes the semantics of log lines with console output. While both log messages and console output are (probably) just text, they do have different purposes. A piece of feedback might be “Server listening on port 3000. Click here to open the browser”. This is useful to a local user running the app but isn’t valuable in your logs. Not to mention, its plaintext output on the console, rather than structured, so now your log tailer needs to figure out what to do with it.

With OpenTelemetry, you instead configure an exporter to handle all your traces (and logs and metrics, if you desire.) This is typically sent as OTLP format either directly to a vendor or to an OTEL Collector instance, which can add/remove/modify data and send it to one or multiple places.

Now you are free to write whatever feedback you want to stdout.

Relationships

Loglines don’t have a causal relationships. At best, structured logs might have some kind of request identifier (such as requestID or correlationID) for all lines written during a request. This allows you to find all the log lines for a given ID, but the log lines themselves don’t have a fixed order. Ordering in structured logging relies on the timestamp field, but this is limited to the accuracy and resolution of the time source. It means that it is possible to get lines appearing at the same time when they happened at different times.

Traces come with automatic parent-child relationships, allowing us to see not only all spans in a single request, but what caused each span, and (as we’ll get to in the next point) when they happened.

Tracing also takes these relationships another step further but having a standard set of formats to pass trace and span IDs along to other services, embedded in HTTP headers, message queue metadata, or other locations. Assuming all your other services send their traces to the same system, you can start to visualise who calls your service and what the effects are of you calling other services.

Imagine opening a trace from your service and discovering another team in a downstream service has started tracing their stuff, and you suddenly can see even more information about the request. You didn’t even do anything! You notice that the way your service is structured causes a lot of load on the downstream, and start a conversation to see if there is a way to make it faster/better.

Timings

The only timing data you get on log lines automatically is the timestamp of when the log line was written. When you want to see how long an operation took, you must start a timer, and then stop the timer, and log the elapsed duration yourself. As this is a manual process, it is often not done, and when it is done, it tends to have inconsistencies across the application, namely timing source (and thus resolution), property name, and format. Is it elapsed, elapsed_ms, duration, or length? does it contain a number of seconds, milliseconds, nanoseconds, or a timestamp format?

By comparison, traces come with startTime, finishTime, and duration attributes, which are not only guaranteed to be there but are set from the same timing source and are always written in the same format.

Combine this with the relationship attributes, and you can now render timing graphs, allowing for easy visualisation of how long each part of a process takes, what can be parallelised, and what parts depend on other parts.

For example, this is a graph showing how long a CI job took to run, showing all the different steps, their durations, and child steps:

Querying

Querying can have wildly different performance characteristics depending on which log aggregation service you are using. What unifies all these systems is their slowness, which is mostly down to vast amounts of data which needs indexing so that it can be free text searched. Filtering logs by their structured properties is however pretty quick (usually.)

Where querying falls down is trying to find trends in data, and trying to find answers to negative queries. For example, how would you search for “all requests to x endpoint, where users were not found in the cache”? This requires you to group the logs by request ID, then find an entry with a particular url path, then see if it is missing a log line. The same query in a tracing system would be where path = "/some/api" && !user_in_cache, as the tracing system is already aware of all the spans in a trace, and does the grouping automagically.

Finally, visualising missing data is hard. Take this small example; it is 4 parallel requests to a system, and one of them is missing a log line. Which line is missing?

Is it easier to see the one that is different now?

Not only is it easy to see at a glance that 915d273db25c didn’t find the user in the cache, but also how much less space this takes up (both visually and in terms of storage.)

We can also then use this to query further: show me all traces where in_cache != true, and see what’s different about them.

Evolving logs

So, with all that being said, let’s look at a practical example of how to go about tracing an existing system and what that looks like.

Ripping all the log statements in an application in one go is not a feasible strategy, especially on a large codebase. However, we can use logs as a decent starting place and evolve our system to something better. Namely, to OpenTelemetry Tracing.

We’ll start off with a real pair of functions from one of the codebases I work on. Lots of information changed to protect the guilty innocent. This is run as part of an api call to publish a container, but this part has no web specific code.

func PrepareContainer(ctx context.Context, container ContainerContext, locales []string, dryRun bool, allLocalesRequired bool) (*StatusResult, error) {

	logger.Info(`Filling home page template`)

	homePage, err := RenderPage(ctx, home, container, locales, allLocalesRequired)
	if err != nil {
		return nil, err
	}

	templateIds := []string{homePage.ID}

	if container.PageSlugs.FAQ != "" {
		faqPage, err := RenderPage(ctx, faq, container, locales, allLocalesRequired)
		if err != nil {
			return nil, err
		}

		templateIds = append(templateIds, faqPage.ID)
	}

	if dryRun {
		return &StatusResult{Status: StatusDryRun}, nil
	}

	logger.Info(`Marking page template(s) for usage`, "template_ids", templateIds)

	if err := MarkReadyForUsage(ctx, container, templateIds); err != nil {
		return nil, err
	}

	return &StatusResult{Status: StatusComplete}, nil
}

func RenderPage(ctx context.Context, source Source, container ContainerContext, locales []string, allLocalesRequired bool) (string, error) {

	logger.Info(fmt.Sprintf(`Filling %s page template`, source.Name))

	template, err := FetchAndFillTemplate(ctx, source, container, locales)
	if err != nil {
		return nil, err
	}

	page, err := ConfigureFromTemplate(ctx, container, template, locales)
	if err != nil {
		return nil, err
	}

	if len(page.Locales) != len(locales) {
		const message = fmt.Sprintf(`Failed to render %s page template for some locales`, source.Name)
		if allLocalesRequired {
			return nil, fmt.Errorf(message)
		} else {
			logger.Warn(message, "locales", locales, "pages", page.Locales)
		}
	}

	return page, nil
}

Step 1: Add a Tracer

The first step is to import a tracer and start a span for each method. As is somewhat common in Go, the methods already have a ctx parameter, so we just need to wrap it with the tr.Start call.

var tr = otel.Tracer("container_api")

func PrepareContainer(ctx context.Context, container ContainerContext, locales []string, dryRun bool, allLocalesRequired bool) (*StatusResult, error) {
	ctx, span := tr.Start(ctx, "prepare_container")
	defer span.End()

func RenderPage(ctx context.Context, source Source, container ContainerContext, locales []string, allLocalesRequired bool) (string, error) {
	ctx, span := tr.Start(ctx, "render_page")
	defer span.End()

Just this step alone already gives us value over logging: As mentioned above, the spans automatically come with timing data and parent-child relationships.

Step 2: Wrap the Errors

OTEL Spans support a status attribute, along with a status message which is used when there is a non-success status. By creating a small wrapper function like this:

func Error(s trace.Span, err error) error {
  s.RecordError(err)
  s.SetStatus(codes.Error, err.Error())

  return err
}

We can wrap all error returns so that we capture the error itself (SetStatus) and there is an error event recorded on the trace too (RecordError):

if err := MarkReadyForUsage(ctx, container, templateIds); err != nil {
- return nil, err
+ return nil, tracing.Error(span, err)
}

Step 3: Add Attributes and Replace Messages

The next steps is to replace any logger messages with attributes by turning them into statements that can be filtered on.

- logger.Info(fmt.Sprintf(`Filling %s page template`, source.Name))
+ tracing.String(span, "source_name", source.Name)

We also want to add attributes for any parameters we might want to filter on later.

	ctx, span := tr.Start(ctx, "prepare_container")
	defer span.End()

	tracing.StringSlice(span, "locales", locales)
	tracing.Bool(span, "dry_run", dryRun)
	tracing.Bool(span, "locales_mandatory", allLocalesRequired)

Finally , we can simplify a code block: there is no point in the logger.Warning call here, as we can have all the required information as filterable properties:

+ allLocalesRendered := len(page.Locales) == len(locales)

+ tracing.Bool(span, "all_locales_rendered", allLocalesRendered)
+ tracing.StringSlice(span, "locales_rendered", page.Locales)


- if len(page.Locales) != len(locales) {
+ if !allLocalesRendered && allLocalesRequired {
+   return nil, tracing.Errorf(`Failed to render %s page template for some locales`, source.Name)
- const message = fmt.Sprintf(`Failed to render %s page template for some locales`, source.Name)
- if allLocalesRequired {
-   return nil, tracing.Errorf(message)
- } else {
-   logger.Warn(message, "locales", locales, "pages", page.Locales)
- }
}

The Result

The diff between the two functions shows that the tracing version is longer - by 9 lines. However, the traced version contains so much more information than the logged version.

var tr = otel.Tracer("container_api")

func PrepareContainer(ctx context.Context, container ContainerContext, locales []string, dryRun bool, allLocalesRequired bool) (*StatusResult, error) {
	ctx, span := tr.Start(ctx, "prepare_container")
	defer span.End()

	tracing.StringSlice(span, "locales", locales)
	tracing.Bool(span, "dry_run", dryRun)
	tracing.Bool(span, "locales_mandatory", allLocalesRequired)

	homePage, err := RenderPage(ctx, home, container, locales, allLocalesRequired)
	if err != nil {
		return nil, tracing.Error(span, err)
	}

	templateIds := []string{homePage.ID}

	hasFaq := container.PageSlugs.FAQ != ""
	tracing.Bool(span, "has_faq", hasFaq)

	if hasFaq {
		faqPage, err := RenderPage(ctx, faq, container, locales, allLocalesRequired)
		if err != nil {
			return nil, tracing.Error(span, err)
		}

		templateIds = append(templateIds, faqPage.ID)
	}

	tracing.StringSlice(span, "template_ids", templateIds)

	if dryRun {
		return &StatusResult{Status: StatusDryRun}, nil
	}

	if err := MarkReadyForUsage(ctx, container, templateIds); err != nil {
		return nil, tracing.Error(span, err)
	}

	return &StatusResult{Status: StatusComplete}, nil
}

func RenderPage(ctx context.Context, source Source, container ContainerContext, locales []string, allLocalesRequired bool) (string, error) {
	ctx, span := tr.Start(ctx, "render_page")
	defer span.End()

	tracing.String(span, "source_name", source.Name)

	template, err := FetchAndFillTemplate(ctx, source, container, locales)
	if err != nil {
		return nil, tracing.Error(span, err)
	}

	page, err := ConfigureFromTemplate(ctx, container, template, locales)
	if err != nil {
		return nil, tracing.Error(span, err)
	}

	allLocalesRendered := len(page.Locales) == len(locales)

	tracing.Bool(span, "all_locales_rendered", allLocalesRendered)
	tracing.StringSlice(span, "locales_rendered", page.Locales)

	if !allLocalesRendered && required {
		return nil, tracing.Errorf(`Failed to render %s page template for some locales`, source.Name)
	}

	return page, nil
}

If you wish to view the files as individual steps: initial, step 1, step 2, step 3, final

Questions & Answers

Some questions I have seen come up when talking about doing this.

How do I use this?

If you’re debugging some error, taking the span name from your UI, and searching the codebase is a good start. Use the attributes named in code to search your UI for things that match/don’t match.

Also, try Observability Driven Development. It involves using your trace data before, during, and after a change so you know if its actually behaving as you expect.

What do you recommend to view traces?

Honeycomb is the best, no question. Lightstep is a close second.

I think this is ugly!

That is not a question. It is also a matter of opinion - I happen to think traced code looks better than logged code. It also could be that it takes some getting used to when you first start seeing it! Remember how unit testing felt weird to start with?

I need a log message…

I try to turn log messages into statements. Like in the example above:

• found user in cache => user_in_cache: true
• loading template ${name} => template_loading: true, template_name: name

When I need to mark that code execution has gotten to a particular point in a method:

• add an attribute, like initial_processing_complete: true
• contemplate if the method should be split, and thus have its own span

I have a loop, I am overwriting attributes on each iteration

Similar to above: either move the loop body into its own method, use a closure to create a new span in the loop, or don’t have a span per iteration.

You can always add summary information after a loop:

foos := 0
bars := 0
for i, thing := range things {
  // ...
  if thing.prop == "foo" {
    foos++
  } else {
    bars++
  }
}

tracing.SetAttribute(span, "foos", foos)
tracing.SetAttribute(span, "bars", bars)

Thanks

Thanks to Aki for some of the questions and some discussions which expanded a few of the sections here.

Thanks to Lari for prompting this blog post initially!

Thanks to you, for making it to the end.

Given a repository with the following structure:

.
├── libraries
│   ├── core
│   ├── events
│   └── ui
├── services
│   ├── catalogue
│   ├── billing
│   └── shipping
└── tools
    └── admin-cli

There are a few rules we should enforce:

• Services cannot reference each other
• tools cannot reference each other
• Services cannot reference tools
• Libraries can only reference other libraries
• Libraries cannot have circular dependencies

There are also the conventions that we want to enforce:

• Feature folders should be used, not models, views and controllers
• Specific libraries should not be used
• all services should expose a /api/stats endpoint

How to write these tests will vary greatly depending on what programming languages and tools you use, but I know for sure they can be written in Go, C#, and TypeScript. Not only that, but the tests can be written in a different language than the applications; in this example, our applications are written in a mix of NodeJS and Go, and the architectural tests are written in Go.

Testing for a Convention

The convention we will test for is that we strongly prefer folder-by-feature over folder-by-type.

The test itself uses a couple of helper methods: repositoryFolders returns a slice of every folder recursively in the project, with information such as all child folders and all child files, along with names, paths, etc., populated.

The hasLayers function itself is just checking if the direct children of a folder contain “models”, “views” and “controllers” or “models”, “views” and “presenters”.

func TestFolderByFeature(t *testing.T) {
  folders := repositoryFolders()

  for _, folder := range folders {
    if hasLayers(folder) {

      assert.Failf(t, "found type folders, not slices", wrap80(`
It looks like '%s' is using this folder structure, known as "folder-by-type", which is discouraged:

%s

Instead, you should use folders-by-feature:

%s

For more information, see this ADR: ./docs/arch/005-folder-layout.md

If this test failure is a false positive, please let us know, or you can either improve the test or add your folder path to the '.architecture-ignore' file.  Here is the fragment that can be added:

%s
      `, folder.Path, layers(folder), slices(folder), folderByTypeArchitectureIgnore(folder)))
    }
  }
}

The error message in this kind of test is very important; it needs to cover:

• What was wrong
• Where the failure was (i.e. the path)
• Why this is considered wrong (with links to more information if needed)
• How to fix it
• How to add an exception to the rules (if desired)
• How to handle false positives

For example, this is what the rendered output of the test above looks like, showing the folder that was detected to have folder-by-type, showing an example of how it should look, and linking to the adr, which documents why this was chosen.

It looks like 'services/catalogue/src' is using this folder structure, known as
"folder-by-type", which is discouraged:

services/catalogue/src
├── controllers
├── models
└── views

Instead, you should use folder-by-feature:

services/catalogue/src
├── details
│   ├── controller.ts
│   ├── model.ts
│   └── view.ts
├── indexing
└── search

For more information, see this ADR: ./docs/arch/005-folder-layout.md

If this test failure is a false positive, please let us know, or you can either
improve the test or add your folder path to the '.architecture-ignore' file.
Here is the fragment that can be added:

```toml
[[services]]

[service.catalogue]
allowFolderByType = true
```.

There is also the text on how to skip a test if there is a good reason to or the test failure is a false negative. Adding to the .architecture-ignore file notifies the core team about an addition, but does not block the PR, as teams are all trusted; we just want to verify if something is happening a lot or if there is some case the tests are not handling.

An example of a good reason for ignoring this test is when a team is taking ownership of a service and adding it to the repository: they want to pull its source in and make as few changes as possible until it is under their control; refactoring can then happen later.

Testing a Project Rule

Now let’s look at how we verify that our services don’t reference other services. The test is similar to the previous one other than the repositoryServices() function returns a map of service names and Services. The Service struct is an abstraction which allows us to handle both NodeJS projects and Go projects with the same test.

func TestServicesCannotReferenceOtherServices(t *testing.T) {
  allServices := repositoryServices()

  for _, service := range allServices {

    for _, reference := range service.References {
      if other, found := allServices[reference.Name]; found {

         assert.Failf(t, "service references another service", wrap80(`
It looks like the '%s' service is referencing the '%s' service.

1.  Service Boundary
Needing data from another service is often an indication of non-optimal service boundary, which could mean we need to refactor our design a bit.

1.  Distributed Ball of Mud
Having many service to service dependencies make all our services more tightly coupled, making refactoring and deployment harder.

Sometimes a service to service reference is fine however!  You can add your service to service definition to the '.architecture-ignore' file if this is the case.  Here is the fragment that can be added:

%s
         `, service.Name, other.Name, serviceToServiceArchitectureIgnore(service, other)))
      }

    }
  }

The error message when rendered looks like this, again adding as much detail as we can along with how to add the exception if needed:

It looks like the catalogue 'service' is referencing the 'offers' service.

Service to Service references are discouraged for two main reasons:

1.  Service Boundary
Needing data from another service is often an indication of non-optimal service
boundary, which could mean we need to refactor our design a bit.

2.  Distributed Ball of Mud
Having many service to service dependencies make all our services more tightly
coupled, making refactoring and deployment harder.

Sometimes a service to service reference is fine however!  You can add your
service to service definition to the '.architecture-ignore' file if this is the
case.  Here is the fragment that can be added:

```toml
[[services]]

[services.catalogue]
references = [
    "offers"
]
```.

Further Work

Using tests like this also allows you to build extra things on top of them; for migrating from one library to another, you can add tests that specify that the number of usages can only go down over time, never up.

You can also use codeowners (or equivalent) to keep an eye on what is being added to the .architecture-ignore file, allowing you to react to emerging patterns and either guide teams towards the pattern or away from it.

The key with this is that you trust your teams; this is all “trust but verify” with the ignore file. You should (almost) never be blocking a team from working.

What if we could create graphs of what parts of the build took time? Something like this?

Being someone who cares about build pipelines and speed, I decided to add OpenTelemetry to our builds, and see what information we could get. It turns out that there is far more useful information available than just timings. For example:

• number of main builds; are we merging often? is this speeding up or slowing down?
• number commits merged to main at once; is our batch size going up? why?
• deployments per day; are we still moving fast? Are people scared to deploy on Friday? why?
• pass and failure ratios; are failures becoming more often? why?
• runtime of failed builds; failing builds should be fast, so we re-ordered steps so that likely failures are hit first
• what fails most often?; a test suite testing too much? flaky tests? a dependency not being locally cached (and thus unavailable sometimes)?

Terminology

The OTEL website has details on what all the terminology means, but for a brief summary:

• span: the basic units which make up a trace. They can be parented to other spans and can represent the entire build, a logical grouping of operations, or a single operation.
• trace: a collection of spans with one “root span” which has no parent.
• attributes: key-value pairs attached to spans to provide more context.
• [Otel Collector][otel-collector] - a service which accepts traces in a variety of formats and can forward them to other places. Generally, you run one of these locally and all applications send to it, and it is configured to batch, enrich, and forward to a tracing service, such as Honeycomb or Jaeger

Tracing Builds

The first step when tracing builds is to start with the overall picture: one span for the entire build. Once this is in place, you can move on to adding details, focusing your efforts on figuring out what is the most likely place to find speed improvements.

To do this, I use the trace tool, which is an opinionated CLI that creates OTEL traces for your build pipeline. If you need more flexibility or don’t like its opinions, you can either open a PR/Issue on Github, or there is the otel-cli which is much more low-level.

The trace command will send spans to localhost:4317 by default. By setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable, our traces will instead go to our local [OTEL Collector][otel-collector] instance, which is configured to send our traces elsewhere:

Install the trace tool:

env:
  OTEL_EXPORTER_OTLP_ENDPOINT: https://otel.internal.xyz:443

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Setup Trace
      uses: pondidum/trace@main
      with:
        version: "0.0.9"

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.internal.xyz:443
version="0.0.9"

curl -sSL "https://github.com/Pondidum/trace/releases/download/${version}/trace" -o /usr/bin/trace
chmod +x /usr/bin/trace

Now we can start the trace; by default this will be marked as starting when the trace start command is run; we can change this with the --when flag, which is being fed the created_at field from Github so that our trace shows when the build was started.

    - name: Start Trace
      uses: pondidum/trace/start@main

json=$(curl -sSL \
  --url "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}" \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "X-GitHub-Api-Version: 2022-11-28")

created_at=$(echo "$json" | sed -n 's/.*"created_at".*"\(.*\)".*/\1/p')
trace_parent=$(trace start "${GITHUB_REPOSITORY}/${GITHUB_WORKFLOW}" --when "${created_at}")
export "TRACEPARENT=${trace_parent}"

So that we can capture the overhead of the build job starting and the first build command running, we also store the current time as an attribute:

trace attr "first_command" $(date +%s)

At the end of the build, we finish the trace - this needs to happen no matter how the build finishes, pass or failure.

    - name: Finish Trace
      if: always()
      uses: pondidum/trace/finish@main

trap '
    rc=$?; # store the exit code
    [ $rc = "0" ] && echo trace finish ${TRACEPARENT}
    [ $rc != "0" ] && echo trace finish --error="exit ${rc}"
    trap - EXIT;
    exit
  ' EXIT INT HUP

Tracing Build Steps

Now that there is a trace for the entire build, we can start adding more details.

For example, we might want to pull a few docker containers so that we have warm caches, and want to keep track of how long this takes:

group=$(trace group start "docker_pull")

  trace task -- docker pull app:builder || true
  trace task -- docker pull app:latest || true
  trace task -- docker pull alpine:3.18 || true

trace group finish "${group}"

Tracing and Feature Flags

When you are using feature flags in your CI system, adding their state to the trace is important; it allows us to filter traces by what flags were active on a given run, letting us see if a particular flag has an impact on success rate or time taken.

# flagon supports the `TRACEPARENT` environment variable, so you
# also get spans for it querying your flag service too!
vitest=$(flagon state "ci-enable-vitest" "false" \
  --user "${email}" \
  --attr "branch=${branch}" \
  --output "template={{.Value}}" || true)

trace attr "enable_vitest=${vitest}"

# later


group=$(trace group start "testing")

  if [ "${vitest}" = "true" ]; then
    pnpm run vitest
  else
    pnpm run jest
  fi

trace group finish "${group}"

This will give us a trace with the enable-vitest flag state, and we can group by this to see if vitest is faster than jest and what effect it had on test count etc.

The Easy Example

One of the two examples given is wanting to change how an address is stored in a database. The schema starts off looking like this:

The requirement is that the schema is changed to look like this:

The way you would traditionally achieve this is with a migration:

alter table buildings
  add column street text,
  add column postcode text, -- postcodes can start with a 0, so store them as text
  add column town text,
  add column country text

update buildings set
  street    = split_part(address, ',', 1),
  postcode  = split_part(address, ',', 2),
  town      = split_part(address, ',', 3),
  country   = split_part(address, ',', 4)
where
  address != ""

alter table buildings
  drop column address

The problem with doing this is that the software using this table needs to be stopped while the update is happening; if the old version is running, the app will suddenly be trying to query a non-existing column. If the new version is running, it will also be trying to query non-existing columns.

The process has to look like this:

1. stop the old app
2. run the migration
3. start the new app

Step 2 however can be long, especially if there is lots of data. And what happens if you cannot have downtime for your service?

The Expand Contract Way

1. add a new column to the table (nullable)
2. release new software
   • for reads, read both old and new columns; prefer data in new columns if it exists
   • for writes, write to new columns
3. run a script to migrate any remaining data
4. release new software
   • only reads new columns
   • only writes new columns
5. drop the old column

This is more steps than the original method, but it means there is no downtime in your system. Also, if you make step 2 write to both columns, the migration is easily reversible as no data is lost until the fourth step runs. .

What about APIs? Services?

Expand Contract doesn’t have to just be about services either. For example, you have two services and have decided that part of service A should be migrated into service B, which has a similar system. The process is broadly similar to the database example above but with service releases instead:

1. Service B’s data model is expanded
2. Service A is released:
   • for reads, read both it’s own datastore and Service B. Return result from B if available
   • for writes, write to it’s own datastore and Service B
3. Run a script/application to migrate the remaining data
4. Release Service A:
   • uses Service B for all operations
5. Drop old data store tables

As you can see, the process is broadly similar to when implementing a database change; the only difference is some coordination with the other service team. The coordination is only to make sure their data model is ready; no need to release anything at the same time, and no downtime in either service is required.

Downsides

This may sound like a silver bullet, but as with all techniques, it has drawbacks.

The primary drawback is the extra steps required. There are multiple releases, and data migrates lazily/on demand. Then there is the extra step of migrating the remaining data, which is an additional effort.

The other drawback is a symptom of the first drawback: time. It takes far longer to do expand-contract than to have a short downtime. Depending on your application, short downtime might be the better choice to make. For example, a queue processing service which doesn’t have a synchronous API would probably be better choosing the downtime, assuming it can catch up with any messages which queue up during the downtime!

So, why not use feature flags in your CI pipeline?

TLDR

Reduce the risk of breaking a CI pipeline for all of a project’s developers by using the flagon CLI to query Feature Flags, opting developers into and out of new CI features and processes by targeting groups of developers or branch naming patterns.

What would we use them for?

There are a few things that spring to mind that we could use feature flags for:

• Migrating CI system
• Job migration
• Replacing a step
• Trying a new step

Why would using flags for this help?

The answer is risk reduction. I don’t want to break parts of the build and deployment process for everyone in the project when I make a mistake in the pipelines, and a way to help mitigate that risk is feature flags.

With a feature flag, I can quickly opt people into or out of changes to the CI system, meaning that if something goes wrong, the impact is minimal. It also allows me to monitor the effects of new vs old by having the flag states stored in our OTEL traces. This lets me ask and answer questions like: is it faster? Is it more reliable? Does it work?

One of the most significant risks is migrating from one CI system to another, which is exactly what I have been doing recently. We are leaving Truly Awful CI and migrating to Github Actions. Let’s see how that goes.

Migrating From Old to New CI

The CI process, on a high level, looks like this. The three types of deployment are ephemeral, which are short-lived environments named after the branch which created them, development, which is the common development environment, and production, which is the live application. The production and development environments are deployed to whenever something is merged to main, and ephemeral is for any other branch.

To phase the changeover to GitHub Actions, I am using the flagon CLI to access our feature flags. The query uses both the user id (committer email) and branch name so that I can target rollouts based on a group of users or perhaps with a branch name pattern.

First, I create a duplicate workflow in GitHub Actions. The docker container is published to a different tag, and all deployments have a feature flag condition added to them:

jobs:
  flags:
    outputs:
      enable_ephemeral: ${{ steps.query.outputs.enable_ephemeral }}
    steps:
    - name: Query Flags
      id: query
      run: |
        ephemeral=$(flagon state "ci-enable-gha-deployment" "false" \
          --user "${email}" \
          --attr "branch=${branch}" \
          --output "template={{.Value}}" || true)

        echo "enable_ephemeral=${ephemeral}" >> "${GITHUB_OUTPUT}"        

  # build, test, etc.

  deploy_ephemeral:
    uses: ./.github/workflows/deploy.yaml
    needs:
    - flags
    - build
    if: ${{ github.ref_name != 'master' && needs.flags.outputs.enable_ephemeral == 'true' }}
    with:
      target_env: ephemeral

Then update the old CI pipeline to wrap the deployment trigger with a flag query:

if ! flagon "ci-enable-gha-deploy" "true" \
  --user "${email}" \
  --attr "environment=ephemeral" \
  --attr "BRANCH=${branch}"; then

  awful-ci trigger "deploy - ephemeral" \
    --sha "${commit}" \
    --branch "${branch}"
fi

Note that the two CI systems are both querying the same flag, but the old system defaults to active, and the new system defaults to inactive. This means that if the flagging service (LaunchDarkly in this case) cannot be reached, only one of the systems will be doing the deployment.

Rolling out

The plan for starting the switchover was as follows:

1. ephemeral for just me
2. ephemeral environment for a small group of developers
3. ephemeral for everyone
4. development for everyone
5. production for everyone
6. WAIT
7. Remove old implementation, remove flags

During the rollout, the flag was switched on and off for various stages as small bugs were found.

For example, I discovered that the deployments only looked like they were working in GitHub Actions due to some artefacts still being uploaded to CDN by the old CI system.

Take Away

Based on my experience of using flags in this migration, it is a technique that I will be using more in the future when updating our CI pipelines.

After reading keep a changelog, I was inspired to implement this into a couple of CLI tools that I am working on at the moment: Flagon (feature flags on the CLI, for CI usage), and Cas (Content Addressable Storage for Make), but I also wanted to solve my versioning and release process.

Requirements

• Version number should be defined in only one place
• A changelog should be associated with the version number
• The binary should be able to print its version and changelog
• The release (Github Release in this case) should also have the changelog and version
• The commit released should be tagged with the version

I came up with an idea: drive everything from the changelog.

The changelog can be the source of truth: it contains the version number, date of release, and the actual changes within that version. As the changelog is written in a standardised format it should be fairly easy to parse, and thus be handled by the binary itself.

The Format

I decided to follow the format from keep a changelog as it is pretty minimal, in markdown, and easily parsable with a regex. As an example, here is one of the versions lifted from flagon’s changelog.

# Changelog

## [0.0.1] - 2022-11-14

### Added

- Exit with code `0` if a flag is `true`, and `1` otherwise
- Add `--silent` flag, to suppress console information

### Changed

- Expand what information is written to traces

Each version entry follows the same format, which is parsable by a regex:

var sectionRegex = regexp.MustCompile(`## \[(?P<version>.*)\]\s*-\s*(?P<date>.*)`)

The parser itself is very short, and the result is an array of ChangelogEntry, giving the version, date, and text of the changes.

Using the changelog from the application

The changelog is embedded in the binary using the go embed package, and can then be exposed as CLI commands. The application’s version command exposes this information with several flags:

• no flags: print the version number and git short sha

• --short: only print the version number

  ./flagon version --short
  0.0.1
  

• --changelog: pretty print the current version’s changelog entry

  ./flagon version --changelog
  

  

• --raw: causes --changelog to print the markdown as written in the changelog.md

  ./flagon version --changelog
  0.0.1 - local
  ### Added
  
  - Exit with code `0` if a flag is `true`, and `1` otherwise
  - Add `--silent` flag, to suppress console information
  
  ### Changed
  
  - Expand what information is written to traces
  

Using the changelog for Releases

In github actions when building the main branch, I use this to generate a version number, and write the current changelog entry to a temporary file:

- name: Generate Release Notes
  if: github.ref_name == 'main'
  run: |
    echo "FLAGON_VERSION=$(./flagon version --short)" >> "${GITHUB_ENV}"
    ./flagon version --changelog --raw > release-notes.md    

Which are then passed to the action-gh-release step:

- name: Release
  if: github.ref_name == 'main'
  uses: softprops/action-gh-release@v1
  with:
    name: ${{ env.FLAGON_VERSION }}
    tag_name: ${{ env.FLAGON_VERSION }}
    body_path: release-notes.md
    files: flagon

Which makes my releases match the version number of the binary, and have the correct release notes.

Further Work

This system isn’t perfect (yet), but it works well for my projects. I’ve considered extracting it into its own package, but so far with only two applications using it I haven’t hit the rule of 3 yet.

While doing continuous delivery does change the QA process, when done well, it improves everyone’s lives and makes the software better quality. Are silver bullets incoming? Not quite, but we don’t have to make someone’s life worse to improve other people’s lives.

This article is going to rely heavily on Feature Flags, so a passing familiarity is useful. In summary, feature flags are the ability to switch features on and off at runtime of the application without requiring re-deployment. Feature flags can also be used to switch on features for specific users or groups of users.

Aside; this post is a bit different from my usual style. This time I have written a story about a fictional dev team and QA team and how they move towards continuous delivery together.

TLDR

Move your QA Engineers inside the dev teams; DevOps is a way of working in a cross-functional team; this should include everyone who can contribute.

Test things early. Involve QA with features hidden behind flags. De-couple your deployments from your releases.

Setting The Scene

A team has gotten to the point where they want to switch from deploying at the end of each sprint to deployments happening as often as needed, be that 5 minutes for a text change or a few days for a bigger feature.

When they happily announce this to the rest of their organisation, the QA Team reacts with dismay; how are they going to manage to do full testing before every deployment if the team is constantly deploying? They object; this is ludicrous.

Being level-headed people, everyone decides to sit down and talk through their concerns and what to do next. The key points are written down:

• The development team wants to ship faster
• The QA team wants to test everything before it is deployed
• The management team doesn’t want to hire 10 more QAs to try and keep up

So what to do?

The first step

It is important to realise that while we want quality, not all changes are created equal; some need much closer scrutiny than others. For example, fixing some spelling mistakes probably needs no one else’s input (other than a spell-checking tool, perhaps) other than the person doing it.

The teams agree on this; after some discussion, they write down the following:

• Small fixes can be released without a QA approval

This raises a few further questions however:

1. How big is small?
2. If a small fix can be deployed without QA, what about a small feature?
3. Why is QA the final authority on what can be released?

Changing Perspective

While we could try and answer these questions (and spend countless hours deciding how many lines of code “small” is. Does it depend on line length too?), a better tactic is to investigate why the QA process is happening so late in the process.

We agree that features need QA testing, but what happens if features can be hidden? What happens if we can move the testing from “before deployment” to “before release”? Because as I have written before deploy doesn't mean release.

The team realises that they have a Feature Flagging tool available. Currently, they are not really using it, but they have been meaning to for a while. What if new features were developed behind flags? It could be deployed to production without affecting anyone, and QA could test at their leisure by enabling the flag for just one tester or for the whole team.

The QA team thinks this could work in principle, but how do they know a change is really isolated behind a flag? What happens if it escapes?

Let’s look at the process they came up with, with an example.

The New Feature

The current web application has a notification system. It’s nothing glamorous; it’s an icon in the app which gets a small dot when there is a new notification. Currently, only notifications from the system itself are supported, but there has been a request to have other parts of the system send notifications there too, along with feature requests for being able to remove read notifications and mark notifications to trigger again later.

This seems like the ideal candidate for a feature flag, so the development team writes down their next steps:

1. create a flag enable-rich-notifications
2. develop all the capabilities (API, UI)
3. deploy
4. QA can test it with the flag
5. release it to the world

Someone points out that Step 2 looks like several weeks of work on its own, and that isn’t very continuous. They break down the tasks a bit further:

1. create a flag enable-rich-notifications
2. update the API with a new /rich endpoint, which can only be queried if you have the flag.
3. create some fake data for the /rich endpoint to return
4. create a new UI component which uses the new endpoint
5. update the application to use the new component if you have the flag and the old component otherwise

With implicit “Deploy” steps after each step. This seems reasonable to the development team, but the QA team still have questions: when should they test the UI? once it is fully complete? And how do they know it is working?

The development team also realises that the new notifications system will be using the same data model as the old system, and they need to make sure the old system continues to work correctly. Come to think of it, QA involvement would be useful here too…

Moving QA Earlier

As an aside, I find it much better to have a QA Engineer be part of the development team. The whole DevOps thing is about working in one cross-functional team, and why should QA, Security, or anyone else be excluded from this? Regrettably, this is a slow organisational change to make, so we come up with ways to make it work as best we can and iterate towards the embedded QA model.

When the new notifications feature is being designed, the development team requests someone from QA be involved from the start; there are things which they should be aware of, and have useful input on.

1. Update the data model in place with the new design
2. QA to test it in an isolated environment; no changes expected
3. Deploy

The QA points out that as far as they are aware, there aren’t any tests for the old notifications system; it was so barebones and unused that no one bothered. The QA also points out that they have been evaluating switching to a more code-first UI automation tool, and this might be the ideal candidate to start with, and could they put the UI testing code in the repo alongside the feature?

This is well received by the dev team; this might help the regressions they keep causing when a selector is updated, and the UI tests break; if it’s all in the same repository, grep can find all the instances at once! It’s win-win again.

The again updated list of actions is now:

1. QA creates UI tests for the current system (and verifies against isolated environment)
2. Devs Update the data model
3. QA verifies nothing has changed in a staging environment
4. Deploy

Note there are no flags involved yet!

The team goes ahead and makes all the discussed changes; however, when the new UI tests are run against the environment with the new data model, they break, and it isn’t apparent why. The QA and the developer sit down and dig around until they find the problem; the format of a field has changed slightly, and the UI tests are catching the problem.

They fix the issue, test again, and this time deploy into production.

The New API and UI

Now that involving QA earlier has been tried and seems to work, the team decide to move forward with the API changes and the feature flag for the original version and the rich version of notifications.

The flag is created, the API is wrapped with a check for the flag, the developers test it works, and deployment is done. No problems so far.

The UI is up next; as this is early on in the process, the dev team, designer, and QA engineer are all sitting together to figure out exactly how it will work. As the QA is present, they can start writing outlines for UI testing. As the code for tests is alongside the application code, the developers can help keep the tests working as they flesh out the UI, and they might even write a test themselves too.

The interesting realisation comes that with a feature flag, two QAs can be involved at once; one is running tests for the flag off, and one is running the tests for the flag on. It isn’t required to be like this of course, but it does mean you can spread the work further.

Features are developed, tests are written, and deployments are done.

Ready for Release

The team, which now includes the QA by default, is getting close to being ready to release their new rich notifications to the world. They have one more test they would like to conduct: what is the load like when users are re-notifying themselves? How do they even go about testing this?

The answer, perhaps unsurprisingly, is a feature flag. In this case, a new feature flag called load-generator-rich-notifications. When this flag is enabled, the rich notifications system is still hidden, but a small piece of code randomly activates notifications for re-notifying and varying intervals. The team can switch it on for a few percent of users and then watch their traces and monitoring systems to keep an eye on the health of the system.

They can add more and more users to the test until they are happy. Then disable the load generator and clean up all the mess it has left.

Aside; this is how Facebook Messenger was load tested before the public saw anything!

Wrapping Up

The key takeaway from this is that QA is an important part of the delivery lifecycle. Your QA Engineers are smart people who want to make things better, so involve them early and see what conversations and ideas can happen when you put smart people together and task them with making things better.

This was a lot longer than it sounded in my head when I thought this up while cycling home, but I like how it’s gone. I might even turn this into a talk to give to clients if it is well received.

also deploy != release

This turned out to be somewhat controversial until we discussed what I specifically meant by deploy and release.

As with all things, agreeing on definitions or understanding what someone means when they use a specific term is essential, so I thought I would write down a short blog post on it.

To start with, a picture helps in my experience:

TLDR

Deploy is the act of moving software into an environment and running it.

Release is the process of making a feature visible to a user or subset of users.

Read on for longer descriptions.

Build

A build is a process, usually run in a central CI system, which produces one or many artefacts. A build process can consist of testing, linting, compilation, transpilation, or any other number of steps.

Artefact

An artefact is the result of the build. It has a version, and can be deployed to an environment. An artefact can contain many features which can be uniquely controlled.

It should also have metadata embedded in it to link it back to the build which produced it and also to the source it was built from.

If a build is producing multiple different versioned artefacts, having a way to link them all to the same process is important.

Version

A version is an identifier which uniquely labels an artefact. This can be a chosen format such as SemVer, a datestamp, or a commit hash. It could also be an auto-incrementing build counter.

Deploy

The process of getting an artifact into an environment. Doesn’t necessarily cause any visible changes to a user or client of the application.

Environment

A location running the application. An environment may have multiple applications running, making up one complete product.

Release

Release is switching on (or off) a feature to users, independent of deploy. This is usually done with Feature Flags, and can mean releasing to all users, or just a subset (either a random sample or specific users.)

You can also automate feature rollout by combining it with your observability data, rolling out to more users over time if, for example, error rates don’t increase.

TLDR

We can use Make and a couple of short shell scripts to implement file content-based caching and read/write that cache to remote storage, such as S3. The demo repository contains a version using minio for ease of demonstration.

Bazel

However, Bazel has quite a high barrier to entry; there are two drawbacks: a specialised build language and the need to host extra components. While the specialised language is not much of a drawback, the hosting side is more of an issue. If you wish to have a shared cache (which is required to get fast builds), you need to either run bazel-remote, which is not actually part of the Bazel project, and requires some shared storage such as S3, or Nginx, which again requires some shared storage somewhere.

It boils down to not wanting to have to maintain a lot of infrastructure on top of all the usual CI bits just to have fast builds.

So what about Make?

Whereas Bazel’s caching method is based on a hash of the input artifacts, Make’s is based on the input sources and output artifacts’ lastModified times.

I tried adding distributed caching to Make by copying the output artifacts to S3, and on the next build (on a different agent), restoring them to the working directory, and seeing what would happen.

As both Git and S3 set the file lastModified dates to the time they ran, the build process either never ran (artifacts are newer than source), or always ran (sources are newer than artifacts).

This sent me on a relatively short journey to see if I could add hash-based change detection to Make, without recompiling Make.

Spoiler: it is!

Hashing

The first question is how to hash all our source files reliably. It turns out you can do all of this with sha256sum in a one-liner:

1
2
3
4
5

find src -iname "*.ts" -print0 \
  | xargs -0 sha256sum \
  | LC_ALL=C sort \
  | sha256sum \
  | cut -d" " -f 1

This does the following:

1. Find all typescript files (for example)
2. Sort all the files using the “simple” locale
3. generate a hash of the content of each file
4. generate a hash of all the path+hash pairs
5. trim the output to only the hash

Now that I have a hash for the files, its time to figure out how to use that with Make.

We’ll be trying to make this totally legitimate build target in a Makefile run only when content changes, regardless of file edit dates:

dist/index.js: $(shell find src -iname "*.ts" -not -iname "*.test.ts")
	@echo "==> Building"
	@sleep 3s
	@mkdir -p "dist"
	@echo "compiled at $(shell date)" > "$@"
	@echo "==> Done"

All this build step does is write the current date to a file called dist/index.js. To make this more realistic, you could change the sleep 3s to sleep 10m ;)

The idea I have to make this hashing work is to use a file that I control and mess with its edit date:

1. Check if a file called ${current_hash} exists
2. If it doesn’t exist, write the current timestamp to a new file called ${current_hash}
3. If it does exist, set the file ${current_hash}’s modified date to the timestamp stored inside the file
4. echo the filename so that it can be consumed by Make

This way, the file’s edit date will change whenever the hash changes, and if the hash doesn’t change, we leave the edit date as is (which fixes the S3 file edit date being wrong.)

Code wise, it’s a few lines of shell script:

STORE_PATH="${CAS_STORE_PATH:-.state}"
mkdir -p "${STORE_PATH}"

now=$(date "+%s")
current_hash=$(echo "$@" | xargs -n 1 | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d" " -f 1)

key="${current_hash}.sha256"
state_path="${STORE_PATH}/${key}"

if [ -f "${state_path}" ]; then
  # this hash is in the state store, re-apply it's date to the state file (as something like s3
  # sync might have changed the file's modified date.
  last_date=$(sed -n 's/date:\s*\(.*\)/\1/p' "${state_path}")
  touch -d "@${last_date}" "${state_path}"

else
  # this is a new hash
  echo "date: $now" > "${state_path}"
fi

echo "${state_path}"

And the usage inside the makefile is only adding an extra $(shell ./build/cas.sh .... ) around our dependency list:

dist/index.js: $(shell ./build/cas.sh $(shell find src -iname "*.ts" -not -iname "*.test.ts"))
	@echo "==> Building"
	@sleep 3s
	@mkdir -p "dist"
	@echo "compiled at $(shell date)" > "$@"
	@echo "==> Done"

Testing

We have a few test cases to cover:

1. Entirely blank repository; after all, it should work when you first run git clone

   $ git clean -dxf
   $ make build
     ==> Building
     ==> Done
   

2. Files have not changed at all; it should have the same behaviour as normal make, i.e. nothing happens

   $ git clean -dxf
   $ make build
     ==> Building
     ==> Done
   $ make build
     make: Nothing to be done for 'build'.
   

3. File lastModified date has changed; this should cause nothing to happen also, as the content of the files hasn’t changed:

   $ git clean -dxf
   $ make build
     ==> Building
     ==> Done
   $ touch src/index.ts
   $ make build
     make: Nothing to be done for 'build'.
   

4. File content has changed (but lastModified hasn’t); forcing a file to have different content with the same lastModified to show that its only content that matters:

   $ git clean -dxf
   $ make build
     ==> Building
     ==> Done
   $ set old_date (date -r src/index.ts "+%s")
   $ echo "// change" >> src/index.ts
   $ touch -d @$old_date src/index.ts
   $ make build
     ==> Building
     ==> Done
   

Collecting Assets

Before we can implement remote caching, we need to be able to mark what assets should be included for the given source hash.

I initially tried to achieve this by passing the name of the make target into the cas.sh script, but this involves a lot of repetition as the special “target name” make variable ($@) doesn’t work if it’s included in the source list:

dist/index.js: $(shell ./build/cas.sh $@ $(shell find src -iname "*.ts" -not -iname "*.test.ts"))
  @echo "==> Building"

Besides not working, this is also not very flexible; what happens if you have other artifacts to store, other than the one acting as your make target? What happens if you are using a sentinel file instead of actual output as a make target? or a .PHONY target?

The answer to these questions is an extra script to store artifacts, called artifact.sh, which writes the path of an artifact to the hash file with a prefix of artifact: :

#!/bin/sh

key="$1"
artifact="$2"

if [ -n "$CAS_VERBOSE" ]; then
  echo "Storing ${artifact}"
fi

echo "artifact: ${artifact}" >> "${key}"

Which is used in the makefile, utilising some of Make’s magic variables: the $< is the filepath to the first dependency (which is the hash file produced by cas.sh), and usually, we use $@, which is the name of the target being built. In this example, a second invocation marks another file as an artifact of the make rule:

	@./build/artifact.sh "$<" "$@"
	@./build/artifact.sh "$<" "artifacts/coverage.json"

Remote Caching

As mentioned earlier, I want to manage as little infrastructure for this as possible, so cloud object storage such as S3 is ideal. For local testing, we’ll use a minio docker container.

First up, as I want this to be reasonably extensible, rather than hardcode s3 logic into the scripts, I check for an environment variable CAS_REMOTE, and execute that with specific arguments if it exists, both in cas.sh and artifact.sh:

#!/bin/sh

STORE_PATH="${CAS_STORE_PATH:-.state}"
mkdir -p "${STORE_PATH}"

now=$(date "+%s")
current_hash=$(echo "$@" | xargs -n 1 | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d" " -f 1)

key="${current_hash}.sha256"
state_path="${STORE_PATH}/${key}"

if [ -f "${CAS_REMOTE}" ]; then
  ${CAS_REMOTE} fetch-state "${key}" "${state_path}"
fi

if [ -f "${state_path}" ]; then
  # this hash is in the state store, re-apply it's date to the state file (as something like s3
  # sync might have changed the file's modified date.
  last_date=$(sed -n 's/date:\s*\(.*\)/\1/p' "${state_path}")
  touch -d "@${last_date}" "${state_path}"

  if [ -f "${CAS_REMOTE}" ]; then
    artifacts=$(sed -n 's/artifact:\s*\(.*\)/\1/p' "${state_path}")

    if [ -n "${artifacts}" ]; then
      echo "${artifacts}" | xargs "${CAS_REMOTE}" fetch-artifacts "${key}"
      echo "${artifacts}" | xargs touch -d "@${last_date}"
    fi
  fi

else
  # this is a new hash
  echo "date: $now" > "${state_path}"

  if [ -f "${CAS_REMOTE}" ]; then
    ${CAS_REMOTE} store-state "${key}" "${state_path}"
  fi
fi

echo "${state_path}"

#!/bin/sh

key="$1"
artifact="$2"

if [ -n "$CAS_VERBOSE" ]; then
  echo "Storing ${artifact}"
fi

echo "artifact: ${artifact}" >> "${key}"

if [ -f "${CAS_REMOTE}" ]; then
  hash="$(basename "${key}")"
  ${CAS_REMOTE} store-artifact "${hash}" "${artifact}"
  ${CAS_REMOTE} store-state "${hash}" "${key}"
fi

The main point is keeping how state and artifacts are copied around separate from the logic of how their lasModified dates are manipulated. In the case of the fetch-artifacts call, we first pull all the artifacts using the remote script, and then update their lastModified dates to match the state’s lastModified date:

  if [ -f "${CAS_REMOTE}" ]; then
    artifacts=$(sed -n 's/artifact:\s*\(.*\)/\1/p' "${state_path}")

    if [ -n "${artifacts}" ]; then
      echo "${artifacts}" | xargs "${CAS_REMOTE}" fetch-artifacts "${key}"
      echo "${artifacts}" | xargs touch -d "@${last_date}"

S3 Remote Cache

The S3 remote script implements four functions: fetch-state, fetch-artifacts, store-state, and store-artifact, with the convention that the first parameter is always the key - e.g. the state file name.

In this demo, the actual S3 command is defaulted to use the local minio endpoint, unless CAS_S3_CMD is specified, as I cannot find a way to set the --endpoint-url via an environment variable directly:

s3="${CAS_S3_CMD:-"aws --endpoint-url http://localhost:9000 s3"}"

This is used in each of the four functions to interact with S3. For example, to fetch the state; note how we use both --quiet and >&2 to redirect all output to stderr, as anything on stdout make will pick up as a filename, causing issues. We also use || true for fetching state, as it might not exist:

fetch_state() {
  key="$1"
  state_path="$2"
  log "$key: Fetching remote state to $state_path"


  $s3 cp "s3://${bucket}/state/${key}" "${state_path}" --quiet >&2 || true
}

Testing Remote Caching

First, we need to start our minio container and configure the environment:

docker-compose up -d
export "AWS_ACCESS_KEY_ID=minio"
export "AWS_SECRET_ACCESS_KEY=password"
export "CAS_REMOTE=./build/remote_s3.sh"
export "CAS_S3_BUCKET_PATH=makestate/cas-demo/"
export "CAS_READ_ONLY=0"
export "CAS_VERBOSE=1"

Also, we need to create the S3 bucket using the AWS cli:

aws --endpoint-url http://localhost:9000 s3 mb s3://makestate

We’re now ready to try a build:

$ git clean -dxf
$ make build
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Fetching remote state to .state/c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Storing state from .state/c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256
  977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256: Fetching remote state to .state/977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256
  977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256: Storing state from .state/977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256
  ==> Building
  ==> Done
  Storing dist/index.js
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Storing artifact dist/index.js
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Storing state from .state/c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256
$ cat dist/index.js
  compiled at la 17.9.2022 13.16.48 +0300

If we now clean the repository and build again, we should end up with all the artifacts from the original build but no build process actually running:

$ git clean -dxf
$ make build
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Fetching remote state to .state/c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256
  c2bac686e507434398d9bf4e33f63f275dfd3bfecfe851d698f8f17672eeccbe.sha256: Fetching dist/index.js
  977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256: Fetching remote state to .state/977e50e9421f0a2749587de6a887ba63f2ddf9109d27ab7cae895a6664b2711a.sha256
  make: Nothing to be done for 'build'.
$ cat dist/index.js
  compiled at la 17.9.2022 13.16.48 +0300

Extra Features

I added a CAS_READ_ONLY environment variable, which by default prevents the scripts from pushing state and artifacts to remote storage but does allow fetching from storage. The idea of this is that local development can make use of the caches, but only CI machines can write to the cache:

if [ -f "${CAS_REMOTE}" ] && [ "$CAS_READ_ONLY" = "0" ]; then

Wrapping Up

Overall, I am very happy with how this has gone; it all works, and hopefully I’ll be testing it in parallel to normal build processes over the coming weeks.

The problem occours when I am embedding code from another repository; often there will be tweaks or bug fixes, and keeping the code in the blog post in sync with the code in the other repo is annoying and manual.

To make life easier for myself, I wrote a shortcode called git-embed, which at build time will fetch the specified file, and embed either the whole file, or a line range in a highlighted code block. It supports line ranges (start, finish), and it will use the file’s extension as the highlight language, unless you override with lang parameter.

Usage is like this:

{{< git-embed
  user="Pondidum"
  repo="pondidum.github.io"
  ref="master"
  file="layouts/shortcodes/git-embed.html"
  lang="go"
>}}

Which I am using to embed the shorcode’s own shortcode:

{{- $user := .Get "user" -}}
{{- $repo := .Get "repo" -}}
{{- $ref := .Get "ref" -}}
{{- $file := .Get "file" -}}


{{- $source := printf "https://github.com/%s/%s/blob/%s/%s" $user $repo $ref $file -}}
{{- $url := printf "https://raw.githubusercontent.com/%s/%s/%s/%s" $user $repo $ref $file -}}
{{- $remote := resources.GetRemote $url -}}

{{ $all_lines := split $remote.Content "\n" -}}

{{- $start := 1 -}}
{{- if isset .Params "start" -}}
  {{- $start = (.Get "start") | int -}}
{{- end -}}

{{- $finish := len $all_lines -}}
{{- if isset .Params "finish" -}}
  {{ $finish = (.Get "finish") | int -}}
{{- end -}}

{{- $start = math.Max $start 1 -}}
{{- $finish = math.Min $finish (len $all_lines) -}}

{{- $lines := slice -}}
{{- range $index, $num := (seq (add $start -1) (add $finish -1)) -}}
  {{- $lines = $lines | append (index $all_lines $num) -}}
{{- end -}}

{{- $lang := strings.TrimPrefix "." (path.Ext $file) -}}
{{- if isset .Params "lang" -}}
  {{- $lang = .Get "lang" -}}
{{- end -}}

{{- $options := "" -}}
{{- if isset .Params "options" -}}
  {{- $options = .Get "options" -}}
{{- end -}}

{{- highlight ( delimit $lines "\n") $lang $options -}}

<footer style="margin: 10px auto;">
  <small><a href="{{- $source -}}">Source</a></small>
</footer>

Future Improvements

I’d quite like to add a parameter that would allow me to embed a named function, rather than a line range. I think I’d use Tree-Sitter to do this, but Hugo doesn’t seem to have a way to execute an arbitrary command on build, so I’d need to make a small API that would do all the work… So that can wait for when the itch needs scratching more than this.

TLDR

Out of 14 types of diagram there are 3 that I use on a regular basis: Activity Diagram, State Machine Diagram, and Sequence Diagram. I think the Timing Diagram is borderline, but I can only think of a couple of occasions when it has been useful.

Writing the diagrams in text and rendering them with Mermaid makes including them in documentation and websites painless, and the project is under active development.

What I use often

The diagram I use the most is the Sequence Diagram; It’s a great way to document how multiple systems (or micro services) will interact with each other. This diagram type has worked really well on both physical whiteboards, and in documentation. For example, part of a diagram I used to help design a system recently:

Next most used is the Activity Diagram, also more commonly known as a Flow Chart. This is mostly used when discribing an algorithm or process without needing to indicate different participants in the algorithm.

The last type that I use often is the State Machine Diagram; I think that State Machines themselves are an under utilised design pattern, and that a lot of complex problems can be rendered into the state pattern quite easily.

In a previous job there was a state machine with around 34 different states; being able to render this in a diagram made understanding the process much more approachable; even our support staff used the diagram to answer user questions.

For example, a line processor could be represented as follows, where depending on the kind of error the process will either terminate, or skip the line and wait for the next:

Who else is using UML?

The Mermaid library to render these kind of diagrams is being integrated all over the place: Github uses it to handle any ```mermaid blocks in your markdown files.

Likewise, Hugo uses it when rendering your markdown content too (which is what has drawn these nice diagrams here.)

There are extensions for vscode to render them too.

Why should I care?

Because explaining concepts is hard; and pictures often help. The old phrase of a picture being worth a thousand words is applicable here; its far easier to glance over a diagram than read a few paragraphs of prose.

This being the case, it is a good idea to have some common formats, standards or symbols to use, making the diagrams even clearer to people who already know the symbols (and don’t forget to explain them if they don’t know, and perhaps include a legend in your diagram.)

When creating these diagrams was done in tools like Visio, and then screenshots embedded in documents (usually buried in a wiki where no one will find them), the barrier to making a useful diagram was high. Being able to embed them in the markdown in your repo, so they can be read from code even if they aren’t rendered lowers that barrier to making useful diagrams considerably.

So do your co-workers, contributors, and your future-self a favour, and add some simple diagrams to your docs.

For example, I have a CLI, that creates a cluster of machines for a user; the machines use IAM Authentication with Vault so that they can request certificates on boot. The trouble with this application is that it is slow; it takes 175 seconds on average to provision the machines, write the IAM information to Vault, and then re-run the cloud-init script on all the machines in the cluster (as when they first booted, the configuration hadn’t been written to Vault yet.) so that they can request a certificate. The process is roughly this:

• Create infrastructure
• Write configuration to Vault
• Wait for the machines to be ready
• Wait for SSH
• Re-run cloud-init

The CLI can’t write the configuration to Vault before the machines boot, as the configuration values are from the same infrastructure stack as the machines themselves. You can see the process in the Honeycomb trace UI (with more details about what infra is created thanks to my pulumi-honeycomb stream adaptor):

I don’t want to make two separate stacks for this, one containing IAM Roles and other configuration data, the other containing all the other infrastructure (load balancers, auto-scale groups, etc.) But what if I could dynamically change what the stack does?

By adding an IsInit property to the configuration of the stack, we can change the pulumi program to return early when the value of IsInit is true, meaning we only create the minimal amount of infrastructure for the configuration call to succeed:

func DefineInfrastructure(ctx *pulumi.Context, cfg *ClusterConfiguration) error {

  role, err := iam.NewRole(ctx, Name+"-iam-role", &iam.RoleArgs{
    NamePrefix:       pulumi.String(Name),
    AssumeRolePolicy: allowEc2Json,
  })

  _, err = iam.NewRolePolicy(ctx, Name+"-iam-policy-cluster", &iam.RolePolicyArgs{
    NamePrefix: pulumi.String(Name),
    Role:       role.ID(),
    Policy:     findMachinesJson,
  })
  ctx.Export("role-arn", role.Arn)

  if cfg.IsInit {
    return nil
  }

  asg, err := autoscaling.NewGroup(ctx, Name+"-asg", &autoscaling.GroupArgs{
    LaunchConfiguration: createLaunchConfiguration(ctx, cfg, role),
    VpcZoneIdentifiers:  cfg.ZoneIdentifiers,
    DesiredCapacity:     cfg.ClusterSize,
    MinSize:             cfg.ClusterSize,
    MaxSize:             cfg.ClusterSize,
  })
  ctx.Export("asg-name", asg.Name)

  return nil
}

Now that the stack can be run to create only partial infrastructure, the process changes to this:

• Create minimal infrastructure
• Write configuration to Vault
• Create remaining infrastructure

But is the new process faster? I had hoped it would be a little faster, as waiting for cloud-init and SSH can take a while, and thankfully, it is significantly faster. It takes on average 98 seconds, so around 77 seconds faster.

Comparing the before and after traces, I can see that the additional pulumi call adds 20 seconds to the processes, but the consul_configure span drops from 100 seconds to 3.5, which is quite the speed increase.

What about Terraform?

This is still possible to do with a terraform stack, but not in a pleasant way; in pulumi I can return early from the infra function, but with terraform, I would have to add a count = var.is_init ? 0 : 1 to every resource I didn’t want to create up front, which quickly becomes unwieldy.

There is also the downside of not being able to embed the Terraform inside a CLI tool like I can with Pulumi.

Overall, I am happy with how this has turned out. The diff for enabling this optimisation is 3 files changed, 15 insertions, 7 deletions, which included some explanatory comments!

I was running a very simple Nginx proxy, relaying an internal service to the outside world. The internal service is behind an AWS ALB, and the Nginx configuration was proxying to the ALB’s FQDN:

http {
  server {
    listen              8000;
    server_name         server.example.com;

    location ~* ^/some/path {
      proxy_pass              https://some.internal.alb.address.amazonaws.com;
      proxy_set_header        Host $host;
      proxy_read_timeout      120;
      proxy_ignore_headers    Cache-Control;
      proxy_ignore_headers    Expires;
      proxy_ignore_headers    Set-Cookie;
    }
  }
}

The proxy was working fine for several weeks, until suddenly it wasn’t. To make matters more strange, when we checked the internal site directly, it showed as up and responding. No deployments of any services had happened, and we had made no changes in any infrastructure either. We restarted the Nginx service, and everything started working again.

The first is that AWS’s can, and does, change the IP addresses associated with load balancers. This can happen for many unknown reasons as the underlying implementation of the AWS load balancers is a black box. One known reason is the load balancer scaling to handle more or less traffic. There is no API that we are aware of that allows you to see when these changes have happened; the only way we know is to run dig in a loop and send the results to our observability tool when they change.

The second detail is how Nginx resolves DNS. My initial expectation was that it worked like most DNS clients, and would query an address on the first request and then again after the TTL had elapsed. It turns out my assumption was wrong, and that by default, Nginx queries addresses once on startup, and never again.

So with these two facts, we can see why the proxy stopped working at some point; the target ALB had removed whichever IP address(es) Nginx had received from DNS at startup. There are two different ways this can be fixed.

The first way is to force Nginx to cache all IPs resolved for a fixed time window:

http {
+  resolver_timeout 30s;

  server {
    listen              8000;
    server_name         server.example.com;

    location ~* ^/some/path {

The second fix is to cause Nginx to re-resolve the upstream when it’s DNS record expires (based on the DNS TTL):

http {
  server {
    listen              8000;
    server_name         server.example.com;
+    set $upstream some.internal.alb.address.amazonaws.com;

    location ~* ^/some/path {
-     proxy_pass              https://some.internal.alb.address.amazonaws.com;
+     proxy_pass              https://$upstream;
      proxy_set_header        Host $host;
      proxy_read_timeout      120;
      proxy_ignore_headers    Cache-Control;
      proxy_ignore_headers    Expires;
      proxy_ignore_headers    Set-Cookie;
    }

While I am glad there are two easy ways to solve this issue, I still find the default “only resolve once at startup” behaviour odd, as it goes against the Principle of least surprise; I expect Nginx to re-query based on the TTL of the DNS Record. I suspect this behaviour exists for performance reasons, but I don’t know for sure.

As a Nomad user, I wanted to do something similar for my clusters, so I set about seeing how it would be possible. It turns out; it is much easier than I expected! While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API.

The Setup

We’re going to build an automated backup operator! We’ll use the Nomad Streaming API to watch for jobs being registered and deregistered. If a job has some metadata for auto backup, we’ll create (or update) a backup job. If a job is deregistered or doesn’t have any auto backup metadata, we’ll try to delete a backup job if it exists.

The complete source code is available in the Nomad-Operator repo on my GitHub.

Consuming the Nomad Streaming API

The Nomad Go API library makes it easy to consume the streaming API, handling all the details, such as deserialisation for us.

The client is created with no additional parameters, as the Address and SecretID will be populated from environment variables automatically (NOMAD_ADDR and NOMAD_TOKEN respectively):

client, err := api.NewClient(&api.Config{})
if err != nil {
  return err
}

As we want to only listen to jobs that have been modified after our application deploys, we need to query what the current job index is at startup:

var index uint64 = 0
if _, meta, err := client.Jobs().List(nil); err == nil {
  index = meta.LastIndex
}

Next, we use the EventStream API and subscribe to all job event types (in practice, this means JobRegistered, JobDeregistered, and JobBatchDeregistered):

topics := map[api.Topic][]string{
  api.TopicJob: {"*"},
}

eventsClient := client.EventStream()
eventCh, err := eventsClient.Stream(ctx, topics, index, &api.QueryOptions{})
if err != nil {
  return err
}

The Stream(...) call itself returns a channel which we can loop over forever consuming events, ignoring the heartbeat events:

for {
  select {
  case <-ctx.Done():
    return nil

  case event := <-eventCh:

    if event.IsHeartbeat() {
      continue
    }

    c.handleEvent(event)
  }
}

Finally, this operator only cares about jobs being registered and deregistered, so we loop through all the events and only handle the JobRegistered and JobDeregistered events:

for _, e := range event.Events {

  if e.Type != "JobRegistered" && e.Type != "JobDeregistered" {
    return
  }

  job, err := e.Job()
  if err != nil {
    return
  }

  c.onJob(e.Type, job)
}

Handling Jobs

When we see jobs, we need to handle a few different cases:

• Jobs which are backup jobs themselves should be ignored
• Jobs without backup settings should have their backup job removed (if it exists)
• Jobs with backup settings should have their job created (or updated if it exists)
• Deregistered jobs should have their backup job removed (if it exists)

We’re using the job level meta stanza in the .nomad files for our settings, which looks something like this:

task "server" {
  meta {
    auto-backup = true
    backup-schedule = "@daily"
    backup-target-db = "postgres"
  }
}

func (b *Backup) OnJob(eventType string, job *api.Job) {

  if strings.HasPrefix(*job.ID, "backup-") {
    return
  }

  backupID := "backup-" + *job.ID
  settings, enabled := b.parseMeta(job.Meta)

  if eventType == "JobDeregistered" {
    b.tryRemoveBackupJob(backupID)
    return
  }

  if !enabled {
    b.tryRemoveBackupJob(backupID)
    return
  }

  b.createBackupJob(backupID, settings)
}

Attempting to remove the job is straightforward as we don’t care if it fails - it could be that the job doesn’t exist, or is already stopped, or any other number of reasons, so we can use the Deregister() call and discard the output:

func (b *Backup) tryRemoveBackupJob(jobID string) {
  b.client.Jobs().Deregister(jobID, false, &api.WriteOptions{})
}

Creating the backup job involves rendering a go template of the nomad file we will use, and then calling Register to submit the job to Nomad. We’re using the fact that our backup IDs are stable, so re-running the same backup ID will replace the job with a new version.

func (b *Backup) createBackupJob(id string, s settings) error {

  t, err := template.New("").Delims("[[", "]]").Parse(backupHcl)
  if err != nil {
    return err
  }

  var buffer bytes.Buffer
  if err := t.Execute(&buffer, s); err != nil {
    return err
  }

  backup, err := jobspec.Parse(&buffer)
  if err != nil {
    return err
  }

  _, _, err = b.client.Jobs().Register(backup, nil)
  return err
}

The nomad file is embedded using the Go embed package to store the .nomad file in the binary, so we still have a single artefact to deploy:

//go:embed backup.nomad
var backupHcl string

And the backup.nomad file itself is a go template with custom delimiters ([[ and ]]) for fields, as the .nomad file, can contain {{ }} when using the inbuilt templating for populating secrets, amongst other things:

job "[[ .JobID ]]" {
  datacenters = ["dc1"]

  type = "batch"

  periodic {
    cron             = "[[ .Schedule ]]"
    prohibit_overlap = true
  }

  group "backup" {

    task "backup" {
      driver = "docker"

      config {
        image   = "alpine:latest"
        command = "echo"
        args    = [ "backing up [[ .SourceJobID ]]'s [[ .TargetDB ]] database" ]
      }

      env {
        PGHOST     = "postgres.service.consul"
        PGDATABASE = "[[ .TargetDB ]]"
        AWS_REGION = "eu-west-1"
      }
    }
  }
}

Testing (Manual)

The great thing about developing against Nomad is that testing is straightforward. We can start a local copy by running nomad agent -dev, and then run our application locally to check it works properly, before needing to package it up into a Docker container and deploying it to a real cluster. It also doesn’t need to be packaged in a container for Nomad; we could use Isolated Exec or Raw Exec too.)

There is a start.sh script in the repository which will use tmux to start 3 terminals, one to run a Nomad agent in dev mode (nomad agent -dev), one to build and run the operator (go build && ./operator), and one to register and deregister nomad jobs.

When all is ready, submit the example job with the following command:

nomad job run example.nomad

Will cause the following output in the operator’s terminal:

==> JobRegistered: example (pending)...
    Registering backup job
    Backup created: backup-example
--> Done
==> JobRegistered: backup-example (running)...
    Job is a backup, skipping

We can also check the Nomad UI, running on http://localhost:4646, which shows our two jobs:

Note how the example job is a service, which continuously runs, and the backup-example is a periodic job, scheduled to run daily.

Removing the example job with the following command:

nomad job stop example

This will be seen by the operator, which will remove the backup job:

==> JobDeregistered: example (running)...
    Trying to remove a backup, if any
==> JobDeregistered: backup-example (dead)...
    Job is a backup, skipping

Note how it also sees the backup-example job being deregistered and ignores it as, in our case, backups don’t have backups!

Testing (Automated)

We can also write automated tests in two ways for this operator; Tests that run against a saved or synthetic event stream, and tests that work in the same way as the manual test; start Nomad, run a test suite; stop Nomad.

Reading from a file of known events, we can test the handleEvent function directly:

seenEvents := []string{}

c := NewConsumer(nil, func(eventType string, job *api.Job) {
  seenEvents = append(seenEvents, eventType)
})

for _, line := range strings.Split(eventsJson, "\n") {
  var events api.Events
  json.Unmarshal([]byte(line), &events)

  c.handleEvent(&events)
}

assert.Len(t, seenEvents, 2)
assert.Equal(t, []string{"JobRegistered", "JobDeregistered"}, seenEvents)
}

The other way of testing is running a nomad instance in dev mode next to the application and registering jobs to it. Usually, when doing this, I would start the Nomad application before running the tests and then stop it after, to save the time of waiting for Nomad to start between each test:

wait := make(chan bool, 1)

client, err := api.NewClient(&api.Config{})
assert.NoError(t, err)

seenJobID := ""
c := NewConsumer(client, func(eventType string, job *api.Job) {
  seenJobID = *job.ID
  wait <- true
})

go c.Start()

//register a job
job, err := jobspec.Parse(strings.NewReader(withBackupHcl))
assert.NoError(t, err)

client.Jobs().Register(job, &api.WriteOptions{})

// block until the job handler has run once
<-wait

assert.Equal(t, *job.ID, seenJobID)

As this is running against a real copy of Nomad, we need to wait for jobs to be registered and only stop our test once things have been processed; hence we use a bool channel to block until our job handler has seen a job.

In a real test suite, you would need to make the job handler filter to the specific job it is looking for; as this would prevent shared state issues (currently this will stop after any job is seen), and thus allow you to run the tests in parallel.

Deployment

No operator pattern would be complete without pushing the operator itself into the Nomad cluster, and while we could just run the binary directly in Nomad (utilising the Artifact Stanza and Isolated Exec), its probably easier to create a docker container.

We have a single Dockerfile with a multistage build so that our output container only contains the binary itself, rather than all the layers and intermediate artefacts from the build process:

FROM golang:1.16.10-alpine3.14 as builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . ./
RUN go build


FROM alpine:3.14 as output
COPY --from=builder /app/operator /usr/local/bin/operator

Once the container is built and tagged:

docker build -t operator:local .

We can verify it works as intended by running the container directly; --net=host is passed to the run command so that the operator can connect to Nomad on localhost:4646, rather than having to pass in our host IP through an environment variable. If you want to do this, add -e NOMAD_ADDR=http://SOME_IP_OR_HOST:4646 to the docker run command:

docker run --rm -it --net=host operator:local

Assuming we’re happy, we can run the Operator container in our local Nomad instance without pushing it:

task "operator" {
  driver = "docker"

  config {
    image = "operator:latest"
  }

  template {
    data = <<EOF
    {{ with secret "nomad/creds/operator-job" }}
    NOMAD_TOKEN={{ .Data.secret_id  | toJSON }}
    {{ end }}
EOF
    destination = "secrets/db.env"
    env = true
  }

  env {
    NOMAD_ADDR = "nomad.service.consul"
  }
}

Wrapping Up

The Operator Pattern is a great way to handle everyday tasks that a cluster operator would normally, and I have used it to handle things like automatic backups, certificate generation (at least until Vault supports LetEncrypt), and job cleanup (for example, developer branch builds only stay in the cluster for 3 days.)

As luck would have it, I had pushed for a change in tagging format at a client not so long ago as the method we were using didn’t make a lot of sense and, worst of all, it was a manual process. One of the things that I push at all clients is documenting all architectural decisions made, in the form of Architecture Decision Records, so I’m reproducing it here, with a few details changed to mask where this happened.

One of the most interesting points of this is that I went in with an idea on the right way to do this, and over the course of discussion and review of the document, changed my mind.

Change Versioning Scheme

Status

Accepted

Context

Currently, the UI uses a SemVer style version number. However, we have no convention for what kind of modifications constitute a major, minor, or patch change. We also have no processes or people who care specifically about what kind of change it is, just that a new version was deployed.

The other problem with using SemVer is that people wait until a branch has been approved, and then make an additional commit with the version number change (as another prod deployment might have happened in the meantime), meaning they need to wait for an additional build before they can deploy.

Not to mention, it’s possible to accidentally go backwards in numbers if a value was misread or if someone forgets to update the version number in their branch.

Considered Options

1. Auto-incrementing integer version

On production deployment, we would write a version number to the application. The negative of this approach is not having a version number in pre-production environments, such as test environments.

We could generate the number on the build phase (when the container is created), but this means that we might not release versions “in order”, as the order of what feature is deployed to production is not guaranteed, although the need to merge master into your branch would mean a rebuild, so a new version could be generated.

This method would also mean gaps in version numbers, as not all builds hit production, which might be a touch confusing.

Another issue with this method is that we build multiple containers from the same commit in separate pipelines, so we would need some way to generate a version in both pipelines which would match, which would mean either a function deriving from the commit hash or a service which would calculate and cache version numbers so they could be generated and looked up by multiple pipelines.

Example Version:

1870

2. Git (short) sha of the commit

On build, write the short (7 char) SHA as the version number. The negative of this approach is not having an easy to understand order of version numbers. However, this scheme means we can easily see exactly which commit is currently running in production (or any environment, for that matter.)

Example Version:

84d33bb

3. Build ID from CI System

On build, embed the buildID as the version number. The pipeline id is a 24 character string consisting of numbers and letters, so this is functionally similar to Option 2, but with a longer number that doesn’t tie back to a commit.

As with Option 1, we would need to decide if this number comes from the build pipeline, or from the deployment pipeline. This also has the same multi-pipeline problem too.

Example Version:

611a0be261ddea19dab67c22

4. Datestamp

On build, use the current commit’s datestamp as the tag.

As long as we keep the resolution of the datestamp large enough, the multiple pipelines needing to generate the same ID shouldn’t be a problem. I guess 1-minute resolution would be enough, although if a rebuild is needed (e.g. flakey internet connection), we would end up with a different datestamp.

Example Version:

2021-08-16.13-07

5. Commit Datestamp

Similar to Option 4, except we use the commit’s commit date to build the version number. This solves multiple pipelines needing to generate the same tag in parallel, as well as being unique and ordered. The timestamps can also be higher precision than Option 4, as we don’t need to hope that pipelines start at a close enough time.

This is how we would generate it:

timestamp=$(git show -s --format=%cd --date="format:%Y-%m-%d.%H-%M-%S")

Example Version:

2021-08-16.13-07-34

6. Automatic SemVer

On build, calculate the version number using Semantic-Release.

This method means that we would need to start enforcing commit message styles, and I am not sure the format that Semantic Release is ideal for us, so it might be better to cover the commit message formatting outside this process.

The commit format would be as follows:

<type>(<scope>): <short summary>
│       │             │
│       │             └─⫸ Summary in the present tense. Not capitalized. No period at the end.
│       │
│       └─⫸ Commit Scope: animations|bazel|benchpress|common|compiler|compiler-cli|core|
│                          elements|forms|http|language-service|localize|platform-browser|
│                          platform-browser-dynamic|platform-server|router|service-worker|
│                          upgrade|zone.js|packaging|changelog|dev-infra|docs-infra|migrations|
│                          ngcc|ve
│
└─⫸ Commit Type: build|ci|docs|feat|fix|perf|refactor|test

Having worked in repositories with this enforced, I would recommend against it, as it causes a lot of frustration (“omg why has my commit been rejected again?!”) and as mentioned in other options, I am not sure semver itself makes sense for our UI (or UI projects in general.)

We will still need developers to decide if a given commit is a major/minor/patch.

Example Version:

13.4.17

6. Combination: Datestamp + Git

On build, use a combination of Option 5 and Option 2 to generate a unique build number.

This method had the advantage of the meaning of the date, with the uniqueness of the git commit, but the likelihood of us needing to distinguish two commits made at identical times by their commit sha is unlikely, especially as we require clean merges to master.

Example Version:

2021-08-16.13-07-34.84d33bb

Chosen Decision

Option 5

We will also embed other build information as labels in the docker container, such as:

• branch name
• pipeline/build number
• git hash
• git commit timestamp

Consequences

• No need to tag commits as a released version, but we could automate this if we wanted
• No need to rebuild for changing the version number
• No need to remember to change the version number
• No need to decide on major/minor/patch semantics
• Gain an understandable version number, with meaning

Summary

As I said earlier, I went into this process (which I drove) wanting to pick the 2nd option - Short Git Sha, and I came away agreeing that the commit datestamp was the best thing to use.

Not only was my mind changed in the course of this, but also people who join the project later can check out the ./docs/adr/ and see what options we considered for everything about this project, and how we arrived at the conclusions. It also means I have examples to refer back to when people ask interesting questions at work.

How do you tag your containers?

os .cpus() returns the number of cores on a Kubernetes host, not the number of cores assigned to a pod.

Investigating excessive memory usage

Recently, when I was looking through a cluster health dashboard for a Kubernetes cluster, I noticed that one of the applications deployed was using a considerable amount of RAM - way more than I thought could be reasonable. Each instance (pod) of the application used approximately 8 GB of RAM, which was definitely excessive for a reasonably simple NodeJS webserver. Combined with the application running 20-30 replicas or so, it makes the total RAM usage between 160 GB and 240 GB.

One of the first things I noticed was that the deployment manifest in Kubernetes had the NODE_MAX_MEM environment variable specified and set to 250 MB:

environment:
  NODE_MAX_MEM: 250

Interesting. So how is a single container using more RAM than that?

The application used to be deployed to EC2 machines and to fully utilise the multiple cores in the machines, the cluster library was used.

This library essentially forks the node process into n child processes, and in this case, n was set to os.cpus(), which returns the number of cores available on the machine in NodeJS.

While this works for direct virtual machine usage, when the application was containerised and deployed to Kubernetes, it used about the same amount of ram as before, so no one realised there was a problem.

os.cpus() and Kubernetes

The interesting thing about os.cpus() when called in a container in Kubernetes is that it reports the number of cores available on the host machine, not the amount of CPU assigned to the container (e.g. through resource requests and limits).

So every replica for the application spawns 32 child processes, as our EC2 hosts have that many cores. As they had a limited per-pod CPU budget, was there any benefit to doing this?

So I did what seemed natural - I replaced os.cpus() with 1, and deployed the application to production, and watched the performance metrics to see what happened.

And what do you know? No difference in request performance at all - and the memory usage dropped by 7.75 GB per pod.

This means overall, we have saved 155 GB to 232.5 GB of RAM, with no performance difference!

While Vault has a lot of data available in Prometheus telemetry, the kind of information I am after is best taken from the Audit backend. Setting up an audit backend for Vault is reasonably easy - it supports three methods of communication: file, socket and syslog. For this application, I use a Unix socket and a small daemon running on the same machine as the Vault instance to send the data to a tracing system.

The Goal

Write a small application that receives audit events and writes traces (spans) to an observability tool. In this case, I am implementing both Honeycomb and Zipkin via OpenTelemetry.

The code is available on Github, and the most interesting parts are covered in the rest of this blog post.

Receiving and Processing Messages

ln, _ := net.Listen("unix", "/tmp/observe.sock")
conn, _ := ln.Accept()

for {
  message, _ := bufio.NewReader(conn).ReadBytes('\n')

  // do something with the message
}

We only need to do minimal processing of the data for this application before sending it on to Honeycomb or Zipkin. As the messages contain nested objects, we need to flatten the object hierarchy for easier viewing in spans. So instead of this:

{
  "request": {
    "operation": "update",
    "namespace": { "id": "root" },
    "path": "sys/audit/socket",
    "data": {
      "local": false
    }
  }
}

We want to send this:

{
  "request.operation": "update",
  "request.namespace.id": "root",
  "request.path": "sys/audit/socket",
  "request.data.local": false
}

We also want to get a few strongly typed pieces of data out of the message, too, such as the type (request or response) and the request’s id, which is in both messages and can be used to group the spans.

To save us from deserialising the json twice, we can do the following:

1. deserialize into a map[string]interface{}
2. create a flattened version of the event using the flatten library
3. turn the map into a typed struct using the mapstructure library

// 1 deserialize
event := map[string]interface{}{}
if err := json.Unmarshal(message, &event); err != nil {
  return err
}

// 2 flatten
flat, err := flatten.Flatten(event, "", flatten.DotStyle)
if err != nil {
  return err
}

// 3 type
typed := Event{}
if err := mapstructure.Decode(event, &typed); err != nil {
  return err
}