Custom JSON serialisation in golang to fix a Kong bug

I wrote an open source terraform provider for kong towards the end of last year.  The provider gave me a chance to write some go code and also the chance to do an open source project.  It seems to be pretty popular which is cool as it is nice to have written something that people are finding useful.

Recently someone logged a really detailed (thanks) github issue against a bug they found with the terraform provider.  It turned out the cause of this bug was the way that Kong was returning JSON objects when one of the fields was an empty list.  If a list property on the kong API object is set to [] then instead of returning [] Kong returns {}.

I wanted to fix this in a generic way for all list fields on the API object and also fix it for every operation on API (GET, POST, PUT etc).  Ie the ideal fix would see the code only being needed in a single place.  So to fix this I first wrote a test (test first of course).  As an aside the nice thing about the gokong project (and the kong terraform provider) are tested using docker containers so they are actually against the real Kong (no mocks anywhere to be seen!!!).

I quickly realised that I could write my own deserialise JSON method by simply implementing the UnmarshalJSON interface.  So my first stab at this was:

The idea behind this code was to simply replace the erroneous {} with [] in the JSON and then call the normal json.Unmarshal func.  Unfortunately this code goes into an infinite loop as you keep calling yourself.  Which was a shame as this code was nice and concise and would’ve done everything I wanted as it delegated the hard work of actually doing the deserialisation over to the normal json.Unmarshal func.

Then I found out that I can do what I want to do if I use a type alias.  Here is the corrected code:

The trick is to use a type alias to alias my Api type over to Alias (note this is a private declaration so is only scoped to the method).  Then I use an anonymous struct that uses an embedded type to inherit all of the fields from Api.  I then call json.Unmarshal on my anonymous struct with the fixed JSON.  Because my type Alias has a single embedded type which is Api it will serialise all of the Api fields to JSON using the fixed JSON.

I think this is a pretty neat solution as it means that anywhere in my whole library that deserialises the Api type will automatically use this code just by virtue of the fact that it implements the UnmarshalJSON interface.

This is a handy thing to know for your toolbox when you need to get involved in the JSON deserialisation pipeline in go.  Happy coding!

Terraform provider kong – fully tested using docker containers

A couple of posts ago I talked about how you could achieve full stack testing in Go using Docker containers. I have just finished the first version of terraform provider kong, a terraform provider for Kong that is built on top of gokong.

At the moment at Form3 we configure Kong using a custom Ruby script that runs on our Kong container and configures Kong upon boot up. Whilst this works there are a number of problems with this approach:

  • It is hard to know what the end state of Kong is as you end up with a whole bunch of Ruby code that is hard to follow
  • If you want to migrate the container over time it can be quite tricky as its hard to clean up APIs, consumers and other Kong resources as you delete them (the database Kong uses is long lasting)

By writing a custom terraform provider for Kong all of these problems go away.  As terraform uses a declarative language you simply declare the state that you want the thing you are configuring to be in (in this case Kong) and terraform looks at the current state and automatically works out how to get there.  This makes everything so much simpler.  A good example is if you remove an API in our current world you have to remember to write a line of Ruby to delete the old API.  When using terraform you simply remove the API from your configuration and terraform deletes it for you.

Building upon the way that I wrote gokong I wrote terraform provider kong in the same vein, ie full stack testing using real components and no mocks! This means that the acceptance tests in terraform provider kong spin up a real Kong container and apply terraform plans to it, check it is in the expected state and then destroy it again. Testing in this way gives you ultimate confidence that the code works!

To automate releases of new versions of terraform provider kong I have used the excellent goreleaser. Goreleaser automates building your code for multiple platforms, zipping/tar balling it up and uploading it to github. It is so easy to use, kudos to the authors. To setup you simply need to create a goreleaser.yml file:

My file specifies the binary and builds for mac (Darwin), Linux and Windows.  The last part is to define a task to run goreleaser as part of your build.  Goreleaser will only run when the build is running as a tag build, therefore you can create a task and run it on every build and goreleaser will only build a release for a tagged build, pretty smart.  My build task to create a release looks like:

I run the release task as part of every travis build.  You also need to create a github token which is used by goreleaser to upload the release to your github account.  To generate a github token go to https://github.com/settings/tokens/new.  Then simply set it as a secure environment variable in your .travis.yml and goreleaser will pick it up for you.

Then when you want to create a release you can simply do:

Which will build a new v0.1.0 release and automatically create the binaries and upload the release to Github.  Pretty smart!

The last thing I wanted to go through was that I’m taking advantage of the build matrix feature of travis to run the tests against multiple versions of kong. The following section of my travis.yml file:

Travis will automatically run a build for each line under the matrix definition.  Each with the values on the line.  I pass the KONG_VERSION parameter all of the way through to the code that pulls the Kong container for Docker so it will pull the version that is specified here.  Hence I can easily update this build to run against many versions of Kong.  As new versions are released I can simply add a line to the build matrix and it will automatically be tested on travis.  This is a really powerful feature, being able to run your tests against multiple versions of a target piece of software is immense!

If you are interested terraform provider kong or gokong I would love to hear your feedback and get your ideas and contributions.