Custom JSON serialisation in golang to fix a Kong bug

I wrote an open source terraform provider for kong towards the end of last year.  The provider gave me a chance to write some go code and also the chance to do an open source project.  It seems to be pretty popular which is cool as it is nice to have written something that people are finding useful.

Recently someone logged a really detailed (thanks) github issue against a bug they found with the terraform provider.  It turned out the cause of this bug was the way that Kong was returning JSON objects when one of the fields was an empty list.  If a list property on the kong API object is set to [] then instead of returning [] Kong returns {}.

I wanted to fix this in a generic way for all list fields on the API object and also fix it for every operation on API (GET, POST, PUT etc).  Ie the ideal fix would see the code only being needed in a single place.  So to fix this I first wrote a test (test first of course).  As an aside the nice thing about the gokong project (and the kong terraform provider) are tested using docker containers so they are actually against the real Kong (no mocks anywhere to be seen!!!).

I quickly realised that I could write my own deserialise JSON method by simply implementing the UnmarshalJSON interface.  So my first stab at this was:

The idea behind this code was to simply replace the erroneous {} with [] in the JSON and then call the normal json.Unmarshal func.  Unfortunately this code goes into an infinite loop as you keep calling yourself.  Which was a shame as this code was nice and concise and would’ve done everything I wanted as it delegated the hard work of actually doing the deserialisation over to the normal json.Unmarshal func.

Then I found out that I can do what I want to do if I use a type alias.  Here is the corrected code:

The trick is to use a type alias to alias my Api type over to Alias (note this is a private declaration so is only scoped to the method).  Then I use an anonymous struct that uses an embedded type to inherit all of the fields from Api.  I then call json.Unmarshal on my anonymous struct with the fixed JSON.  Because my type Alias has a single embedded type which is Api it will serialise all of the Api fields to JSON using the fixed JSON.

I think this is a pretty neat solution as it means that anywhere in my whole library that deserialises the Api type will automatically use this code just by virtue of the fact that it implements the UnmarshalJSON interface.

This is a handy thing to know for your toolbox when you need to get involved in the JSON deserialisation pipeline in go.  Happy coding!

Full stack testing in golang with docker containers

I like to practice the approach of full stack component testing where the guiding principle is that you test the entire component from as high level as possible and only stub out third party dependencies (read other APIs) or something that isn’t easily available as a docker container.  I have recently started a golang project to write a client for kong and I thought this would be a good opportunity to use this testing strategy in a golang project.

I love Go but the one thing I don’t like so much about it is the approach that most people seem to be taking to testing. A lot of tests are at the method level where your tests end up being tightly coupled to your implementation. This is a bad thing. You know a test is not very good when if you have to change your test when you change your implementation. The reason this is bad is because firstly as you are changing your test the same time as you are changing your code once you have finished you have know way of knowing if the new implementation still works as the test has changed. It also restricts how much you can get in and edit the implementation as you have constantly having to update the way the tests mock out everything. By testing at the top component level the tests do not care about the implementation and the code runs with real components so it works how it will in the real world. By writing tests in this way I have seen a lot less defects and have never had to manually debug something.

Anyway back to the subject of the full stack testing approach in Go. To start I used the excellent dockertest project which gives you a great API to start and stop docker containers. I then took advantage of the fact that in a Go project there is a special test function that gets called for every test run:

In the above method you can do your test setup code where I have placed the //setup comment and your teardown code where I have placed the //teardown comment.  The code that gets returned by m.Run() is the exit code from the test run.  Go sets this to non zero if the test run fails so you need to exit with this code so your build will fail if your test run fails.  Now using this method I can start the kong docker container, run the tests and then stop the kong docker container.  Here is the full TestMain code at time of writing:y

I have wrapped the starting and stopping of the kong container in a method to abstract away the detail.  Notice how the StartKong method takes the Kong version as a parameter.  It gets the Kong version either from the environment variable KONG_VERSION or if that environment variable is not set then it uses the default Kong version which I set to the latest version 0.11 at time of writing.  The cool thing about this is that if I want to run my tests against a different version of Kong I can do that easily by changing this value.  The really cool thing about this is that I can run the build against multiple versions of Kong on travis-ci by taking advantage of the env matrix feature.  If you list multiple values for an environment variable in travis-ci then travis-ci will automatically run a build for each entry.  This means it is really easy to run the whole test pack against multiple versions of Kong which is pretty neat.  You can check out the gokong build to see this in action!

The one part you may be wondering from all of this is how do I get the url of the container that Kong is running on for use in my tests.  That is done by setting an environment variable KONG_ADMIN_ADDR.  The client uses that environment variable if set and if not then it defaults to localhost:8001.

With all of this in place it allows me to test the client by hitting a real running Kong in a container, no mocks in sight!  How cool is that.  Plus I can run against any version of Kong that is built as a docker container with a flick of a switch!

Here is an example of what a test looks like so you can get a feel:

I think that is really clean and readable.  All of the code that boots up and tears down Kong is out of sight and you can just concentrate on the test.  Again with no mocks around 🙂

If you want to see the rest of the code or help contribute to my gokong project that would be great.  I look forward to any feedback you have on this.