Running Linkerd in a docker container on AWS ECS

I recently solved an interesting problem of configuring linkerd to run on an AWS ECS cluster.

Before I explain the linkerd configuration I think it would help to go through a diagram showing our setup:

A request comes in the top it then gets routed to a Kong instance.  Kong is configured to route the request to a local linkerd instance.  The local linkerd instance then uses its local Consul to find out where the service is.  It Then rewrites the request to call another linkerd on the destination server where the service resides (the one that was discovered in consul).  The linkerd on the service box then receives the request and uses its local consul to find the service.  At this point we use a filter to make sure it only uses the service located on the same box as essentially the service discovery has already happened by the calling linkerd.  We then call the service and reply.

The problem to solve when running on AWS ECS is how to bind to only services on your local box.  The normal way of doing this is to use the interpreter “io.l5d.localhost” (localhost).  Which will then filter services in consul that are on the local host.  When running in a docker container this won’t work as the local ip address of the linkerd in the docker container will not be the ip address of the server it is running on.  Meaning when it queries consul it will have no matches.

To solve this problem we can use the specificHost filter (added recently). We can use this to provide the IP address of the server to filter on. Now we run into another problem where we do not know the ip address of the server until runtime. There is a neat solution to this problem. Firstly we can write our own docker container based off the official one. Next we define a templated config file like this:

interpreter:
    kind: default
    transformers:
    - kind: io.l5d.specificHost
      host: $LOCAL_IP

Notice that I have used $LOCAL_IP instead of the actual ip. This is because at runtime we can write a simple script that will set the $LOCAL_IP environment variable to the IP of the box the container is running on and then substitute all environment variables in the config and then run linkerd.

To do this we use the following code inside an entrypoint.sh file:

export LOCAL_IP=$(curl -s 169.254.169.254/latest/meta-data/local-ipv4)
envsubst < /opt/linkerd/conf/linkerd.conf.template > /opt/linkerd/conf/linkerd.conf
echo "starting linkerd on $LOCAL_IP..."
./bundle-exec /opt/linkerd/conf/linkerd.conf

The trick here is to use the AWS meta endpoint to grab the local ip of the machine. We then use the envsubst program to substitute out all environment variables in our config and build a real linkerd.conf file. From there we can simply start linkerd.

To make this work you need a Dockerfile like the following:

FROM buoyantio/linkerd:1.1.0

RUN wget -O /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64
RUN chmod +x /usr/bin/dumb-init

RUN apt-get update && apt-get install gettext -y

ADD entrypoint.sh /entrypoint.sh

ENTRYPOINT ["/usr/bin/dumb-init", "--"]

This dockerfile uses dumb-init to manage the linkerd process. It installs the gettext package which is where the envsubst program lives. It then kicks off the entrypoint.sh script which is where we do our environment substitution and start linkerd.

The nice thing about this is we can put any environment variables we want into our linkerd config template file and then supply them at runtime in our task definition file. They will then get substituted out when the container starts. For example I am supplying the common name to use for ssl this way in my container.

Although this solution works it would be great if linkerd supported having environment variables inside its config file and swapped them out automatically. Maybe a pull request for another time…

SqlJuxt – Active patterns for the win

F# Active Patterns are awesome!! I wanted to start this blog post with that statement. I was not truly aware of F# active patterns until I read the article on F Sharp for Fun and Profit when I was looking at a better way to use the .Net Int32.Parse function.

Active patterns are a function that you can define that can match on an expression. For example:

let (|Integer|_|) (s:string) = 
    match Int32.TryParse(s) with
        | (true, i) -> Some i
        | _ -> None 

To explain what each line is doing the | characters are called banana clips (not sure why) and here we are defining a partial active pattern. This means the pattern has to return Option of some type. So be definition the pattern may return a value or it may not so it is a partial match. This pattern takes a string and then returns Some int if the pattern matches or None if it does not. Once this pattern is defined it can be used in a normal match expression as follows:

let printNumber (str:string) =
	match str with
		| Integer i -> printfn "Integer: %d" i
		| _ -> "%s is not a number" %s

Using this technique it allows us to define a really cool active pattern for matching regular expressions and parsing out the groups that are matched:

let (|ParseRegex|_|) regex str =
    let m = Regex(regex).Match(str)
    match m with 
        | m when m.Success -> Some (List.tail [ for x in m.Groups -> x.Value] )
        | _ -> None

This regex active pattern returns all of the matched groups if the regex is a match or None if it is not a match. Note the reason List.tail is used is to skip the first element as that is the fully matched string which we don’t want, we only want the groups.

The reason why all of this came up is that the getNextAvailableName function that I wrote about in my last blog post is very long winded. For those who haven’t read my last post this function generates a unique name by taking a candidate name and a list of names that have been taken. If the name passed in has been taken then the function will add a number on to the end of the name and keep incrementing it until it finds a name that is not taken. The getNextAvailableName function was defined as:

let rec getNextAvailableName (name:string) (names: string list) =

    let getNumber (chr:char) =
        match Int32.TryParse(chr.ToString()) with
            | (true, i) -> Some i
            | _ -> None

    let grabLastChar (str:string) =
        str.[str.Length-1]

    let pruneLastChar (str:string) =
        str.Substring(0, str.Length - 1)

    let pruneNumber (str:string) i =
        str.Substring(0, str.Length - i.ToString().Length)

    let getNumberFromEndOfString (s:string)  =

        let rec getNumberFromEndOfStringInner (s1:string) (n: int option) =
            match s1 |> String.IsNullOrWhiteSpace with
                | true -> n
                | false -> match s1 |> grabLastChar |> getNumber with
                            | None -> n
                            | Some m ->  let newS = s1 |> pruneLastChar
                                         match n with 
                                            | Some n1 -> let newN = m.ToString() + n1.ToString() |> Convert.ToInt32 |> Some
                                                         getNumberFromEndOfStringInner newS newN
                                            | None -> getNumberFromEndOfStringInner newS (Some m) 
        let num = getNumberFromEndOfStringInner s None
        match num with
            | Some num' -> (s |> pruneNumber <| num', num)
            | None -> (s, num)
        

    let result = names |> List.tryFind(fun x -> x = name)
    match result with
        | Some r -> let (n, r) = getNumberFromEndOfString name
                    match r with 
                        | Some r' -> getNextAvailableName (n + (r'+1).ToString()) names
                        | None -> getNextAvailableName (n + "2") names
                    
        | None -> name

With the Integer and Regex active patterns defined as explained above the new version of the getNextAvailableName function is:

let rec getNextAvailableName (name:string) (names: string list) =

    let result = names |> List.tryFind(fun x -> x = name)
    match result with
        | Some r -> let (n, r) = match name with
                                    | ParseRegex "(.*)(\d+)$" [s; Integer i] -> (s, Some i)
                                    | _ -> (name, None)
                    match r with 
                        | Some r' -> getNextAvailableName (n + (r'+1).ToString()) names
                        | None -> getNextAvailableName (n + "2") names
                    
        | None -> name

I think it is pretty incredible how much simpler this version of the function is. It does exactly the same job (all of my tests still pass:) ). It really shows the power of active patterns and how they simplify your code. I think they also make the code more readable as even if you didn’t know the definition of the ParseRegex active pattern you could guess from the code.

Check out the full source code at SqlJuxt GitHub repository.

XBehave – compiling the test model using the Razor Engine

In the last post I left off describing how I implemented the parsing of the XUnit console runner xml in the XUnit.Reporter console app. In this post I want to talk through how you can take advantage of the excellent RazorEngine to render the model to html.

I am going to talk through the console app line by line. The first line of the app:

var reporterArgs = Args.Parse<ReporterArgs>(args);

Here we are using the excellent PowerArgs library to parse the command line arguments out into a model. I love how the API for PowerArgs has been designed. It has a slew of features which I won’t go into here for example it supports prompting for missing arguments all out of the box.

Engine.Razor.Compile(AssemblyResource.InThisAssembly("TestView.cshtml").GetText(), "testTemplate", typeof(TestPageModel));

This line uses the RazorEngine to compile the view containing my html, it gives the view the key name “testTemplate” in the RazorEngine’s internal cache. What is neat about this is that we can deploy the TestView.cshtml as an embedded resource so it becomes part of our assembly. We can then use the AssemblyResource class to grab the text from the embedded resource to pass to the razor engine.

var model = TestPageModelBuilder.Create()
                                .WithPageTitle(reporterArgs.PageTitle)
                                .WithTestXmlFromPath(reporterArgs.Xml)
                                .Build();

We then create the TestPageModel using the TestPageModelBuilder. Using a builder here gives you very readable code. Inside the builder we are using the XmlParser from earlier to parse the xml and generate the List of TestAssemblyModels. We also take the optional PageTitle argument here to place in the page title in our view template.

var output = new StringWriter();
Engine.Razor.Run("testTemplate", output, typeof(TestPageModel), model);
File.WriteAllText(reporterArgs.Html, output.ToString());

The last 3 lines simply create a StringWriter which is used by the engine to write the compiled template to. Calling Engine.Razor.Run runs the template we compiled earlier using the key we set “testTemplate”. After this line fires our html will have been written to the StringWriter so all we have to do is extract it and then write it out to the html file path that was parsed in.

That’s all there is to it. We now have a neat way to extract the Given, When, Then gherkin syntax from our XBehave texts and export it to html in whatever shape we chose. From there you could post to an internal wiki or email the file to someone, that could all be done automatically as part of a CI build.

If anyone has any feedback on any of the code then that is always welcomed. Please check out the XUnit.Reporter repository for all of the source code.

XBehave – Exporting your Given, Then, When to html

For a project in my day job we have been using the excellent XBehave for our integration tests. I love XBehave in that it lets you use a Given, When, Then syntax over the top of XUnit. There are a couple of issues with XBehave that I have yet to find a neat solution for (unless I am missing something please tell me if this is the case). The issues are

1) There is not a nice way to extract the Given, When, Then gherkin syntax out of the C# assembly for reporting
2) The runner treats each step in the scenario as a separate test

To solve these to problems I am writing an open source parser that takes the xml produced by the XUnit console runner and parses it to a C# model. I can then use razor to render these models as html and spit out the resultant html.

This will mean that I can take the html and post it up to a wiki so every time a new build runs the tests it would be able to update the wiki with the latest set of tests that are in the code and even say whether the tests pass or fail. Allowing a business/product owner to review the tests, see which pieces of functionality are covered and which features have been completed.

To this end I have created the XUnit.Reporter github repository. This article will cover the parsing of the xml into a C# model.

A neat class that I am using inside the XUnit.Reporter is the AssemblyResource class. This class allows easy access to embedded assembly resources. Which means that I can run the XUnit console runner for a test, take the resultant output and add it to the test assembly as an embedded resource. I can then use the AssemblyResource class to load back the text from the xml file by using the following line of code:

AssemblyResource.InAssembly(typeof(ParserScenarios).Assembly, "singlepassingscenario.xml").GetText())

To produce the test xml files for the tests I simply set up a console app, added XBehave and then created a test in the state I wanted for example a single scenario that passes. I then ran the XUnit console runner with the -xml flag set to produce the xml output. I then copied the xml output to a test file and named it accordingly.

The statistics for the assembly model and test collection model are not aligned to what I think you would want from an XBehave test. For example if you have this single XBehave test:


public class MyTest
{
[Scenario]
public void MyScenario()
{
"Given something"
._(() => { });

"When something"
._(() => { });

"Then something should be true"
._(() => { });

"And then another thing"
._(() => { });
}
}

Then the resultant xml produced by the console runner is:

<?xml version="1.0" encoding="utf-8"?>
<assemblies>
<assembly name="C:\projects\CommandScratchpad\CommandScratchpad\bin\Debug\CommandScratchpad.EXE" environment="64-bit .NET 4.0.30319.42000 [collection-per-class, parallel (2 threads)]" test-framework="xUnit.net 2.1.0.3179" run-date="2017-01-27" run-time="17:16:00" config-file="C:\projects\CommandScratchpad\CommandScratchpad\bin\Debug\CommandScratchpad.exe.config" total="4" passed="4" failed="0" skipped="0" time="0.161" errors="0">
<errors />
<collection total="4" passed="4" failed="0" skipped="0" name="Test collection for RandomNamespace.MyTest" time="0.010">
<test name="RandomNamespace.MyTest.MyScenario() [01] Given something" type="RandomNamespace.MyTest" method="MyScenario" time="0.0023842" result="Pass" />
<test name="RandomNamespace.MyTest.MyScenario() [02] When something" type="RandomNamespace.MyTest" method="MyScenario" time="0.0000648" result="Pass" />
<test name="RandomNamespace.MyTest.MyScenario() [03] Then something should be true" type="RandomNamespace.MyTest" method="MyScenario" time="0.0000365" result="Pass" />
<test name="RandomNamespace.MyTest.MyScenario() [04] And then another thing" type="RandomNamespace.MyTest" method="MyScenario" time="0.000032" result="Pass" />
</collection>
</assembly>
</assemblies>

If you look carefully at the xml you will notice a number of things which are counter-intuative. Firstly look at the total in the assembly element it says 4, when we only had a single test. This is because the runner is considering each step to be a separate test. The same goes for the other totals and the totals in the collection element. The next thing that you will notice is that the step names in the original test have had a load of junk added to the front of them.

In the parser I produce a model with the results that I would expect. So for the above xml it I produce an assembly model with a total of 1 test, 1 passed, 0 failed, 0 skipped and 0 errors. Which I think makes much more sense for XBehave tests.

Feel free to clone the repository and look through the parsing code (warning it is a big ugly). Next time I will be talking through the remainder of this app which is rendering the test results to html using razor.

SqlJuxt – Implementing the builder pattern in F#

As an imperative programmer by trade a pattern that I often like to use is the builder pattern for building objects. This pattern is especially useful for test code.

The reasons the builder pattern is so useful is because:

  • It means you only have to “new” the object up in a single place meaning your test code effectively goes through an API (the builder API) to create the object
  • It makes you code really readable so you can understand exactly what it is doing without having to look into how the code works

In my SqlJuxt project I started by coding it in C#. I wanted a builder to make a table create script. This would mean that in my integration tests I could fluently create a script that would create a table and then I could run it in to a real database and then run the comparison. This makes the tests very readable and easy to write. In C# using the table builder looks like:

    var table = Sql.BuildScript()
                   .WithTableNamed("MyTable", t => t.WithColumns(c => c.NullableVarchar("First", 23)
                                                                       .NullableInt("Second")))

I think that is pretty nice code. You can easily read that code and tell that it will create a table named “MyTable” with a nullable varchar column and a nullable int column.

I wanted to achieve the same thing in F# but the catch is I did not just want to translate the C# in to F# (which is possible) I wanted to write proper functional code. Which means you should not really create classes or mutable types! The whole way the builder pattern works is you store state on the builder and then with each method call you change the state of the builder and then return yourself. When the build method is called at the end you use all of the state to build the object (note an implicit call to the Build method is not needed in the above code as I have overridden the implicit conversion operator).

I hunted around for inspiration and found it in the form of how the TopShelf guys had written their fluent API.

This is how using the table builder looks in F#:

    let table = CreateTable "TestTable"
                    |> WithNullableInt "Column1"
                    |> Build 

I think that is pretty sweet! The trick to making this work is to have a type that represents a database table. Obviously the type is immutable. The CreateTable function takes a name and then returns a new instance of the table type with the name set:

    let CreateTable name =
        {name = name; columns = []}

Then each of the functions to create a column take the table and a name in the case of a nullable int column and then return a new immutable table instance with the column appended to the list of columns. The trick is to take the table type as the last parameter to the function. This means you do not have to implicitly pass it around. As you can see from the CreateTable code above. The Build function then takes the table and translates it in to a sql script ready to be run in to the database.

Here is an example of a complete test to create a table:

    [<Fact>]
    let ``should be able to build a table with a mixture of columns``() =
       CreateTable "MultiColumnTable"
           |> WithVarchar "MyVarchar" 10
           |> WithInt "MyInt"
           |> WithNullableVarchar "NullVarchar" 55
           |> WithNullableInt "NullInt"
           |> Build 
           |> should equal @"CREATE TABLE [dbo].[MultiColumnTable]( [MyVarchar] [varchar](10) NOT NULL, [MyInt] [int] NOT NULL, [NullVarchar] [varchar](55) NULL, [NullInt] [int] NULL )
GO"

I think that test is really readable and explains exactly what it is doing. Which is exactly what a test should do.

If you want to follow along with the project then check out the SqlJuxt GitHub repository.

SqlJuxt – A database comparison tool written in F#

As part of a new project I have decided to write a Sql database comparison tool in F#. I wanted a project that I could learn F# with (having dabbled a bit in the past). I have just finished re-reading the excellent Thinking Functionally series on the F# for fun and profit site. Scott has done a fantastic job on there of explaining F# concepts from the perspective of an imperative programmer. Kudos to Scott for that.

So what is the database comparison tool going to be? Well it is initially going to be an API library written in F# that allows you to compare two Sql databases to find the differences. Then from there I might extend it to allow you to script the differences to make the databases the same and then possibly write a front end for it. The other idea I have in mind is to write a C# shim to allow you to use the library nicely from F#. Although the shim library is not strictly necessary using F# from C# or vice versa can be a bit cumbersome unless you put a bit of effort in to making it work well.

That’s it for now stay tuned as I blog about my foray into the world of functional programming. If you want to keep up with how the project is going then you can check out the Sql Juxt GitHub repository.

Converting decimal numbers to Roman Numerals in C#

I decided to create a little project to implement converting decimal numbers to Roman Numerals in C#. You can solve this problem in quite a few different ways, I wanted to talk through the pattern based approach that I went for.

The Roman Numerals from 0 to 9 are as follows:

  • 0 = “” (empty string)
  • 1 = I
  • 2 = II
  • 3 = III
  • 4 = IV
  • 5 = V
  • 6 = VI
  • 7 = VII
  • 8 = VIII
  • 9 = IX

To start the project I wrote a set of tests that checked these first 10 cases. I like taking this approach as it allows you to solve for the simple base case, then you can refactor your solution to be more generic. Our first implementation that solves for the first ten numbers is:

public static string ToRomanNumeral(this int integer)
{
       
     var mapping = new Dictionary<int, string>
     {
         {0, ""},
         {1, "I"},
         {2, "II"},
         {3, "III"},
         {4, "IV"},
         {5, "V"},
         {6, "VI"},
         {7, "VII"},
         {8, "VIII"},
         {9, "IX"},
     };

     return mapping[integer];
}

Obviously there is no error checking etc we are just solving for the 10 first cases. I decided to implement the method as an extension method on int as it makes the code look neat as it will allow you to write:

    var romanNumeral = 9.ToRomanNumeral();

The next step is to look at the Roman Numerals up to 100 and see if we can spot any patterns. We don’t want to have to manually type out a dictionary for every Roman Numeral! If we look at the Roman Numeral representation for the tens column from 0 to 100 we find they are:

  • 0 = “” (empty string)
  • 10 = X
  • 20 = XX
  • 30 = XXX
  • 40 = XL
  • 50 = L
  • 60 = LX
  • 70 = LXX
  • 80 = LXXX
  • 90 = XC

We can straight away see that it is exactly the same as the numbers from 0 to 9 except you replace I with X, V with L and X with C. So lets pull that pattern out into something that can create a mapping dictionary given the 3 symbols. Doing this gives you the following method:

private static Dictionary<int, string> CreateMapping(string baseSymbol, string midSymbol, string upperSymbol)
{
    return new Dictionary<int, string>
    {
        {0, ""},
        {1, baseSymbol},
        {2, baseSymbol + baseSymbol},
        {3, baseSymbol + baseSymbol + baseSymbol},
        {4, baseSymbol + midSymbol},
        {5, midSymbol},
        {6, midSymbol + baseSymbol},
        {7, midSymbol + baseSymbol + baseSymbol},
        {8, midSymbol + baseSymbol + baseSymbol + baseSymbol},
        {9, baseSymbol + upperSymbol},
    };
}

We can now call the above method with the symbols for the column we want to calculate passing in the correct symbols. So for the units column we would write:

    var unitMapping = CreateMapping("I", "V", "X");

Now we have this we it is straight forward to create the mapping for the hundreds column. To complete our implementation we want to add some error checks as we are only going to support Roman Numerals from 0 to 4000. The full solution is now quite trivial. We simply check the input is in our valid range (between 0 and 4000). Then we loop through each column looking up the symbol for that column in our mapping dictionary that we generate using our function above. Then we simply concatenate the symbols together using a StringBuilder and return the result.

The full solution with all of the tests is available on my GitHub repository: https://github.com/kevholditch/RomanNumerals.

Partially applying functions in F#

A cool feature of a functional language like F# is the ability to partially apply a function. Take this add function:

let add x y = x + y

By partially applying this function we can create new functions like this:

let plus1 = add 1
let plus5 = add 5

The functions we have created plus1 and plus5 both have the signature int -> int. We have partially applied add to a single parameter leaving us with a function that takes one more int and returns us an int. This is a really neat idea.

Building on this if we want to do the same thing with subtract we find that it does not quite work:

let subtract x y = x - y
let minus1 = subtract 1
printfn "%i" (minus1 7)

The code above prints -6 when we wanted our minus1 function to subtract 1 from the argument given. This is because unlike with add it matters the order that you give the arguments to subtract. We have another cool trick up our sleeves to solve this:

let subtract x y = x - y
let swap f x y = f y x
let minus = swap subtract
let minus1 = minus 1
printfn "%i" (minus1 7)

The minus1 function above now does what we would expect. To make this work we defined a swap function that swaps the order of the arguments. We can then pass our subtract function to our swap function to produce a new subtract function that takes its arguments in the opposite order. We can now partially apply the new function minus with 1 to give us a new function minus1 that works how we would expect. Having functions as a first class citizen really does lead to some neat code.

Simple Web API OWIN host using F#

I thought it would be good to write a simple OWIN Web Api host using F# to show how easy it is to interop between the two languages and as a fun experiment. You can find all of the source code from this blog post at the FSharpWebApi github repository.

If you open the Program.fs solution in the github repository you will see the following two lines of code are reponsible for starting the web server:

let hostAddress = "http://localhost:8000"
let server = WebApp.Start(hostAddress, getAppBuilder())

The getAppBuilder function is defined in the WebServerBuilder module as follows:

let getAppBuilder() =
    let config = new HttpConfiguration()
    config.Routes.MapHttpRoute("default", "{controller}") |> ignore
    fun (appBuilder:IAppBuilder) -> appBuilder.UseWebApi(config) |> ignore

The getAppBuilder function returns a function with the signature (IAppBuilder) -> (). This signature is the same as the one expected by the first parameter of WebApp.Start. The reason for breaking this function off into its own module is so that it can be tested.

The cool thing about the Web Api Owin Self host stack is that there is a nuget package that allows you to test your server without going over http. This is great because it saves you all of the hassle of hosting an endpoint for your tests and tearing it down again. The test framework preserves all of the semantics so the requests will go through the same pipeline it’s just at the final moment they won’t go over http. The nuget package for this is Microsoft.Owin.Testing.

To use the Microsoft.Owin.Testing nuget package you have to call the static method TestServer.Create(IAppBuilder -> ()). You can see that the test server create method takes the same function that is returned by our getAppBuilder function. Which is the reason for isolating this code. The getAppBuilder function contains the logic for how your server will behave (routes, media formatters etc) so by pulling that out into a function we can test it.

As I’ve mentioned before testing in F# is very terse and to the point. Here is the full test suite for my simple hello world server:

module WebServerTests =
    let createTestServer = fun () -> TestServer.Create(new Action<IAppBuilder>(getAppBuilder()))

    [<Test>]
    let ``get /Hello should return "get - hello world"``() =
        let testServer = createTestServer()
        let client = testServer.HttpClient

        client.GetAsync("/Hello").Result.Content.ReadAsStringAsync().Result 
            |> should equal "\"get - hello world\""   

    [<Test>]
    let ``post /Hello should return "post - hello world"``() =
        let testServer = createTestServer()
        let client = testServer.HttpClient

        client.PostAsync("/Hello", new StringContent("")).Result.Content.ReadAsStringAsync().Result 
            |> should equal "\"post - hello world\"" 

To test the web server you need to get the http client that hangs off of the test server (let client = testServer.HttpClient). This gives you a special http client that is wired up directly to your web server. We can now test that we get the correct responses when issuing a get or post request to /Hello.

It is a bit of a shame that there is not an implicit conversion from Action in C# to T -> () in F#. This is the reason that the create test server line at the top of the tests is ugly. We have to new up an action of IAppBuilder and pass our getAppBuilder function into the constructor to convert our F# “action” into a C# one.

The last piece of the puzzle to make the server work is the Hello controller which can be seen below:

type HelloController () =
    inherit ApiController()

    member this.Get () = 
            "get - hello world"

    member this.Post () =
            "post - hello world"

F# shines again. Look how easy and readable this controller class is. Even if you have never written a line of F# code I reckon you could have a pretty good guess at what this controller does. The member keyword is used to provide the semantics of exposing a public property or method. The reason we need the brackets is to cause it to become a method rather than a property.

Apart from the slight ugliness of having to convert from a F# function type to a C# Action type, it is much clener in my opinion to write the code in F#. There is so much less noise. Here is the equivilent HelloController in C#:

public class HelloController : ApiController
{
    public string Get()
    {
        return "get - hello world";
    }

    public string Post()
    {
        return "post - hello world";
    }
}

All of the brackets clutter up this simple class. The brackets do not add anything to the code they just help the compiler parse it.

As already mentioned all of the code is up on a github repository for you to view at your leisure. If you have any feedback, comments or questions I would be happy to hear from you.

Manses and the Stones Hacker Rank Problem in F#

The next problem from Hacker Rank that I’m tackling in F# is a problem called Mansasa and the stones (catchy name). The basics of the problem are that you are given 2 numbers and a length and you have to give all of the distinct sums of those numbers of the lists that can be made up of length l. An example is if stone 1 has the value 2 and stone 2 has the value 4 and the length is 3 we get:

0, 2, 2 = sum 4
0, 2, 4 = sum 6
0, 4, 2 = sum 6
0, 4, 4 = sum 8

Note the problem states that we should start with 0 for each entry. This means that the distinct sums in this example would be 4 6 8 (in order).

My idea to solve this problem was to write a function that takes a list and a length and then gives you all permutations for that list of that length. In the example above if you passed this function 2 and 4 and a length of 2 it would give the 4 possible ways of arranging 2 and 4 shown above. Once I have this function I can simply sum the numbers, do a distinct and then format the result. Simples, or so I thought more on that later.

Now that I’ve got a great way to write tests for my code I started with the unit tests for the function:

module combinations =

 
module combinations =
    []
    let ``combinations for [2;3] of length 2 should be [[2;2];[2;3];[3;2];[3;3]]``() =
        combinations [2;3] 2 |> should equal [[2;2];[2;3];[3;2];[3;3]]

    []
    let ``combinations for [2;3] of length 3 should be [[2;2;2];[2;2;3];[2;3;2];[2;3;3];[3;2;2];[3;2;3];[3;3;2];[3;3;3]]``() =
        combinations [2;3] 3 |> should equal [[2;2;2];[2;2;3];[2;3;2];[2;3;3];[3;2;2];[3;2;3];[3;3;2];[3;3;3]]

    []
    let ``combinations for [1;4] of length 1 should be [[1];[4]]``() =
        combinations [1;4] 1 |> should equal [[1];[4]]

I think it is so neat that the tests describe themselves. I was stuck on writing this function in a recursive way for a little while. I’m still finding it hard to think functionally. I put this problem to my friend Sam Owens who came up with a great solution:

 
let combination list length =
    let rec solve letters len =     
        seq {
            if len = 1 then 
                for l in letters do yield [l]
            else
                let theRest = solve letters (len-1) 
                for l in letters do
                    for r in theRest do 
                        yield l::r
        }
    solve list length

I thought this solution was ingenious. Before I talk through how it works let me show you the more functional version. I created this by translating Sam’s function above to use pattern matching:

let combinations list length =
    let rec solve lst len =
        match len with
        | 1 -> lst |> Seq.map(fun x -> [x])
        | i -> lst |> Seq.map(fun l -> solve lst (len-1) |> Seq.map(fun r -> l::r)) 
                            |> Seq.collect(fun x -> x) 
    solve list length

Both of these functions are exactly the same (kudos to Sam for coming up with this way of doing it). I think the 2nd function is much more elegantly written and more readable. It’s much terser and really shows the power of F#’s pattern matching.

This function works by the fact that it knows how to print out every combination of a list of length 1. Which is shown on the first line of the match statement. To do this it simply maps out all of the elements as a list of one item. If the function is called with a length greater than one then it calls itself with length – 1 and appends the results from that on to each element in the list passed in.

With this function written composing the function to solve the whole problem was pretty straight forward:

let findMansaNumbers steps (stone1:int) (stone2:int) =
    combinations [stone1; stone2] (steps-1) |> Seq.map(fun l -> Seq.sum(l)) 
                                            |> Seq.distinct 
                                            |> Seq.sort
                                            |> fun x -> String.Join(" ", x)

All we have to do is run the combinations function passing it a list of the two stone numbers and a length. We then sum the numbers from each combination, get a distinct list of the sums, sort them in to order and then print them out in a string where each number is separted by a space. The way that F# lends itself well to composing small building blocks together makes it a fun language to program in.

Here comes the kicker. Although this solution works, it passes the first three test cases on the hacker rank problem page it then times out. The reason for this is that the number of combinations the combinations function will return is given by the formula 2^length. This is exponential growth so for big lengths the time to compute all combinations is too long. As a short aside the fact that hackerrank run the code on their server and then time out if your solution takes too long is a good thing. It stops you from being able to brute force the problems which is one of the things that plagues project Euler in my opinion.

So it’s back to the drawing board with this problem. The code is on the repository under slow solution in the MansasaAndStones.fs file in the Hacker Rank in F# repository on github.

To solve this problem it is actually very straight forward. I was just taking completely the wrong approach. If you think of the case where you have 2 stones of values 2 and 4 and are asked to find 3 steps then you can think of this as:

0, 4, 4 = sum 8 = (0*2) + (2*4)
0, 2, 4 = sum 6 = (1*2) + (1*4)
0, 2, 2 = sum 4 = (2*2) + (0*4)

So the distinct sums are 4, 6 and 8. This is calculated by having two loops, one starting and 0 and incrementing, one starting at the number of steps – 1 (as we the first step is always 0 in the problem) and then multiplying one of the loops by the first stone and one by the second.

let findNumbers steps (stone1:int) (stone2:int) =
    let list1 = [0..steps-1]
    let list2 = [0..steps-1] |> List.rev
    List.zip list1 list2 |> Seq.map(fun (x, y) -> (x*stone1)+(y*stone2))
                         |> Seq.distinct
                         |> Seq.sort
                         |> fun x -> String.Join(" ", x)

The code above is the full solution to the problem. Instead of looping twice we use the zip function to produce tuples of the loop values we can then feed this into the map function which makes the code neater. After that we just have to make sure the list is distinct (as the problem states no duplicates) and then sort it (as the problem states that the values should be in order) then we have to produce a space separated string for each answer.

I really went around the houses with this problem but I did learn some cool things in F#. Solving the combinations problem was fun and helped me practice my skills of turning C# into functional F#. Once I realised I only needed two loops I was quite happy with how nice the solution is. One thing I’m learning more and more about functional programming is that you can write really expressive beautiful code. Erik Meijer would be proud!

To look at the final solution check out the ManasaAndStones module and the solution function from the github repository. As always I would be keen to hear your feedback.