Download github releases from private repos in bash and docker builds

I wanted to add a short post to describe how to automate the downloading of releases from private github repositories using a bash script or in a Docker build.

To start you need to create a Github token that has access to your repository. Once you have your token you can use the following bash script filling in the relevant details:

#!/usr/bin/env bash
set -e
wget -q –auth-no-challenge –header='Accept:application/octet-stream' \
https://$$REPO/releases/assets/`curl -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw" -s$REPO/releases | jq ". | map(select(.tag_name == \"$VERSION\"))[0].assets | map(select(.name == \"$FILE\"))[0].id"` \
-O /tmp/$FILE

view raw

hosted with ❤ by GitHub

This script will download your release to the /tmp/ directory, from there you can untar and move it etc.

To take this a stage further if you want to download your release as part of a docker build you can use the Dockerfile snippet below to give you a starting point:

ENV REPO "kevholditch/demo"
ENV FILE "demo_0.0.1_linux_amd64.tar.gz"
ENV VERSION "v0.0.1"
wget -q –auth-no-challenge –header='Accept:application/octet-stream' \
https://$$REPO/releases/assets/`curl -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw" -s$REPO/releases | jq ". | map(select(.tag_name == \"$VERSION\"))[0].assets | map(select(.name == \"$FILE\"))[0].id"` \
-O /tmp/$FILE

view raw


hosted with ❤ by GitHub

The trick here is that we are passing in the GITHUB_TOKEN using a docker build arg.  This allows you to build the container using travis by setting a secure ENV variable and then passing that into your docker build script as the docker arg parameter.  For example:

if [ "$GITHUB_TOKEN" = "" ]; then
echo "you need to create a github token with access to kevholditch/demo to run this build see"
exit -1
docker build –build-arg "GITHUB_TOKEN=$GITHUB_TOKEN" -t kevholditch/demo .

In the script above we check that the GITHUB_TOKEN env variable is set and if it isn’t then we terminate with a non zero exit code, halting the build.  This then allows developers to run the build with their own GITHUB_TOKEN and you can run this build on travis by setting a secure env variable (or the equivalent in the builder server you are using).


Handy powershell script for deploying nuget packages

I was recently presented with the problem to automate the deployment of a nuget package to a server. Now before you guys say I know that Octopus Deploy is the best way to go and I completely agree. At the moment at the organisation I’m at using a product like Octopus has to go through an “approval” process. Faced between the choice of having to carrying on deploying using file explorer (which is what the guys currently did here) and automating it, I voted for the latter option.

Now Team City is not really geared towards doing deployments. After all it’s designed to be a build server and that’s what it does extremely well. However we can make it work.

The first thing you need to do is package up your built web site into a nuget package and deploy it to a nuget server (either a local one or the public one). For those that dont know a nuget package is literally a zip file, you can even rename it .zip and it will extract. The reason for using a nuget package is that it gives you a nice way to place it onto a server, version it and in the future plug in Octopus Deploy.

So onto the script:

param([string]$computerName, [string]$destinationDir, [string]$packName, [string]$packVersion, [string]$executeLocally)

Write-Host "computer name: $computerName"
Write-Host "destination dir: $destinationDir"
Write-Host "package name: $packName"
Write-Host "package version: $packVersion"
Write-Host "execute locally: $executeLocally"

$deployScript = {
param($dest, $packageName, $version)

$basePath = "d:\packages\"

$packageNameAndVersion = "$packageName.$version"
$extractDir = $basePath + $packageNameAndVersion

Write-Host "deploying to: $dest"
Write-Host "extracting to: $extractDir"
Write-Host "package: $packageName"
Write-Host "version: $version"

$zipDownload = $packageNameAndVersion + ".zip"

$packageUrl = "$packageName/$version"

if (-Not (Test-Path $extractDir))
New-Item -ItemType directory -Path $extractDir
Get-ChildItem -Path ($extractDir + "\*") -Recurse | Remove-Item -Recurse -Force -Confirm:$false

$Path = $basePath + $zipDownload

Write-Host "Downloading package..."
$WebClient = New-Object System.Net.WebClient
$WebClient.DownloadFile($packageUrl, $path )

Write-Host "Extracting package..."

$shell = new-object -com shell.application
$zip = $shell.NameSpace($path)

foreach($item in $zip.items())

Write-Host "Cleaning destination..."

Get-ChildItem -Path $dest -Recurse | Remove-Item -Recurse -Force -Confirm:$false

Write-Host "Deploying..."

Copy-Item ($extractDir + "\*") $dest -Recurse

Write-Host "Removing downloaded package from $path"
Remove-Item $path -Recurse -Force -Confirm:$false

Write-Host "Removing extracted package from $extractDir"
Remove-Item $extractDir -Recurse -Force -Confirm:$false
Remove-Item ("$extractDir\") -Recurse -Force -Confirm:$false

Write-Host "Done."


$output = Invoke-Command -ScriptBlock $deployScript -ArgumentList $destinationDir, $packName, $packVersion

Now when you configure your team city build you need to pass in the following 3 parameters: $destinationDir, $packName and $packVersion. The destination dir is the directory you wish to deploy to e.g. \\\WebSites\MyWebSite. The package name is the package id of the nuget package and the packVersion is the version number of the nuget package. That’s it! Simply huh?

You will need to make sure that the Team City user account has write access to the folder you are deploying to.


In the screenshot above you can see how to setup the powershell script to run. You can either check it into your source control system and then enter the path to the file in the script file dialog above or alternatively you can copy and paste the source directly into Team City. I would recommend having the powershell script in source control as then if you have multiple deploy builds changing the script will mean every build will get the new powershell script.

In the script arguments dialog you need to pass in the parameters to the script described above.  You can either hard code these (not recommended) or you can use Team City parameters and then present the user with a dialog when they go to run the build.  To do this I use the following parameters:

%website-deployment-dir% %package-name-to-deploy% %package-version-to-deploy%

After you have saved this, Team City will prompt you to setup these parameters. Go to the parameters section of your build configuration. Click on one of the parameters. Click spec, then in the dialog that appears next to display select “prompt”. You can enter a default value if you wish or even give the user a dropdown select with for example every website they can deploy. By using the prompt option when the user clicks to run the build they will be presented with a dialog that will force them to enter these parameters.

So there you have it, a quick and simple build that can deploy any nuget package to any server. Simples.

As always if anyone has any questions or wants to go through anything feel free to contact me. Happy deployments!