We have already established that managing several Varnish instance(s) and keeping the configuration coherent between them can be difficult. Part one of this blog series covers this pretty well. Monitoring and deploying configuration, restarting the instances in a synchronized fashion requires some knowledge. The Varnish Administration Console (VAC) provides a full API that will help make this a whole lot easier.

Here in the second post of this series, we'll show you how to do VCL change management and continuous integration using the VAC API and Git.

First, some context

VAC manages multiple Varnish servers, particularly medium to large setups. Therefore the notion of a group of Varnish servers is first class citizens in VAC. Varnish servers belonging to the same group are assumed to have identical VCL and parameter changesets.

VCL management in VAC is inspired by the branch/commit model used in Git. In the context of VAC, a single group can be associated with a single active VCL branch. The branch allows for tracking VCL changes down to timestamp and the user who made the change. The head of the branch, which contains the latest VCL content, is deployed to the Varnish servers. Therefore rolling back VCL consists of pointing the head to another commit on the same branch, and deploying the content to the servers. This means that integrating VAC into a Git workflow is seamless.

The configuration

For this to work we'll need to have a local git repo and also a remote repo. This remote repo could be github or similar.

I used two folders in the same machine, conveniently called local and remote. 

On the remote folder: 

~/remote/$ git init
~/remote/$ git config --bool core.bare true

The second command turns a regular repo into a bare git repository (no working copy and the folder contains the actual repository data )

On the local folder

~/local/$ git init
~/local/$ git remote add origin ~/remote
~/local/$ git remote show origin
* remote origin
 Fetch URL: ~/remote
 Push URL: ~/remote
 HEAD branch: master
 Remote branch:
 master tracked
 Local ref configured for 'git push':
 master pushes to master (up to date)
~/local/$ git remote -v
origin ~/remote (fetch)
origin ~/remote (push)

In the local folder repo, you'll need to create a .vacrc file. 

Content of the .vacrc file:

GROUP_ID 521c5a464e2ceea0eb18334c
VCL_ID 5744511de4b0490e0e5db01c
USER_PASSWD vac:vac
VAC_LOCATION http://localhost
FILE_NAME main

All the ID-related information comes from the VAC. You can use both the VAC API or UI to check it out.

Some insight about these parameters:

• The group of Varnish servers you wish to apply the VCL - GROUP_ID
• The ID of the VCL in VAC - VCL_ID
• The VAC credentials - USER_PASSWD
• The location of the vac relative to your local repo - VAC_LOCATION
• Your VCL name - FILE_NAME

Setting up the git hook

In order to have this working, we'll need to setup a git update hook on the remote repo. 

Usually on a git repo the git hook folder is at .git/hooks. 

Here is the content of the update file in the  ~/remote/.git/hooks folder

#!/bin/bash

# git hook file
newrev="$3"
tree_sha1=$(git cat-file -p "$newrev" | awk '$1 == "tree" { print $2 }')
settings_sha1=$(git cat-file -p "$tree_sha1" | awk '$4 == ".vacrc" { print $3 }')
GROUP_ID="$(git cat-file -p "$settings_sha1" | awk '$1 == "GROUP_ID" { print $2}')"
VCL_ID="$(git cat-file -p "$settings_sha1" | awk '$1 == "VCL_ID" { print $2}')"
USER_PASSWD="$(git cat-file -p "$settings_sha1" | awk '$1 == "USER_PASSWD" { print $2}')"
VAC_LOCATION="$(git cat-file -p "$settings_sha1" | awk '$1 == "VAC_LOCATION" { print $2}')"
FILE_NAME="$(git cat-file -p "$settings_sha1" | awk '$1 == "FILE_NAME" { print $2}')"
TMPFILE=$(mktemp /tmp/updateXXXXXXXX)

PREVIOUS_HEAD=$(curl -u $USER_PASSWD $VAC_LOCATION/api/v1/vcl/$VCL_ID/head | python -mjson.tool | grep \$oid | awk -F \" '{ print $4 }' )

git cat-file --textconv $3:$FILE_NAME > $TMPFILE

echo VCL branch id is: $VCL_ID
echo Group id is: $GROUP_ID
echo Previous head is: $PREVIOUS_HEAD
echo Filename is: $TMPFILE
echo Content is: $(cat $TMPFILE)

echo Save and verify VCL for compile or syntax error against the group

SAVE_STATUS=$(curl -u $USER_PASSWD -H 'Content-Type: text/plain' -X POST --data-binary @$TMPFILE $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null )
#echo "curl -u $USER_PASSWD -H 'Content-Type: text/plain' --data-binary \"@$FILE_NAME\" -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null"
#SAVE_STATUS=$(curl -u $USER_PASSWD -H 'Content-Type: text/plain' --data-binary @$FILE_NAME -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null )
# if the save status is 200, then the VCL is uploaded ok and saved on all varnish cache, ready for deployment
# 400 status code means there are syntax error (see the payload for more details)

echo "Status$SAVE_STATUS"
rollback() {
 echo "Rolling back to last known state"
 ROLLBACK_STATUS=$(curl -u $USER_PASSWD -H "Content-Type: text/plain" -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push/$PREVIOUS_HEAD -s -w "%{http_code}" -o /dev/null )
 echo $ROLLBACK_STATUS
 echo Rollback complete
 exit 1
}

if [ $SAVE_STATUS -eq 200 ]
 then
 echo Verified. Now deploying VCL...
 DEPLOY_STATUS=$(curl -u $USER_PASSWD -X PUT $VAC_LOCATION/api/v1/group/$GROUP_ID/vcl/$VCL_ID/deploy -s -w "%{http_code}" -o /dev/null )
 if [ $DEPLOY_STATUS -eq 200 ]
 then
 echo Deployed.
 exit 0
 else
 echo Deployment unsuccessful. Rolling back...
 rollback
 fi
 else
 echo Roll back to last known state
 rollback
fi

It works?

Every time you commit a change of the FILE_NAME specified in .vacrc and you push it to your remote repo, the update hook will run. If there are no compilation errors or issues, that VCL will be deployed across your group of Varnish instance(s). If an issue occurs, it will rollback to the last working version. It is as simple as that. 

Summing up

This blog post demonstrated the steps required for a git-update hook with VAC API. The example provided was designed to address VCL change management and continuous integration for anybody who needs to manage multiple Varnish instances. The VAC API is a flexible platform for managing groups of Varnish instances, and together with Git, this integrated solution delivers tracking VCL changes, auto deployment of VCL, rollback of VCL and maintaining that the VCL stays consistent in an event of Varnish restart. 

The Varnish Administration Console is part of the Varnish Plus offering. If you are interested in a trial contact us.