One step closer to Varnish automation pt 2 - VCL change management with VAC API & Git

 

We have already established that managing several Varnish instance(s) and keeping the configuration coherent between them can be difficult. Part one of this blog series covers this pretty well. Monitoring and deploying configuration, restarting the instances in a synchronized fashion requires some knowledge. The Varnish Administration Console (VAC) provides a full API that will help make this a whole lot easier.

Here in the second post of this series, we'll show you how to do VCL change management and continuous integration using the VAC API and Git.

First, some context

VAC manages multiple Varnish servers, particularly medium to large setups. Therefore the notion of a group of Varnish servers is first class citizens in VAC. Varnish servers belonging to the same group are assumed to have identical VCL and parameter changesets.

VCL management in VAC is inspired by the branch/commit model used in Git. In the context of VAC, a single group can be associated with a single active VCL branch. The branch allows for tracking VCL changes down to timestamp and the user who made the change. The head of the branch, which contains the latest VCL content, is deployed to the Varnish servers. Therefore rolling back VCL consists of pointing the head to another commit on the same branch, and deploying the content to the servers. This means that integrating VAC into a Git workflow is seamless.

The configuration

For this to work we'll need to have a local git repo and also a remote repo. This remote repo could be github or similar.

I used two folders in the same machine, conveniently called local and remote. 

On the remote folder: 

~/remote/$ git init
~/remote/$ git config --bool core.bare true

The second command turns a regular repo into a bare git repository (no working copy and the folder contains the actual repository data )

On the local folder

~/local/$ git init
~/local/$ git remote add origin ~/remote
~/local/$ git remote show origin
* remote origin
Fetch URL: ~/remote
Push URL: ~/remote
HEAD branch: master
Remote branch:
master tracked
Local ref configured for 'git push':
master pushes to master (up to date)
~/local/$ git remote -v
origin ~/remote (fetch)
origin ~/remote (push)

In the local folder repo, you'll need to create a .vacrc file. 

Content of the .vacrc file:

GROUP_ID 521c5a464e2ceea0eb18334c
VCL_ID 5744511de4b0490e0e5db01c
USER_PASSWD vac:vac
VAC_LOCATION http://localhost
FILE_NAME main

All the ID-related information comes from the VAC. You can use both the VAC API or UI to check it out.

Some insight about these parameters:

  • The group of Varnish servers you wish to apply the VCL - GROUP_ID
  • The ID of the VCL in VAC - VCL_ID
  • The VAC credentials - USER_PASSWD
  • The location of the vac relative to your local repo - VAC_LOCATION
  • Your VCL name - FILE_NAME

Setting up the git hook

In order to have this working, we'll need to setup a git update hook on the remote repo. 

Usually on a git repo the git hook folder is at .git/hooks. 

Here is the content of the update file in the  ~/remote/.git/hooks folder

#!/bin/bash

# git hook file
newrev="$3"
tree_sha1=$(git cat-file -p "$newrev" | awk '$1 == "tree" { print $2 }')
settings_sha1=$(git cat-file -p "$tree_sha1" | awk '$4 == ".vacrc" { print $3 }')
GROUP_ID="$(git cat-file -p "$settings_sha1" | awk '$1 == "GROUP_ID" { print $2}')"
VCL_ID="$(git cat-file -p "$settings_sha1" | awk '$1 == "VCL_ID" { print $2}')"
USER_PASSWD="$(git cat-file -p "$settings_sha1" | awk '$1 == "USER_PASSWD" { print $2}')"
VAC_LOCATION="$(git cat-file -p "$settings_sha1" | awk '$1 == "VAC_LOCATION" { print $2}')"
FILE_NAME="$(git cat-file -p "$settings_sha1" | awk '$1 == "FILE_NAME" { print $2}')"
TMPFILE=$(mktemp /tmp/updateXXXXXXXX)

PREVIOUS_HEAD=$(curl -u $USER_PASSWD $VAC_LOCATION/api/v1/vcl/$VCL_ID/head | python -mjson.tool | grep \$oid | awk -F \" '{ print $4 }' )

git cat-file --textconv $3:$FILE_NAME > $TMPFILE

echo VCL branch id is: $VCL_ID
echo Group id is: $GROUP_ID
echo Previous head is: $PREVIOUS_HEAD
echo Filename is: $TMPFILE
echo Content is: $(cat $TMPFILE)

echo Save and verify VCL for compile or syntax error against the group

SAVE_STATUS=$(curl -u $USER_PASSWD -H 'Content-Type: text/plain' -X POST --data-binary @$TMPFILE $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null )
#echo "curl -u $USER_PASSWD -H 'Content-Type: text/plain' --data-binary \"@$FILE_NAME\" -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null"
#SAVE_STATUS=$(curl -u $USER_PASSWD -H 'Content-Type: text/plain' --data-binary @$FILE_NAME -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push -s -w "%{http_code}" -o /dev/null )
# if the save status is 200, then the VCL is uploaded ok and saved on all varnish cache, ready for deployment
# 400 status code means there are syntax error (see the payload for more details)

echo "Status$SAVE_STATUS"
rollback() {
echo "Rolling back to last known state"
ROLLBACK_STATUS=$(curl -u $USER_PASSWD -H "Content-Type: text/plain" -X POST $VAC_LOCATION/api/v1/vcl/$VCL_ID/push/$PREVIOUS_HEAD -s -w "%{http_code}" -o /dev/null )
echo $ROLLBACK_STATUS
echo Rollback complete
exit 1
}

if [ $SAVE_STATUS -eq 200 ]
then
echo Verified. Now deploying VCL...
DEPLOY_STATUS=$(curl -u $USER_PASSWD -X PUT $VAC_LOCATION/api/v1/group/$GROUP_ID/vcl/$VCL_ID/deploy -s -w "%{http_code}" -o /dev/null )
if [ $DEPLOY_STATUS -eq 200 ]
then
echo Deployed.
exit 0
else
echo Deployment unsuccessful. Rolling back...
rollback
fi
else
echo Roll back to last known state
rollback
fi

It works?

Every time you commit a change of the FILE_NAME specified in .vacrc and you push it to your remote repo, the update hook will run. If there are no compilation errors or issues, that VCL will be deployed across your group of Varnish instance(s). If an issue occurs, it will rollback to the last working version. It is as simple as that. 

Summing up

This blog post demonstrated the steps required for a git-update hook with VAC API. The example provided was designed to address VCL change management and continuous integration for anybody who needs to manage multiple Varnish instances. The VAC API is a flexible platform for managing groups of Varnish instances, and together with Git, this integrated solution delivers tracking VCL changes, auto deployment of VCL, rollback of VCL and maintaining that the VCL stays consistent in an event of Varnish restart. 

The Varnish Administration Console is part of the Varnish Plus offering. If you are interested in a trial contact us.  

 See the VAC demo

Topics: VCL, Varnish automation, VAC, git, automation, Github

09/06/2016, 13:30 by Hugo Cruz

All things Varnish related

The Varnish blog is where the our team writes about all things related to Varnish Cache and Varnish Software...or simply vents.

SUBSCRIBE TO OUR BLOG

Recent Posts

Posts by Topic

see all

Varnish Software Blog