Adding Varnish Cache related statistics to your APM

Typically Varnish Cache sits in front of your application servers, be it in the public or private clouds. The transparent nature of the reverse proxy means that Varnish Cache is agnostic to both inbound client request and outbound backend response, delivering either a cached copy or freshly rendered response from the application servers. Performance gains attained for the overall response is directly correlated to cache hits achievable. Naturally this is highly dependent on the incoming traffic pattern and the supporting application server. Therefore in order to see the full performance picture of your web stack, or to identify degradation in performance, we highly recommend incorporating the monitoring of Varnish Cache as part of your existing Application Performance Monitoring (APM) tools.

This blog post aims to introduce AppDynamics Varnish Extension for gathering cache related statistics. At the core of AppDynamics Varnish Extensions is varnishstat data that are fed back to AppDynamics Pro via varnish-agent in order to provide key statistics for a complete performance overview of your web stack.

varnishstat and its myriad of counters

Monitoring Varnish Cache is not exactly for the faint of heart. There are a myriad of counters that appear to convey information that can only be understood by the core Varnish developers. Have no fear and rest assured that these counters are very useful, although perhaps a little misunderstood. Coupling cache related statistics with information extracted from an APM tool, together these statistic will paint a pretty accurate picture of your web stack from the end-users’ perspective. And why is monitoring your caching server so important? It is essentially the shield for your backend application servers.

varnish-agent to the rescue: Exposing varnishstat

varnish-agent is a small daemon process written in C that communicates with varnish and other varnish related services, such as varnishlog and varnishstat. This enables remote control and monitoring of a single varnish cache instance. In addition, these management and monitoring services are exposed through a RESTful interface, which speaks JSON. This is the core mechanism and the recommended approach for accessing varnishstat.

What is Application Performance Monitoring (APM)

According to Gartner, APM can be defined by the following five dimensions:

• End-user experience monitoring
• Application topology discovery and visualisation
• User-defined transaction profiling
• Application component deep dive
• IT operations analytics

And AppDynamics is classified the market leader in APM at the time of writing.

A vivid canvas painted by statistics

Statistic provided by varnish-agent/varnishstat often hints at the current state of the cache server. When coupled with additional information such as memory, IO usage, a very vivid picture about the exact state of the system starts to surface.

A typical Varnish monitoring example is to be able to identify the symptoms when Varnish Cache has exhausted its cache storage. Cache utilisation counters are provided by varnishstat through varnish-agent. Assuming you have a single cache storage, the counter SMF.s0.g_bytes indicates cache utilisation in bytes, and the counterSMF.s0.g_space indicates available cache storage in bytes.

For this example, the assumption could be that Varnish is assigned 1GB memory for cache storage on a server with 2GB physical memory. Upon exhausting cache storage, Varnish will attempt to evict the least recently used object from cache. The counter n_lru_nuked in Varnish will begin to increment, indicating that objects are being forcibly evicted. The implication for the eviction is that cached content are not staying in the cache for the intended time-to-live duration, thus forcing requests to be sent to the backend, resulting in performance degradation. Early monitoring of the aforementioned counters can be a precursor to safeguard against performance degradation.

Another example could be that Varnish is assigned 2GB memory for cache storage on a server with 1GB physical memory. As available physical memory becomes scarce, the Linux kernel will swap out less used pages into swap space and freeing up memory for processes that requires it immediately.

This process is a potential IO bottleneck for Varnish, as paging into swap involves writing to disks. Swap utilisation is a common statistic available in APM tools, such as AppDynamics Pro, or command line tool such as vmstat. In this case, the performance degradation can be significant if rotational disks are installed in place of SSDs. Again, looking out for abnormal swap usage can also be a precursor to safeguard against performance degradation.

Include Varnish into your APM today. Try it for yourselves.

AppDynamics Pro and varnish-agent is the perfect marriage for obtaining a full performance overview of your web stack. The installation process is easy and well documented every step of the way. Being able to spot performance degradation before it affects your application server, and your business is absolutely critical. And getting started is as easy as pie.