Getting billing data from Varnish

Still doing your billing by hand?

If you have an ISP or a Hosting company there is a chance that you serve parts or most of your traffic using Varnish. Hence the log data Varnish provides is the base for how much traffic you bill your customers.

Quoting from the Varnish documentation:

One of the really nice features in Varnish is how logging works. Instead of logging to normal log file Varnish logs to a shared memory segment. When the end of the segment is reached we start over, overwriting old data. This is much, much faster than logging to a file and it doesn't require disk space. Besides it gives you much, much more information when you need it.

The flip side is that if you forget to have a program actually write the logs to disk they will disappear.

So, all data present in the shared memory log can be viewed either by using varnishlog to display it raw log format. You can also have varnishncsa present the data in the Apache / NCSA "combined" log format, which can be partially customized.

Am I currently able to do my billing in Varnish?

Today, it is possible to get the amount of data (body content of every object) you are serving with one single Varnish, but not the whole traffic data. So, even if you somehow aggregate the data from several Varnish instances, your end-customer bill will be incorrect. The reason for this is that currently Varnish does not provide several information points that will allow you to bill such as server.ip, remote/backend server status, bytes from user to cache, and bytes to and from backend.

We have solved that problem and can provide you with some enhancements that will make it possible for you to get this data with varnishncsa. If you have multiple caches, you can have this data aggregated, and voilà, you will have all the correct information you need to send your bills right away.

But I do my billing per domain. Can you help?

This is a bit more tricky, but do not worry, we might be able to help you cover that as well in the near future (and likely sooner than you think).

As of now Varnish Custom Statistics (VCS) produces real-time aggregated statistics from a group of Varnish servers. The statistics produced are then grouped according to keys defined in the Varnish Configuration Language (VCL) and presented in time series. VCS makes it easy for your web hosting company or ISP to track statistics for each individual hosted site by giving you a per-site breakdown of data such as traffic, hit rates and response times. This tool also allows your system administrators to easily identify and diagnose issues in your Varnish setup, so that you can quickly identify which URLs have the slowest response times and react accordingly.

Our plan is to extend VCS to also provide you with the billing data mentioned above aggregated from all your Varnish servers, so you will be able to get all your billing information from one single source. Isn’t that neat?

If you want us to help you do your billing based on Varnish logs today or want to know more about our plans for VCS so you can do your per domain billing in the future, please send me an email to ruben@varnish-software.com and I will get back to you with more information.