Auto-scaling memory usage to increase cache size in Varnish

During the recent release of Varnish Enterprise 6.0.6, a significant amount of attention focused on in-process TLS.

But that’s not the only game-changing addition to the Varnish Enterprise technology stack. We now want to talk about Memory Governor, and what it means for everyone using Varnish Enterprise for streaming, web acceleration and content delivery.

Or, for those of the TL;DR persuasion: Memory Governor enables significant increases in memory availability and cache size.

Life before Memory Governor

Before the birth of the Memory Governor, the Varnish administrator would control Varnish' memory usage by specifying a memory cache size (bytes of Cache payload) which left enough memory for other needs - object data structures, threads, workspaces and so on.

To be able to sleep at night, the administrator would also leave ample headroom for peak traffic conditions, because several of the data structures would vary throughout the day.

A common question to the experts at Varnish Software would be: How much of my memory can I allocate for the cache, and how much will Varnish use for other things? The question was always: We can create an estimate for you, but it depends. If objects are small, more space will be used for object data structures, and if you have many concurrent connections, you will need more workspaces. Also, your VCL will influence how much each workspace will consume, and there are many sorts of VMODs which can influence the memory usage.

Usually, the wise Varnish administrator would leave a certain amount of headroom, monitor the memory usage, and use their experience to select a cache size which would leave some free memory, even when traffic was very high.

The figure shows a simplified model of how this would work. The administrator controls the level of the black line, while everything above can vary.

Figure 1: At low traffic, more memory is unused.

However, sometimes things changes, and the model breaks. Perhaps there is a breaking story on a news website or a video clip of a cat on your site unexpectedly goes viral. Perhaps a paywall solution is introduced, and the VCL uses more memory than before. The headroom suddenly is all used up, and you find yourself in deep trouble.

When such an event occurs, the wise administrator will, to be able to sleep at night again, allocate even more memory as headroom. The unfortunate effect of this is that under normal circumstances, there will be a significant amount of unused memory.

Since we are Varnish Software and obsessively focused on performance and stability, we asked ourselves a question.

Is there a way we can utilize the machine's memory resources in a more efficient way by reducing the amount needed as headroom, while also maintaining our protection against sudden, huge demand? Can we have it both ways? We can, and we can make configuring Varnish easier in the process. Hello Memory Governor.

What is the Memory Governor?

Memory Governor is a memory management system for Varnish Enterprise that can set limits on memory usage for the whole Varnish process. This limit is called the memory target, and the memory governor handles the cache size automatically, scaling it up or down in accordance with traffic patterns and memory availability. you can even adjust the memory target while Varnish is running!

During usual traffic conditions, this results in a significant increase in the amount of memory available for caching - good news! More objects can be cached, and more content can be given the Varnish low latency treatment. The Varnish memory footprint will no longer rise and fall in line with the fluctuating traffic, but will be a steady figure based on the specified limit.

During periods of very high demand, when Varnish needs more memory for threads and workspaces, the effective cache size will be reduced accoringly, so there is no risk of saturating the memory and causing content delivery to slow down or cease. The least recently used objects will be purged from the cache to make room for newer objects or other data structures, so that memory usage at all times stay within safe parameters. This allows the administrator to leave significantly less memory for headroom compared to what was needed without the memory governor.

In short, switching to the Memory Governor unlocks even greater cache performance by allowing more memory usage. It keeps content providers in control and makes life easier for Varnish admins while allowing them to take full advantage of the capabilities of their infrastructure.

The following figure illustrates how the Memory Governor improves the quality of Varnish. The administrator controls the amount of memory the Varnish process will use (thick line), greatly reducing the need for headroom.

A natural question is, of course, why do we need headroom at all with the memory governor? The answer is that there are other users of memory in a server, even if Varnish is the only running service. For example, the kernel needs some room for socket buffers, page tables, and other data structures for supporting Varnish. The kernel's needs will vary a bit depending on the traffic, but to a considerably lesser extent than Varnish itself. For this reason, the default memory target is 80% of system memory, and we encourage users to monitor system memory usage to understand how it reacts to different traffic patterns. For most users, the 80% figure can be tuned up significantly, but do not do this before you understand your overall system memory usage with different traffic patterns.

How to start using Memory Governor

Visit our Docs page here for more information on configuring Memory Governor in your Varnish instance. New users should follow the process on the updated Massive Storage Engine page, where the examples are now adjusted to use the Memory Governor.