Two-Minute Tech Tuesdays - Transit Buffer

In this week’s episode of Two-Minute Tech Tuesday, we'll talk about the Transit Buffer. The Transit Buffer is a Varnish Enterprise feature that reduces transient memory consumption by pacing backend fetches and only storing data at the pace that the client can consume it.

This is typically used for fast backends that serve large, uncacheable objects that are consumed by slow clients. The goal of this is to reduce the memory consumption in Varnish while that happens.

The Transit Buffer does this by matching the pace of the client while fetching and only storing response data from that fetch in transient memory, at the pace that the client can consume it.

Without the transit buffer, a request for a large, uncacheable file will be sent to the origin. As the origin responds, Varnish temporarily stores the full payload in memory -- transient memory.

If this "movie.mp4" file is 5 GB, 5 GB of memory is held as this slow client is consuming it. And as the data is sent over the wire, transient storage slowly drains until the response is completely consumed by the client and the transient storage is empty again.

With the Transit Buffer enabled, Varnish can reduce the amount of memory that is held in transient storage to satisfy that uncacheable request.

So when that same large uncacheable file gets requested, Varnish will forward that request from the client to the origin, receive it, and as it receives, will notice that the client is slow in consuming it and will slow down the fetch and make sure its ahead in terms of what it is storing in transient storage.

However, we don't want to slow down the fetch too much. If you set the Transit Buffer to 1 MB, up to 1 MB is kept in transient storage under the form of chunks and is continuously sent to the client. For example, 512 KB at a time while another 512 KB is fetched at the same time.
What we want to do is keep enough data in the transient storage to ensure that the client doesn't have to wait.

You can enable the Transit Buffer by setting the "transit_buffer" runtime parameter to a value.

varnishd -p transit_buffer=1M

In this case, that's 1 MB. That means we're staying up to 1 MB ahead of the client, but not more. You can also set it on a VCL level by setting "beresp.transit_buffer" to a value of your choice.

However, Transit Buffer is not limited in terms of size, which means if transient storage starts growing really quickly, your server can run out of memory. Transient storage is used to keep track of shortlived data that doesn't end up in the normal caching storage, but also uncacheable data that needs to sit somewhere in memory while the client is fetching it.

Using the Transit Buffer is a trade-off. On the one hand, we try to keep memory consumption in the transient storage as low as possible. On the other hand, we're not trying to tie up the fetch for too long.

Join us next week for another Two-Minute Tuesday topic, presented to you in 2 minutes or less.