Introducing Varnish Massive Storage Engine 2.0 with Persistence

A couple of years ago, we introduced the Varnish Massive Storage Engine (MSE).

The need for MSE came about when many sites started to have terabytes of content, and Varnish didn’t perform very well with these big data sets. MSE was designed and implemented to suit the requirements of anyone who needed to combine a large cache with high performance.

The first release of MSE included:

• Capacity for up to 100+ terabytes of storage on each node
• Fragmentation-proof allocation algorithm
• Higher hit rates due to LRU being replaced with LFU

A previous blog post on MSE 1.0 provides a deeper dive into MSE, explaining its design and implementation in greater detail.

What is new in MSE 2.0?

The big news in MSE 2.0 is persistence. Ultimately, MSE 2.0 has all the same features as MSE 1.0, the core capabilities of Varnish and adds in the possibility of (optional) persistence. Good news, right? Yes - but what does it mean?

The dictionary defines “persistence” as:

“the fact of continuing in an opinion or course of action in spite of difficulty or opposition.”

This is exactly what we wanted to achieve with MSE 2.0: the ability to oppose difficult situations. We wanted cached content to survive (or persist) over (wanted or unexpected) restarts, maintaining a stable performance over time.

Persistent storage has been in high demand, mostly because re-populating a cache after a restart takes considerable time, in particular when large objects are involved, and time is important when high performance is a hard requirement.

Also, Varnish servers can be rebooted without adding significant strain on backends, avoiding backend failures due to a large number of backend requests.

MSE 2.0 features in detail:

• The ability to persist cached objects through a same-version Varnish restart;
• It is very useful for live video and video-on-demand setups. It supports cut-through-forwarding, streaming and handles range and conditional requests;.
• Weak filesystem transaction semantics. A regular request can be sent to the client before content is confirmed to be stored on disk;
• Urge and ban support so long as they are  persisted before the client request is acknowledged.

Persistence adds a little overhead, which is negligible compared to what you will gain in return.

Availability

The Varnish Massive Storage Engine is a feature that is exclusive to Varnish Plus.

If you would like to evaluate MSE I encourage you to send an email to our sales organization (sales@varnish-software.com) and they will set you up with a trial account. In addition to MSE, this trial will give you access to other great features, such as the Varnish Administration Console, the Varnish Custom Statistics Engine, the Varnish Tuner and other useful stuff.

Join our April 7th, 2016 webinar Five keys to massive data handling on the topics of streaming, persistence and the endless possibilities of the Massive Storage Engine.

Photo (c) 2005 Gray Lensman QX! used under Creative Commons license.