Massive Storage Engine tackles cache, storage & content delivery challenges

Storage is one of those things most of us probably don’t think much about when it comes to provisioning enough for our data. With innovations in cloud storage and the universality of streaming services, most of us - from the average consumer to the small business - don’t have massive storage demands. But what happens when those needs grow? When the business in question is growing, and along with the company, the storage and retrieval needs grow, too? We as consumers rely on on-demand streaming services, for example, but where do we think that storage capacity lives? And how do we get instant access to massive libraries of video or music content? The press-a-button-and-it’s-here near-magic we now expect is nearly miraculous when you consider all the elements that must go right to deliver this content fast and consistently on-demand.

Most companies and their content libraries, however small the data set, can benefit from a streamlined, more efficient, smarter storage setup. But where it really makes a difference is in designing storage for the high-performance needs of video distribution, CDNs, and large-cache use cases. That is, the companies consumers turn to for on-the-go music of their choosing and the content that fuels the cultural phenomenon we know as “Netflix and chill”, whether or not it’s even Netflix we’re using.

The engines of on-demand streaming: Taking care of performance

The mechanics of what happens when a user selects their content and presses play is a mystery to most, and the only time a user will react is when the technology hiccups or fails. When it comes to video streaming especially, which is in an entirely different league from what it was during its early days, users expect an instant, consistent stream from the get-go: no waiting, no buffering, whatever device they choose, wherever they happen to be. This always comes down to performance. While many factors are out of the control of the company serving the content, such as network speed and bottlenecks, there are a lot of things that can be implemented and optimized to make sure that everything else is in place for a superior streaming experience.

One of the pieces of the performance puzzle, obviously, is storage. With or without a massive content library, the same principle of being able to serve both long-tail and hot-set content applies. Your hot-set, in-demand content has to be ready to serve at any scale while the long tail, which might be content that is rarely, if ever, accessed, still has to be served as quickly as if it were part of the hot set. And then there’s everything in between.

Cache, store, serve strategically to achieve performance enhancements

To manage the hot set, long tail and everything in between, Varnish Software developed Massive Storage Engine (MSE) several years ago, and we have updated it continuously to meet the exponentially growing demands of the market and to ensure better performance and efficiency in storage handling and content delivery, making MSE smarter all along.

What does MSE do?

The Massive Storage Engine is responsible for storing objects in cache, handling their metadata, keeping track of the hot data set and object eviction when the cache is full or when certain objects are forcefully purged. Over time we’ve improved MSE to ensure:

• Less fragmented storage
• Fairer and smarter storage/caching and cache eviction policies
• Better time and resource use/efficiency
• Persistence across planned and unplanned restarts, which can also act as a secondary/backup backend
• Better compact memory object structure for less storage overhead
• Better distribution, better disaster recovery
• Performance gains, setting MSE up for high-performance and high-concurrency processing

Because Massive Storage Engine was designed from the ground up to solve the storage and delivery challenges of businesses with large volumes of content, it tackles limitations that exist in the memory and file stevedores that are also available in Varnish. We’ll talk more about the details of MSE as we go, but it’s important for now to know:

• The latest iteration of MSE is available now in Varnish Enterprise 6
• Performance - and performance improvements - is the main driver, as always with Varnish
• MSE is a key component in high-performance streaming content delivery architectures and goes a long way toward helping you scope out your streaming architecture needs.

If you’re ready to learn more, get in contact or download the factsheet below.