Intuition tells us that it is impossible to lower costs without decreasing web performance, or increase web performance without introducing additional expense. This blog post is about the remarkable ways in which Varnish Enterprise’s Massive Storage Engine (MSE) solves this challenge, specifically within DevOps contexts where efficient delivery of large developer artifacts like repositories and images is business critical.

Balancing Cost and Performance within DevOps Infrastructure

Organizations using artifact management tools at scale within software development workflows often face significant challenges, notably in terms of latency and costs. Development teams frequently encounter issues when retrieving large volumes of data from resources like Artifactory, leading to substantial delays.

This is especially true when organizations deploy artifact management tools on premise while global teams require access. High demand, round-trip distance and traffic loads harm developer productivity and overall business efficiency. Addressing these issues often involves increasing infrastructure and licensing, which introduces additional risks and drawbacks. Using artifact management on cloud platforms can also entail large egress bandwidth fees in line with the large object size of the artifacts being pulled.

Massive Storage Engine is Powerful Cache Storage

How does MSE help enterprises experiencing these bottlenecks and cost pressures? Let’s quickly review Varnish and see where MSE fits in. Varnish's core strength lies in its advanced caching engine. This engine stores responses from the origin application, efficiently serving them to clients upon subsequent requests. This process significantly offloads the burden from Artifactory and other HTTP-based applications. Varnish's design enables it to handle numerous requests without requiring extensive server resources. Varnish’s "request coalescing" feature further reduces the impact on the origin application by merging multiple requests for the same resource into a single backend request. As software, it is platform agnostic and deployable across on-premise, cloud, containerized infrastructure wherever additional content delivery resources are needed.

While in-memory caching is effective, artifact caching scenarios demand disk storage solutions for managing large data volumes. The default setup in the open-source Varnish Cache stores objects in memory, but is not optimized for handling large data volumes. This is where the Massive Storage Engine (MSE) comes into play. MSE is Varnish Software’s proprietary storage engine, and offers an optimal blend of memory speed and disk storage capacity and reliability. It is specifically designed for high performance and scalability, mitigating disk fragmentation by storing objects in pre-allocated large files. This approach prevents the operating system from becoming overwhelmed with extensive file descriptors and keeps Input/Output Operations Per Second (IOPs) to a minimum. An equitable eviction strategy automatically removes the least-accessed objects when necessary, reducing competition for storage space.

Cache Persistence

MSE's persistence feature also adds to Varnish's resilience capabilities. Caches remain intact even after restarts, eliminating the need for time-consuming cache reloads and enhancing speed and reliability. It also allows operators to precisely configure storage in Varnish Enterprise, facilitating the creation of a persistent data caching cluster, capable of managing petabytes of data.

As a fully integrated component of Varnish Enterprise, MSE incorporates integrity checks and error handling to prevent and mitigate downtime. It is ideally suited for caching large data sets such as videos, metadata, and DevOps artifacts. MSE also allows for the distribution of caches across various storage devices and supports warming up caches separately before mounting them with hot data for improved disaster recovery strategies.

A Massive Storage Engine Success Story

Here’s one example of MSE delivering an effective solution for artifact caching challenges. A global software business specializing in web development approached Varnish Software for help. Their DevOps team were fetching nearly 1000 TB of data from Artifactory each month - a scale not unusual amongst enterprise technology businesses. Not only were traffic costs becoming an issue, but performance challenges were also hurting productivity for the globally distributed teams.

They deployed Varnish Enterprise caching software between Artifactory instances and end-users, with MSE included. The efficient object storage offered by MSE meant that they could reduce the traffic hitting Artifactory while simultaneously moving vast amounts of data closer to the team that needed access. Monthly traffic costs went down by 20%, while Artifactory performance went up 25%.

To find out more about how MSE contributes to Varnish’s ability to optimize DevOps workflows, watch our webinar on demand.