Scaling Open Source Network Proxy:  Leveraging Hitch for SSL/TLS terminations

Scaling Open Source Network Proxy: Leveraging Hitch to Handle Tens of Thousands of Connections on Multicore Machines

The open source community is celebrating the 10th anniversary of Varnish, the HTTP reverse proxy and web cache powering many of the busiest sites in the world. At Varnish Software, we work on developing key components that keep the infrastructure of the web running. While you likely know about Varnish Cache, you may not be aware of its lesser-known cousin Hitch, and the important role it can play both behind and in front of your Varnish installations.

Hitch is a network proxy that terminates TLS/SSL connections and forwards the unencrypted traffic to a backend, typically Varnish, to provide HTTPS support for websites at scale. It's designed to handle tens of thousands of connections efficiently on multicore machines. As you would expect, it is fast and event-driven, with Marc Lehmann’s libev at its core. It is an open source project, and you can find the code on github. We are happy to review patches/pull requests made by the user community. Hitch has received thorough testing on Linux, but it also works on other *nixes" as well. Features include:

• Support for TLS1.0, TLS1.1 and TLS1.2
• SNI, with and without wildcard certificates
• OCSP stapling support (coming with Hitch 1.3.0 which is now in beta)
• Support for HAProxy's PROXY protocol versions 1 & 2
• Seamless configuration runtime reloads of certificates and listen endpoints
• Performance safety for large installations with up to 15 000 listening sockets and 500 000 certificates

What is less known is the fact that it was made to be a TLS proxy and to run *any kind* of wire protocol over TLS, not only HTTP. So you can put it in front of your email server, video origin or any custom wire protocols you might need to encrypt, and you will get *consistent* TLS support. When all you need is listening on port 443 (or something else), strip TLS, (optionally add a PROXY header) and send to localhost, then Hitch is your best friend. It does one thing and one thing only: TLS/SSL termination.

In the beginning there was stud

The original codebase came from the stud project originally written in 2011 by Jamie Turner while working at Bump Technologies. It got quite a lot of attention during its first year of life and got love (in the form of patches) from the likes of Stripe, Fastly and HAProxy technologies. After announcing future plans for stud things went silent. And a year later Google bought Bump. After the acquisition of Bump stud was effectively abandoned.

But as the project had been picked up outside of Bump it was very much alive with lots of activity in github. The most prominent fork was WhatsApp’s. And yes, you guessed right, it was then acquired. I have the feeling Facebook is pretty good at keeping their devs busy… Or is it that all main maintainers of stud get acquired?

Enter Hitch

How does Varnish Software fit in? Why did Varnish Software need to make a TLS proxy? The background for us making Hitch has many aspects, of which the following three are the most important:

• First, Varnish Cache does not support TLS. Even if you can do TLS/SSL termination in front of it with a range of hardware appliances or software, none of these is made with scalability AND Varnish integration in mind. Our customers required such capabilities be developed, so we did.  Our company provides support for TLS/SSL client termination via Hitch as part of the Varnish Plus software suite.
• Second, we live in the post-Snowden era. Facing the reality of wholesale surveillance, the online world is moving to an “encryption everywhere” model. Adoption of TLS/SSL on the web has only been accelerated by incentives such as Google's ranking algorithm ranking sites using HTTPS higher than those that do not. Initiatives such as Let's Encrypt have made this even easier and more affordable to implement for any and all sites on the net today.
• Third, the ratification of the next generation web protocol HTTP/2. While the specification itself does not require encryption, all main browser vendors currently support the protocol only when encrypted transport is used. So in practice HTTP/2 without TLS is a no-go. This protocol is currently only used by 7.7% of the web according to W3Techs, but given the increasing support for it by browsers and web servers, there is no question that its usage will increase. If we wanted to stay relevant in this brave new world, we needed to supports HTTPS.

So we adopted stud, the Scalable TLS Unwrapping Daemon, codebase and as my colleague Lasse Karstensen’s blog post explains, we turned it into our own. We cleaned it up, took out the cruft, took the time to let it grow, taught it a few new capabilities and, finally, gave it a new name: Hitch.

Quoting Jamie’s recent blog post “I recently learned about hitch, from the fine folks at Varnish Software. To the extent my opinion matters anymore on these things, I recommend people use that instead of stud. I'll be pushing a changeset up to the ‘official’ stud repository redirecting people to this project.” Being the recommended alternative going forward, we have definitely reached our goal with the Hitch project.

The PROXY protocol

Today, it is very common to have a web architecture composed of multiple tiers which can encompass proxies, reverse-proxies, load-balancers, WAF, application servers.

Normally when connecting to a remote server a proxy in a given tier will use its own IP stack, which results in losing the source and destination IP and port of the initial TCP connection information, especially when another proxy is involved. To counter that problem Willy Tarreau from HAProxy Technologies designed a way to chain proxies, load-balancers and reverse-proxies without losing the client IP information. This is what became the PROXY Protocol, which is:

• Protocol agnostic -- It can work with any layer 7 protocols, even when encrypted
• Seamless -- It does not require any infrastructure changes and is not affected by NAT-ing firewalls
• Scalable -- Proxies are expected to be faster than origins; the protocol was designed with implementation details in mind.

For using the PROXY protocol, the only condition is that both endpoints of the connection MUST be protocol compatible. This allows for so-called "dumb proxies," which talk raw TCP on one side and raw SSL on the other in a very reliable fashion. This is done without the proxy having any knowledge whatsoever of what protocol is being sent through the wire, hence "dumb.” In our case Hitch, a TLS terminating proxy, acts as a dumb proxy for Varnish Cache. The protocol is supported by Hitch since its initial release and by Varnish Cache since the 4.1 release.

Thijs Feryn has done a fantastic write-up, how-to and video about Varnish Cache 4.1 and PROXY protocol support. For more details about the PROXY protocol check the HAProxy blog.

Securing Varnish with Hitch

In case you already have a certificate you only need to configure Hitch minimally. If you don’t have a certificate at hand, then the easiest way to get up and running with Hitch is to get a certificate from Let's Encrypt adding this in front of Varnish. Doing this is pretty straightforward, and you can read our own Denis Brækhus’s blog post explaining step-by-step how to do this.

What’s next for Hitch?

The Hitch TLS Terminator project has had a great start. The number of contributors to the project has grown, it has gotten quite a few stars, many thousands of pageviews per month in github and even made it to the frontpage of Hacker News and sparked an interesting thread. It is used together with Varnish Cache by many sites and APIs, and major CDNs use it for TLS termination.

Varnish Cache development efforts are now focused on HTTP/2 support, and as we will need a well integrated facility that can do tightly integrated TLS termination for us, Hitch will be it. Now that OCSP Stapling has hit the master branch, the next step will be adding support for ALPN.

If you have not tried it yet, make sure you add Hitch, the TLS Swiss-Army Knife, to your toolbox. You can also learn more about TLS/SSL by watching our on-demand webinar.

Photo (c) 2008 David Goehring used under Creative Commons license.