Konstantin Ryabitsev's post Creepy crawlies puts numbers on something anyone running a public git host has felt for a while: agents are an increasingly heavy burden on anyone publicly hosting information. So an ever increasing number of people are sending out more and more agents on longer and longer goose-chases and the traffic is increasing at an alarming rate.
Increasing traffic is not an entirely new problem. We’ve seen it before. Back around the turn of the century, computers were slow. Rendering HTML was expensive. To alleviate this Akamai introduced Edge Side Includes; a markup language which allows a cache to cache fragments of a page. Say you want to reuse the computationally expensive weather report on every page. ESI can help you do this by wrapping the feature in ESI tags.
The kernel.org git is a strong candidate for ESI caching because each commit is immutable and has a globally unique ID. As Konstantin clearly expresses in his post, git.kernel.org’s dataset is small, the problem is that every piece of data is presented in 922 different ways. There are 1.5 million distinct commits. That is a caching problem of a particular kind: the key space is much bigger than the content space. Edge Side Includes are built for exactly this, so I spent a few days to see how far I can get making the website go faster.
TL:DR: every commit in linux.git, rendered and gzipped, is about 22 GB of gzipped content. You can hold the entire thing in RAM on one machine, and once you do, the fork count stops mattering.
cgit already sets a ten year Expires on any page that has an id= parameter. The problem is that each variant of the same commit frames the commit in a different way. What differs between those copies is confined to the frame of the page:
So the plan was: cut the page at the diff, let the cheap frame be rendered per request, and make the diff a fragment whose bytes are identical no matter which fork or branch asked for it.
In my fork there is one new cgitrc option, enable-esi=1, and one new page, esi-diff.
With ESI on, the commit and diff pages render the head, the header, the tabs, the commit info and the diff controls exactly as before. Where the diff would start, cgit writes
The VCL lives in contrib/varnish/cgit-esi.vcl. The interesting bits:
The fragment URL still names a repository, because cgit needs one to open the object store. Varnish ignores the repository. It can do that based on the assumption that a git commit, identified by the SHA, is identical in every repo. The first fork to render a certain commit renders it for all of them.
The demo runs on one well worn Linux box: an Intel i7-7700, four cores, 64 GB of memory. It’s publicly available as kernel.varnish.org. Since cgit is CGI, with a fork/exec for each request, I have a lighttpd instance to do the actual CGI. Varnish can’t do CGI on its own. In front of lighttpd sits varnishd. Note the built-in TLS support.
active.vcl is three lines: the version, a backend, and an include of the contrib VCL. I added a vcl_deliver that stamps X-Cache: hit or miss on every response for your viewing pleasure.
My first benchmark picked a 16 thousand line commit in git.git, measured the frame against the whole page, and reported an 84% saving. Whohoo! This was unfortunately an outliner. The cost of the typical commit is much, much lower. So I sampled 150 random git.git commits instead. Median change: 28 lines. Meh.
This section and the two after it are measured on my desktop, a Ryzen 9800X3D, not the old Intel-based server
|
median |
|
|---|---|
|
whole page, ESI off |
6.61 ms |
|
ESI frame only |
5.61 ms |
|
ESI fragment only |
1.42 ms |
The frame cost four times what the fragment cost. Caching the fragment across all 922 forks would save 24%, and a cold miss would be 7% worse than not splitting at all. Disappointing. Profiling the commit path found the cost immediately:
Ninety percent of the frame is one call, and it has nothing to do with the diff. Decorations are the "tag: v2.54.0" labels next to a commit subject. To produce them, git reads every ref in the repository. My first attempt to fix this was to delete the call, which saved ~nothing, because get_name_decoration in log-tree.c calls load_ref_decorations itself if nobody else has. The actual reason it is slow is better. Here is its per-ref callback:
For every annotated tag, it inflates the tag object to find out which commit it points at. git.git has 1008 of them. Meanwhile packed-refs already records the answer on a ^ line next to each tag, and the ref iterator hands it to the callback in ref->peeled_oid. The decoration code ignores it! Using it instead is about twenty lines, and shaves off 4 ms on every commit and log page.
The peel fix stopped git inflating every tag object. It did not stop it asking the object store what every tag was. Two questions survive per ref: what type is the ref target, and what type is the thing it peels to. Each one is a binary search through the pack index.
On git.git that index is 164 MB. On linux.git it is 389 MB covering 11.8 million objects, and a commit page walks 946 refs, so it pays about 1900 of those searches. Every request. In a fresh process, the page table for that index is built from scratch each time too. Seventy-eight percent of the frame, to draw "tag: v6.18" next to a subject line. This is where the CGI becomes cumbersome. Having a persistent web application that can cache this would have been useful.
Neither question needs asking. packed-refs only writes a peeled line for refs that point at a tag object, so if the ref backend handed us a peeled oid, the type is OBJ_TAG and we already know it. And the peeled object is only ever used as a key in the decoration table. Nothing ever reads its type, so lookup_unknown_object() gives you the same struct that a later lookup_commit() will fill in. A control run makes the size of it obvious: serve the same objects through a repository with one ref instead of 946 and the frame costs 1.7 ms.
git.git goes from 6.5 ms to 2.2 ms.
With the frame no longer dominated by decorations, the large diffs stood out again. Rendering a 1.7 MB diff took 100k writes averaging 18 bytes each. html_raw calls write(2) once per fragment of HTML, and cgit emits HTML in very small fragments. A 64 KB buffer takes that to 27 writes. Shaving off 100k system calls really helps.
Median per request, output to a pipe as under CGI, against git.git:
|
stock |
+peel |
+buffer |
|
|---|---|---|---|
|
commit page |
7.67 ms |
2.03 ms |
1.86 ms |
|
16k-line diff |
95.87 ms |
95.90 ms |
77.97 ms |
|
16k-line side-by-side diff |
181.19 ms |
181.89 ms |
93.18 ms |
The two patches are complementary and land on different pages. The peel fix targets fixed per-request cost on repositories with many tags. The buffer targets output volume, which is why the tree page gains nothing from the first and everything from the second.
If the page is entirely in cache, frame and fragment both, the demo box serves about 21,000 pages a second, which is 3 to 5 Gbit/s depending on how fat the commits are. TLS terminated, ESI assembled, gzip on the wire, on four ancient cores. The NIC is a gigabit, so on that workload the network runs out long before Varnish does.
A crawler never asks for the same URL twice, though. The fragment is caches and ready to go,; the frame is not. So the cost of a crawl is one cgit render per page. And cgit is still slow. About 20x slower than Varnish. Which is the useful conclusion, because cgit is fixable. The decoration work took us from 200 to 1,000 frames a second.
The fragment is the expensive half and it does not get cheaper. The decoration fixes do nothing for it, because a fragment has no subject line to decorate. It gets cheaper by being rendered once for all 922 forks instead of 922 times. With the frame down at 2 ms, parsing cgitrc is no longer free: 924 repository stanzas is 137 KB re-read on every CGI invocation, and it costs 1.3 ms. Nearly as much as rendering the page. That is a low hanging fruit if I ever saw one.
Frame rendering is embarrassingly parallel. A fork and exec per request, no shared state. So it scales with cores until memory bandwidth or the process rate gives out. Measured here: 250 frames/s per physical core after the fixes, 50 before. If we assume a 2026 core does about the same amount of work as a 2017 core, the amount of work should more or less scale with the number of cores. So a 64 core system should give us about 16k renders per second.
One surprising thing I learned. Forking on Linux is a lot cheaper today than it used to be. Forking is really, really fast. Exec'ing cgit costs about 0.33 ms. When a frame took 15 ms that was noise. Now that a frame takes 2 ms it is fifteen percent, and a resident cgit starts to look very appealing. Because a persistent process would stop re-faulting the 389 MB pack index on every request, it could keep lookup-tables various caches values, etc.
The patch and rawdiff pages need no ESI at all. They are plain text with no links, so they are already repository neutral, and hashing them on the object id is a VCL that didn’t involve cgit.
There are tons of things that can be done here. But I’m pretty certain one can make this system saturate a 10Gbps link, if you really want to. That should buy the team about 6 months until the next time it breaks.
Build cgit with the submodule, set enable-esi=1 and a virtual-root in cgitrc, include contrib/varnish/cgit-esi.vcl in your VCL, and put Varnish in front. The man page has a section called EDGE SIDE INCLUDES with the details, and tests/t0112-esi.sh shows exactly what the frames and fragments look like. The three performance patches in contrib/perf/ are independent of all of that and help whether or not you run a cache.
See the Varnish ESI demo built from git.kernel.org's repository list here: kernel.varnish.org
This setup was made and tested for Varnish Cache. It will likely run without modification on Vinyl Cache, you just need to set up TLS support somehow.
I'm at per.buer @ varnish-software.com if you're interested in discussing this topic further. Alternatively you can reach out to Varnish in general.