Rubén Romero

Varnish Community Manager
Find me on:

Recent Posts

Varnish Web Developer Wiki Highlights: Installing Varnish on Ubuntu

It doesn't require much introduction, as the idea is completely straightforward. As a part of the Varnish Wiki project, designed to help you do whatever you need to do with Varnish and share your knowledge and experience of putting Varnish into practice, we've put together a series of resources that are open to everyone.

This particular reference has been prepared for web developers to get you started with your Varnish installation and configuration on Ubuntu/UNIX. Ready? Let's get started.

Installing and configuring Varnish

The following text discusses how to configure your web server to use Varnish. Note that the installation is different for systemv and systemd. The following guide is for systemd as many Linux distributions are now adapting to the systemd init system.

Step 1 : Installing Varnish on Ubuntu/UNIX:

It is recommended that you install the Varnish package from its repository.

  • Start by grabbing the repository
  • Add the repository to the source list and save
sudo curl http://repo.varnish~cache.org/debian/GPG~key.txt | sudo apt~key add ~

sudo nano /etc/apt/sources.list

deb http://repo.varnish~cache.org/ubuntu/ trusty varnish~4.1
  • Run update and install
sudo apt~get update
sudo apt~get install varnish

Step 2: Configure Varnish

Varnish comes with two configuration files:

One with the starter parameter:

/etc/default/varnish

This file contains all the starter parameters.

The other is the default VCL file:

For systemd, the VCL file is directed in a different manner. It will be located in:

/etc/systemd/system/varnish.service

which will point to:

/etc/varnish/default.vcl

This default.vcl contains the default policies that the user includes. It also tells Varnish where to find the web content. However, there is a builtin.vcl that is always appended to the VCL you define/specify in this default.vcl.

  1. Modify Varnish config file
  • Open /etc/default/varnish in a text editor
  • You will see a code like the one below

DAEMON _OPTS="-a :80\
  -T localhost:6082
  -f /etc/varnish/default.vcl
  -s malloc,256m"
  -S /etc/varnish/secret

Description:

~T : refers to which port manages this.

~f : refers to the other configuration file containing all the default policies. If you plan to change the name of the default policy file, be sure to come here and change the default.vcl to the correct name.

~S : refers to the file containing private information, such as passwords, etc. also known as the shared-secret file.

~s : refers to the space Varnish Cache is allocated. 256m” is decided based on the current server’s RAM of 1GB.

  • Set the Varnish listen port to 80
  • Replace the -f line with -b 95.85.10.242:8080 as shown below
DAEMON_OPTS="-a :80 \
  -T localhost:6082 \
  -b 95.85.10.242:8081 \
  -S /etc/varnish/secret \
  -s malloc,256m"

These are all the configuration changes required in this file.

  1. Copy the default file named varnish.service

cp /lib/systemd/system/varnish.service /etc/systemd/system/

Edit /etc/systemd/system/varnish.service

Locate the line containing port 80 and change it to 8080

ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

Note that this file points to the /etc/varnish/default.vcl file.

  1. Now modify the default.vcl file

This file contains configuration that points to the content. This is by default set to serve at 8080 and points to host as localhost as shown below.

To minimally configure Varnish:

  • Back up default.vcl
cp /etc/varnish/default.vcl /etc/varnish/default.vcl.bak
  • Open /etc/varnish/default.vcl in a text editor.
  • Locate the following piece of code:
backend default {
       .host = “127.0.0.1”;
       .port = “80”;
}

The value of .host is localhost by default. It should be replaced with the fully qualified host name or IP address (typically a web server) and listen port of the Varnish backend or origin server; that is, the server providing the content Varnish will accelerate.

The value of .port should be replaced with the web server’s listening port, for example 8080 as shown below.

backend default {
       .host = “<your_webserver>”;
       .port = “8080”;
}

It is recommended that if changes are made to these files, they should be copied and renamed, because when Varnish updates, it will replace any changes made with the new default.vcl and Varnish files.

Varnish is now serving the client at port 80 and listening to the backend at port 8080.

This is when you can add ready-made VCL templates or recommended plugins for your web application (WordPress, Drupal, Magento2)

But before you add any new code you must understand VCL.

Step 3: Configure Apache2 to work with Varnish

Configure your web server to listen on a port other than the default port 80 because Varnish responds directly to incoming HTTP requests from the client on this port.

Varnish will communicate on a different port with your backend web servers. In the example above, it is port 8080.

In the sections that follow, we use port 8080 as an example, as shown above. If you have more than one backend, you can put them on another port, such as port 8081, 8082, etc.

To change the Apache listen port:

~ Open /etc/apache2/ports.conf in a text editor

~ Locate the listen directive

~ Change the value of the listen port to 8080 (you can use any available listen port)

~ Save your changes to ports.conf and exit the text editor

~ Also edit /etc/apache2/sites-available/000-default.conf

~ Change the VirtualHost port to 8080:

<VirtualHost 127.0.0.1:8080>

Setting up multiple backends (skip this section if you have one backend)

If you have more than one backend server, you need to add all these backends into the default.vcl file as backends and also define how and when each of them should be accessed.

Here is a simple example:

backend server1 {
        .host = server1;
        .port = "8080";

        #these are needed if you also want to do load balancing for your servers
        .max_connections = 250;
        .connect_timeout = 500s;
        # there are more parameters you can add here, check out the Reference.
}

backend server2 {
        .host = server2;
        .port = "8081";
}


sub vcl_rec {
  if (req.url ~ "^/server1") {
          set req.backend_hint = server1;
      } else {
          set req.backend_hint = server2;
      }
}

Step 4: Restart

It is always required that you restart all services once changes are made in configuration files.

service varnish restart
service apache2 restart

Step 5: Testing

Run

http -p Hh localhost

You can also use varnishtest to test your backend as shown below.

varnishtest "Apache as Backend"

varnish v1 -arg "-b 127.0.0.01:8080" -start

client c1 {
  txreq
  rxresp

  expect resp.http.server ~ "Apache"
  expect resp.http.via ~ "varnish"
} -run

Replace your backend ip-address and port number.

Step 6: Troubleshooting

If Varnish fails to start, try running it from the command line as follows:

varnishd ~d ~f /etc/varnish/default.vcl

This should display any error messages.

Step 7: The management interface

Varnish has a command line interface (CLI) to control any Varnish instance. It can be used to:

  • Reload VCL without restarting
  • Start/stop cache process
  • Change configuration parameters without restarting
  • View up-to-date documentation for parameters, etc.
  • Implement a list of management commands in the varnishadm (varnishadm establishes a connection to the Varnish deamon varnishd)
Read More

07/07/17 11:30
by Rubén Romero

Varnish Web Developer Wiki Highlights: Cache Invalidation in Varnish with Examples

 

At the risk of sounding repetitive, cache invalidation is routinely seen, thanks to a now-immortal statement from Phil Karlton, as one of computer science's two most difficult things. 

And it's true: due to the difficult nature of maintaining real-time, up-to-date cache coherency, cache invalidation is a challenge. It's one we've been working on for just about as long as Varnish has existed.

To delve further into how exactly the different components of cache invalidation work with Varnish, here's an excerpt from the Varnish Wiki on this very subject. 

As usual, I'd like to remind you that the Wiki isn't just ours - it's also yours. I encourage you not only to take a look at it but also to contribute to it based on your own Varnish experience.

Read More

22/06/17 12:44
by Rubén Romero

Varnish Web Developer Wiki Highlights: Running Varnish on your site

Once you have installed Varnish, actually running Varnish is another ballgame. We put together this brief overview of Varnish tips and tricks for helping you get off to a running start when you implement and use Varnish. It's not magic, but once you master the basics and move into more advanced Varnish territory, it will start to seem like it.

Read More

24/05/17 12:32
by Rubén Romero

Varnish Web Developer Wiki Highlights: Magento2 Varnished

A few years ago I wrote the "Doing the right thing: Magento + Varnish Cache" blog post which has been very popular and I have planned to write a new one for a while. I didn't. Instead we added a " Varnish with Magento 2" tutorials section in the Varnish Web Developer Wiki, which includes many resources such as this Step by Step Guide to make Magento .2 and Varnish work like a charm.

It's a wiki, meaning that we hope and expect that you will help us to expand it and keep it up to date. I'm publishing it here in the blog for reference. In this tutorial, we give the step-by-step process that will help you install and configure Varnish to take your Magento2 e-commerce site to the next level. So read on:

Magento is a popular and powerful e-commerce platform for marketing, catalog management and search engine optimization. Magento provides online merchants with a flexible shopping cart system and control over a user-friendly WUI with content and various functionality for online users.

Magento is popular because it's got all this good stuff going for it, but there are some important considerations when implementing a Magento-based e-commerce site. It needs a bit of help in the acceleration department. This is where Varnish comes in; Varnish is a great complement to Magento sites because it adds significant speed and the potential for a real reduction in maintenance costs. 

Magento2 is a PHP-based e-commerce platform. There are two editions:

  • Community edition (CE)
  • Enterprise edition (EE)

Whichever works best for you, you can get started with a performance increase for your website right away!

Let’s get started!

This article assumes that you have a running instance of Magento2 and that you have administrator rights for said instance, both at the OS and application level. We have tested this using Ubuntu LTS 16.04, Varnish Cache 4.1 and Magento2.

If you need guidance on the installation of Magento, please visit the Magento-Site

1. Installing Varnish

In case you also do not have Varnish, you will need to follow the instructional section on how to Install Varnish before we can continue.

2. How to place Magento2 behind Varnish

Here we discuss how to configure your Magento2 behind Varnish. The Magento2 admin states that the built-in application cache is not recommended for production use, but that does not mean that Varnish is already configured. Varnish needs to be installed and the configuration file suitably configured and deployed.

We assume that you have Magento2 installed and running on your backend servers and that your server is a Debian-based Linux server.

Configuring Magento

  1. Visit the Magento2 admin page and go to:
  • > Stores
  • > Advanced
  • > System
  • > Full Page Cache

Here switch caching application to Varnish

This is what the Magento2 mdmin page should look like:

Image courtesy: Magento Site

  1. Configuring for Varnish
  • Under the Additional section, find a button for exporting the ready-made configuration file for Varnish 3 or 4. We recommend you use Varnish 4, as earlier versions are no longer supported.
Click on::
-> Export VCL for Varnish 4 This is usually named varnish.vcl

Place the file in a Varnish folder for configuration (any place that is safe for you).

  • Next, to add Varnish repository, follow the instructions at Install Varnish
  1. Testing your setup

Now check if your services are up and running:

$ ps -e | grep 'apache2|varnish'

If you receive outputs like the ones below:

1143 ?        00:00:03 varnishd
1148 ?        00:00:17 varnishd
1366 ?        00:02:02 varnishlog
1591 ?        00:00:01 apache2
11743 ?       00:00:10 apache2
11744 ?       00:00:10 apache2

Congratulations! You have your services running on the backend.

Now we need to check the Magento2 frontend: As you may have already noticed above, there is a varnishlog process running as well.

3. Restarting services after making changes

sudo systemctl restart varnish.service

sudo systemctl restart apache2.service

4. Basic caching

Varnish does not cache cookies or their headers. But some cookies are marked as safe by the Magento site. Therefore it is recommended to remove or ignore these cookies so that Varnish can cache anything.

An example, such as the following, will help to unset/remove unwanted cookies.

Update your default.vcl with this code or a similar code of your choice.

Under the vcl_recv add the following code.

Warning: Make sure to add the code below the default code given for vcl_recv

sub vcl_recv {
  if (req.http.cookie) {
    set req.http.cookie = ";" + req.http.cookie;
    set req.http.cookie = regsuball(req.http.cookie, "; +", ";");
    set req.http.cookie = regsuball(req.http.cookie, ";(COOKIE1|COOKIE2|COOKIE3)=", "; \1=");
    set req.http.cookie = regsuball(req.http.cookie, ";[^ ][^;]*", "");
    set req.http.cookie = regsuball(req.http.cookie, "^[; ]+|[; ]+$", "");

    if (req.http.cookie == "") {
      remove req.http.cookie;
    }
  }

5. Excluding certain URLs

Not all URLs should be cached. Especially not in sites that deal with personal information such as credit cards or financial details.

if (req.url ~ "magento_admin|magento_login") {

return (pass);

}

6. Extended caching

There is a subroutine called vcl_fetch which is by default set to 120 seconds as can be seen. You can extend this caching value by:

sub vcl_fetch {
       set beresp.ttl = 5s;
}

This sets the TTL to 5 seconds, making Varnish pick up changes every 5 seconds. Add this subroutine right below the backend default.

However, there is a downside to short TTL values in that they increase the load not only on the backend servers but also on frontend servers. Of course this gives you a better control over your cache, but it also increases overheads, such as network traffic, slower response times, which diminish the whole purpose of Varnish.

Decreasing TTL values is not a good solution for high-traffic servers. Varnish has a better solution for that.

7. Specific TTL-based caching

Varnish creates a TTL value for every object in the cache. The most effective way of increasing a website’s hit ratio is to increase the time-to-live (TTL) of the objects.

8. Go further

If you are interested in Varnish, you can always give Varnish Plus a go with a free trial. You can capture real-time traffic statistics, create a paywall for premium content, simultaneously work on administration across all Varnish servers, record relationships between web pages for easy content maintenance, detect devices used for browsing, accelerate APIs and more.

Check out the links below to take your Varnish-Magento site to new performance heights.

9. For more guidance

You can always refer to the Configure and Use Varnish at the Magento site.

To see the guide on installing and configuring Magento with Varnish on web servers, please look at here.

If you are interested in trying out an installation, download Marko’s Vagrant Box marko_magento2github. His installation used nginx with Varnish and Magento. You can also read more about in Marko’s blog post about placing Magento2 behind Varnish marko_magento2post.

Read More

04/04/17 10:49
by Rubén Romero

All things Varnish related

The Varnish blog is where the our team writes about all things related to Varnish Cache and Varnish Software...or simply vents.

SUBSCRIBE TO OUR BLOG

Recent Posts

Posts by Topic

see all