Getting Started with Varnish Edge Side Includes and WordPress

There are a lot of cases on blogs where once you write the post, the cache for that page doesn’t really have to be updated all that often.  Comments can be powered by Disqus so you don’t need to bust the cache every time someone comments. If you make a change to a post, it’s page and any pages that display it (home, category, etc) should be updated automatically anyway; wp-varnish is the best plugin for that btw.

Now, the sidebar. That’s where it can get tricky.  On a lot of the content sites I work on, we can cache most pages for days, the backend doesn’t need to keep generating them, the content just lives in varnish. Except for the sidebar.  This could have things like “Popular Posts” or “New Posts”. Things that are going to change outside of the context of the post you are currently looking at.

This is where Edge Side Includes (ESI) comes in.  For a while, Akamai was the only way to get this behavior, and they are quite expensive. Don’t get me wrong, they are amazing, but you have to be getting serious traffic to need them. Varnish is one of the best caching solutions you can use, especially for WordPress. If you are unsure where to start with Varnish and wordpress, this article is a great starting point. I’v also bookmarked some other articles here.

ESI was introduced in 2.1, and really fleshed out in 3.0 and can be thought of as fragment caching for Varnish. It’s really easy to fix the example I gave above: we want to be able to cache most of the content for 24 hours but refresh just the sidebar every 10 mins.  Varnish will do this processing for us, keeping the rest of the page cached, and splicing in the sidebar (and having your webserver process just the sidebar) when it needs to. Most of what I’m going to do below is in the ESI documentation.

We’re going to assume that our theme has a dynamic sidebar called “Sidebar” and that there are cases where we want ESI to be active (production) and some where wordpress should just behave normally (dev).  This is what we have before we start thinking about Varnish:

The two varnish tags we are going to use are an esi comment, and an esi:remove tag. These are the switch that will process the page properly whether you have ESI or not. esi:remove will ignore everything between them if ESI is enabled. This is where we put our “normal processing” code. If ESI is not active, this code will run as normal. Inside the esi comment we put the code that we want to run if ESI is enabled. If it is not, the code will be ignored.

Now our sidebar display is ESI ready, but we need to give Varnish an endpoint to hit so it can generate the fragment of the sidebar. So lets create esihandler.php:

Lastly, we’ll have to tell Varnish to cache for different timeframes. A full Varnish config is out of the scope of this post, see the links above for more info. What we need is in the sub vcl_fetch call. Add this at the top:

And you should be all set. When developing you can use smaller timeframes (10 seconds, and 1min). Drop some logging in to see it in action.

Let me know of any questions in the comments

  • Paul Thomson

    Great article Tim, thanks for detailed instructions…

    I’m trying to get just one widget to be handled via ESI. Is this possible? I’m struggling to find the best way to go about doing it. Any help would be greatly appreciated.


    • Tim Broder

      It’s possible. You would have to copy the output of the widget to an esi handler file and then put the esi tags in the widget output instead of the sidebar

      • Paul Thomson

        Legend Tim thanks I got it working, check it out on (it’s the widget on the right hand side labelled: My Gear Reviews)…

        Take care,

        • Tim Broder

          awesome! glad you got it working

  • NudeWeb

    Hi Tim

    I’m a bit of noob with varnish. I added the o=above lines to my default.vcl file but it produced the following error when restarting, any help much appreciated.

    Message from VCC-compiler:

    ‘beresp.ttl’: cannot be set in method ‘vcl_recv’.

    At: (‘input’ Line 35 Pos 9)

    set beresp.ttl = 10m;


    Running VCC-compiler failed, exit 1

    • Tim Broder

      What version of varnish are you using? Off the top of my head it sounds like it’s 2.x instead of 3

  • littleguy

    Great post! A question:

    When the cache for the sidebar expires, does the next user visiting the site going to have to wait for esihandler.php to return data?

    Or, is it possible to configure Varnish to send that user old data, and
    then subsequent users will get the new data. That way, everyones pageload is fast.

  • hamiseady

    There’s nothing in Varnish itself (that I know of) that could do that. You are looking to instantly re-warm the storage cache when it ends.

    Spybubble Gratis

  • Pingback: Varnish ESI not displaying content - HelpDesk