A really effective way to speed up a website is to add some form of caching layer in front of it. If your web server doesn’t have to keep generating the same web pages over and over, odds are things will be a lot faster for your site visitors. This is where Varnish comes in.
According to our research, 5.2% of the world’s top 10,000 websites are currently using Varnish, a popular open source HTTP accelerator (also called a reverse proxy) that acts as a caching layer between a website and its visitors.
Or as the Varnish website explains it:
Varnish is a web application accelerator. You install it in front of your web application and it will speed it up significantly.
According to the documentation on its website, Varnish can speed up delivery anywhere between 300 to 1,000 times.
Sites that use Varnish
There are a lot of people and companies who want to speed up their websites. Perhaps this is why Varnish was downloaded over 25,000 times the last two months of 2011.
Notable sites using Varnish include Wired, Slideshare, Zappos, SoundCloud, Weather.com, Business Insider, Answers.com, Posterous, Urban Dictionary, Macrumors, DynDNS, OpenDNS, Lonely Planet, Technorati, Thinkgeek, Economist.com, and obviously quite a few more.
Varnish has been around since 2006, and has received quite a following in that time.
A Q&A with the creator of Varnish
We spoke with Poul-Henning Kamp, the creator of Varnish, to get some additional insights into the project. He’s also famous for his contributions to FreeBSD, so the man has some serious credentials in the open source developer community.
Pingdom: What were your goals when you started the Varnish project, and do you think you’ve met them?
PHK: Back in 2005-6 we had a very concrete goal of “being much better than Squid on the server-side” and that goal has clearly been obliterated many times over.
I had a personal goal of trying to inspire (by example) people to program for the kind of kernels and hardware we have today, rather than for VAX/11 computers from 1980. Some people here and there have been inspired, but there is a lot of way to go yet.
Pingdom: You say you wanted to inspire people to program for the kind of kernels and hardware that we have today. Could you give some examples of what you mean by this?
PHK: For instance modern kernels have Virtual Memory systems, which means
that the old
“io” – “cpu” – “memory”
model of programming really isn’t correct any more, memory instead becomes a cache of views of (file-)objects. I wrote an article about it here.
Pingdom: What have been the major challenges with developing an HTTP accelerator?
PHK: Probably that I don’t run a big website myself, so I have to rely entirely on other people’s observations and feedback. It also doesn’t help that HTTP is a pile of badly thought out quick hacks.
Pingdom: Your statement that “HTTP is a pile of badly thought out quick hacks,” could you elaborate on that?
PHK: That is a long story, but basically nobody ever sat down and thought “how do we make this protocol work the next 20 years”, they all said “if I add this nifty extra header I can …”
For instance cookies came about because HTTP does not have anything remotely looking like a “session” concept, but cookies were defined without any thought about caching in the network, and have resulted in the server’s session related data being stored on the client which is just totally wrong, not to mention all the bandwidth it costs.
HTTP/2.0 should be able to last 20 years, which means that we will see traffic densities of 1 Tb/sec on a single box in the network. The current protocol is very badly suited for that, with all its badly thought out field/header-termination criteria, and proposals like SPDY are actually even worse in that respect.
Pingdom: How many people work with and contribute to Varnish today?
PHK: In total I think we approach a dozen full-time now.
Pingdom: What’s next for Varnish? What lies ahead?
PHK: Next up is a release that truly does streaming fetch/delivery and conditional backend requests. For reasons of real-world interference, that has been dragging on for ever.
On the long range radar the IETF is aiming for a HTTP/2.0 protocol and while that is almost entirely up in the air still, some patterns are emerging, for instance tagged pipelining, and I have started to rearrange our source code for that.
Big thanks to Poul-Henning for taking the time to talk with us.
Final words
Site performance matters. Few things are as frustrating for site visitors as having to wait while a web page loads. But you already know this. All web users do.
Performance optimization is a large and complex subject, but the next time you’re looking over your website’s performance, perhaps you should try Varnish as part of your optimization strategy. It could help you boost site performance significantly.
After all, what webmaster wouldn’t like to make his site 300 times faster?
How we got the Varnish usage stats: Sites that use Varnish normally return the X-Varnish HTTP response header when you access the site. We used this as the indicator if a site is using Varnish or not, and scanned the response headers for the top 10,000 websites in the world according to Alexa.
