Cache Rest Api’s In Varnish

8 min readAug 24, 2020

Oh god, I have done all the optimization in code, joining the tables and also have followed standards in writing the code. But still why my api’s are so slow and can’t do anything about it ?

Wait, you haven’t tried the one thing in your api’s ? Caching.

Caching is the process of storing data in a cache. A cache is a temporary storage area. For example, the files you automatically request by looking at a Web page are stored on your hard disk in a cache subdirectory under the directory for your browser.

There are various types of standard caching available like varnish, memcache. Everyone has their own specific and distinctive feature. Today I am going to talk about the one i have used and it works like charm is Varnish Cache.

Varnish Cache : A brief introduction

If you will search for the meaning of varnish on the google, you will find something like this.

To cover with varnish.
To give a smooth and glossy finish to.
To give a deceptively attractive appearance to; gloss over.

It works like as name suggests, it smoothens the api response time 😅

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents.Varnish Cache is really, really fast. It typically speeds up delivery with a factor of 300–1000x, depending on your architecture.

One of the key features of Varnish Cache, in addition to its performance, is the flexibility of its configuration language, VCL. VCL enables you to write policies on how incoming requests should be handled. In such a policy you can decide what content you want to serve, from where you want to get the content and how the request or response should be altered.

Oh Man ! You talk too much, I can search more on https://varnish-cache.org/intro, Just continue with the installation and its application. Cool.

Varnish Installation

Varnish serves as a HTTP reverse proxy. You can install it alongwith the popular servers like nginx or apache. I am going to install with nginx.

Installation with nginx on ubuntu

Installation of varnish with the nginx server on ubuntu :STEP 1 : Update the System
--------------------------$ sudo apt-get update && apt-get upgradeSTEP 2 : Install the nginx webserver
------------------------------------$ sudo apt install nginx -ySTEP 3 : Start the nginx server
---------------------------------$ sudo systemctl start nginx 
$ sudo systemctl status nginxSTEP 4 : Install the varnish server
-----------------------------------$ sudo apt-get install varnishSTEP 5 : Start the varnish server
--------------------------------$ systemctl start varnish
$ systemctl status varnishSTEP 6 : Configure the nginx to listen on the port 8080
---------------------------------------------------------To configure Nginx to listen to port 8080, open the configuration file as shown with$ vim /etc/nginx/sites-available/defaultlisten 8080 default_server;
listen [::]:8080 default_server;To test the nginx configuration and restart the server :$ nginx -t
$ systemctl restart nginx
$ netstat -plntu
    
STEP 7 : Restart the nginx server
----------------------------------------------------------$ sudo systemctl restart nginxSTEP 8 : Setting up the varnish server to listen on port 80
----------------------------------------------------------$ vim /etc/default/varnishScroll and locate the attribute ‘DAEMON_OPTS’. Be sure to change the port from 6081 to port 80DAEMON _OPTS="-a :80\
      -T localhost:6082
      -f /etc/varnish/default.vcl
      -s malloc,256m"
      -S /etc/varnish/secret-T : refers to which port manages this.-f : refers to the other configuration file containing all the default policies. If you plan to change the name of the default policy file, be sure to come here and change the default.vcl to the correct name.-S : refers to the file containing private information, such as passwords, etc. also known as the shared-secret file.-s : refers to the space Varnish Cache is allocated. 256m” is decided based on the current server’s RAM of 1GB.STEP 9 : Check whether the port in the default varnish configuration is 8080
--------------------------------------------------------------------$ vim /etc/varnish/default.vclbackend default {
    .host = “127.0.0.1”;
    .port = “8080”;
}STEP 10 : Change the varnish server port to 80.
-----------------------------------------------$ vim /lib/systemd/system/varnish.serviceLocate the line like below and change the port to 80.ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256mSTEP 11 : Restart the varnish server, daemon server and the nginx server.
--------------------------------------------------------------------------$ systemctl restart nginx
$ systemctl daemon-reload
$ systemctl restart varnishSTEP 12 : To check if the varnish is installed.
----------------------------------------------You can see the header in the curl.$ curl -I server_IP
$ curl -I 127.0.0.1

Installation with apache2 on ubuntu

For installation alongwith apache2, all the same boring steps of installation except Step 6 which goes as follows :

STEP 6 : Configure the apache2 to listen on the port 8080
 ---------------------------------------------------------
 
     To configure Apache to listen to port 8080, open the configuration file as shown with
 
     $ vim /etc/apache2/ports.conf
 
     Change the listen to 8080 server
 
     To configure the apache default server to port 8080, open the configuration file as shown with
 
     $ vim /etc/apache2/sites-enabled/000-default.conf
 
     <VirtualHost *:8080> instead of <VirtualHost *:80>
 
     Now our apache2 default server has started listening on the port 8080 instead of port 80.

Confirmation OF HTTP Request Via Varnish

You will find the X-varnish and Via response header set as the response header.

sagar@root:~$ curl -I 127.0.0.1
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Mon, 24 Aug 2020 09:02:15 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 08 Jul 2020 18:13:54 GMT
ETag: "5f060ce2-264"
Accept-Ranges: bytes
X-Varnish: 163856
Age: 0
Via: 1.1 varnish (Varnish/5.2)
X-Cache: MISS
Connection: keep-alive

Note: You can see all the configuration files in the link below for verification during installation of varnish.

Varnish Application

Oh man !! I have gone through a lot. For god sake, Please tell me how to apply it.

Don’t worry bro 😏. Here is configuration files which is most important to you. It’s native place on ubuntu is /etc/varnish/default.vcl and looks like this.

vcl basic configuration

Pretty Scary Huh ??

“Everything which is unkown is scary.” — By Me. LOL.

Let’s know these configurations little bit and makes things easy.

std

The “std” vmod is one you get with Varnish, it will always be there and we will put “boutique” functions like toupper, to lower and many more.

2. backend

This section helps in identification of the backend server and host, there is a lot more configuration like healthcheck for checking of more than one varnish server and many more.

Note -The built in subroutines all have names beginning with vcl_, which is reserved. Built-in functions in varnish do not receive arguments.

Aahaa, I get it all surprisingly, vcl_backend_response, vcl_deliver, vcl_receive are the built-in varnish functions. Let’s see them one by one 😸.

3) vcl_receive

It is the function which is invoked before the request is being served from either varnish or server. It is mostly used for the clean-up and rewrite process. You can set or remove some cookies header or do some kind of preprocessing to the requests according to the need.

It is the place you can restrict or unrestrict any requests to be served by varnish based on some desired configuration like in here based on the request api’s. Hash is being returned for the cached api’s and pass is being returned otherwise.

Note : “req” is the object you get in vcl_receive phase and it contains all the information about the requests like request method, url, origin, accept type.

4. vcl_backend_response

It is the function which is being invoked just after we have read the response from the backend. Here you clean the response headers, removing silly Set-Cookie headers and other mistakes your backend does. It is basically the refining of the response being given by the backend server (apache or nginx) server.

Note : “bres” is the response object you receive which contains the response given by the backend server like body, headers etc. “req” in vcl_receive is called as “breq” in this function.

5. vcl_deliver

It is function which is invoked after we have all the pieces we need, and are about to send the response to the client.

Wait a minute 😕 !! Is vcl_backend_response will not serve this purpose ? Why to learn about vcl_deliver, man ?

After, getting response from the backend server (apache or nginx), vcl_backend_response is invoked and after processing from this function, the response is being processed by the varnish server and that’s why we need vcl_deliver to perform final modification to response after processing from the varnish server. 😄

Summing It Together

Magic starts now. 😈

Any requests that come firstly goes into the varnish server which is being served by default.vcl configuration file.

Varnish server always listens to port 80 and it is being cofigured to act as a reverse proxy for localhost and port 8080.

In vcl_recv section, the request is checked if it PURGE request and in cookie if “bookstorecache” is set, it will purge all the varnish cache. If the request method is not PURGE, then it checks whether the url follows the regex expression of “readbook”, “writebook”, “deletebook”, “addbook” and checks whether the accept type is not xml (because we have to cache it only in the case of application/json content type) and if both condition satisfies, it returns from varnish cache server.

After getting the response from backend, it again checks for various url pattern (“readbook”, “writebook”, “deletebook”, “addbook”) and the status is 200, then it unset the response set-cookie and set the “ttl” and “Cache-Control” as shown in the code. TTL (time to live) is the amount of time or “hops” that a packet is set to exist inside a network before being discarded by a router. Here ttl is set to 90 seconds.

Finally, the code comes into the deliver section and checks for hits with the obj, if the request is being served by the varnish server. “X-Cache” header is being set to “HIT” and “X-Cache-Hits” is set to no of times. Otherwise, “X-Cache” is being set to “MISS”.

Results 😍

Without Varnish : (~736 ms)

With Varnish : (~42 ms)

You can decide which is better 736 ms or 42 ms ? 😏

Important Warnings And Precautions ❎

“A sword must be handle properly and used it at appropriate places” — Anonymous

Don’t use varnish for POST, PATCH, PUT requests. Mostly GET is used.
Set the ttl time according to the need and feasibility. Don’t set it too low (otherwise no need use of varnish) or too high (the latest content will not be updated in varnish cache).

Important Commands And Reference 🎫

If Varnish fails to start, try running it from the command line as follows:
 
 $ varnishd ~d ~f /etc/varnish/default.vcl
 
 To check all the process whether the varnish is running on the port 80 and nginx or apache2 is running on port 8080 :
 
 $ netstat -plntu
 
 To check the varnish log, we can use
 
 $ varnishlog
 
 To check the histogram of varnish hit, we can use the following command
 
 $ varnishhist
 
 To list all the logs of purge request only, we can use the following command
 
 $ varnishlog -g request -q 'ReqMethod eq "PURGE"'
 
 To restart the varnish server
 
 $ service varnish restart
 
 To reload the varnish server
 
 $ service varnish reload
 
 To get the statistics of the varnish
 
 $ varnishstat