Why?

I was reading a few interesting posts about graphite. When I tried to install it however, I couldn’t find anything that really covered all the steps. Some covered it well for Apache, others covered Nginx, but had steps missing or assumed the reader knows about them etc.

I’m a big fan of fabric, and try to do all deployments and installations using it. This way I can re-run the process, and also better document what needs to be done. So instead of writing another guide, I created this fabric script.

Requirements

• Workstation running python (version 2.7 recommended). All platforms should be supported.
• Fabric – can be installed via pip install fabric or easy_install fabric
• a new VPS/Dedicated server running a Debian-based distribution (Debian, Ubuntu etc)

Target Host

Best to execute this on a clean virtual machine running Debian 6 (Squeeze).
Also tested successfully on Ubuntu 12.04 VPS.

Installation Instructions

download the project zip file, or clone the project from github.

run fab graphite_install -H root@{hostname}
(hostname should be the name of a virtual server you’re installing onto)

It might prompt you for the root password on the host you are trying to instal onto.

You can use it with a user other than root, as long as this user can sudo.

During the installation, you would be asked to set up the django superuser account. You might want to create an account, but it’s not strictly necessary. If you answer no, the installation will still work fine.

After Installation

Simply open your browser and go to http://[your-hostname]/graphite/
It should be up and running.

Of course there’s a lot more configuration to be done, but at the very least you should have a working environment to play with Graphite.

Thanks

Thanks to the authors of these online guides and resources who provided very useful information that I stitched together into this fabric script, and others who provided inspiration about Graphite in General:

• Graphite Docs
• frl1nuX – Graphite and Nginx
• Agile Testing – Installing and configuring Graphite
• Corey Goldberg – Installing Graphite 0.9.9 on Ubuntu 12.04 LTS
• Tom Purl – Installing Graphite on Ubuntu 10.4 LTS

Although not installed with this fabric script, I’d love to try these some time:

• Graphene
• GDash
• Logster

DISCLAIMER

Please try this at your own risk. Please run this only with a newly installed host that you can easily throw away!
I tested it with both Debian 6 and Ubuntu 12.04 successfully. However, you may experience different results.

Help and Contribution

I’d be happy to try to help if I can, but given the complexity of linux-based operating-systems, and my limited time, I might not be able to know why a certain operation fails or an error is generated. Feel free to fork the project on github for your own special requirements or needs.

That’s one of the reasons I am creating bootstrap scripts that completely automate a server (re)build process. Given an IP address and root password, the script should connect to the instance, install all necessary packages, pull the code from the repository, initialize the database, configure the web server and get the server ready for restore of user-data.

I’m primarily using fabric for automating this process, and use a standard operating system across different cloud providers. This allows a fairly consistent deployments across different providers. This also means the architecture is not dependent on a single provider, which in my opinion gives a huge benefit. Not only can my architecture run on different data centres or geographic locations, but I can also be flxeible in the choice of hosting providers.

All that aside however, building and refining this bootstrapping process allowed me to run it across different cloud providers, namely: Rackspace, Linode, and EC2. Whilst running the bootrstrapping process many times, I thought it might be a great opportunity to compare performance of those providers side-by-side. My bootstrap process runs the same commands in order, and covers quite a variety of operations. This should give an interesting indication on how each of the cloud providers performs.

Tested platforms

The tests were carried out using the default Debian 6 Squeeze on the lowest-end cloud instances on all three providers:

• Rackspace 256Mb and 512Mb – using the London data centre.
• Linode 512 – using the London data centre.
• EC2 micro instance (EBS volume) – using the Ireland data centre.

Bootstrap process

The bootstrap process executes the following tasks:

• apt-get update && apt-get upgrade and installing a list of prerequisite packages
• Installing Postgresql from backports
• Downloading, compiling and installing ruby and sphinx from source
• Setting up SSH keys
• Pulling code from a remote git repository
• Creating a couple of small (empty) databases and user accounts
• Tweaking some configuration files
• Performing bundle install on a rails project
• Performing rake tasks to set the database schema and seed the database

These are relatively I/O intensive operations, but also involve CPU tasks (compiling code) and network access (downloading sources and packages), so should provide a reasonable benchmark for comparing the performance of those cloud providers.

Results

These highly-unscientific results are quite basic. No fancy charts or anything. All I measured was how long the entire bootstrap operation was taking on each of the cloud providers.

• Rackspace 256: 1269 seconds (~21 minutes)
• Rackspace 512: 1144 seconds (~19 minutes)
• Linode 512: 1053 seconds (~17.5 minutes)
• EC2 micro: 4090 seconds (1 hour and 8 minutes!!??)

Linode seems to be the winner, running around 20% faster than Rackspace 256 and 8% faster than rackspace 512. What’s much more surprising however (for me anyway), is how slow EC2 is in comparison, running 378% slower than Linode… I am guessing this is down to EBS storage. Quite a big performance hit for persistent storage though.

Spoiler – Conclusion

So I know there isn’t such a thing as too much cash, but does the same apply to cache?

The results were rather disappointing for me. It turns out that my existing configuration of W3TC was already closely matching that of a much more complex set-up involving two or three proxy layers involving varnish and nginx. The W3TC disk-enhanced page caching is simple but powerful enough to give nearly the same performance. How? The basic principle is that it creates a static version of every page, and hooks the webserver (Apache or Nginx) to serve this file directly if it exists. On most Linux platforms, file-caching is powerful enough to serve those files from memory. This means that it does in fact match the performance characteristics of an in-memory cache. Simple solutions are not necessarily weaker. With 20% of the effort, I reached 80% of my performance goal. In actual fact, I believe the results are more like 10/90… More on that (and some caveats) at the end.

Hosting platform

Unlike most guides and optimization benchmarks, I decided to try this “on a budget”. Not on a high-performance dedicated server, not even on a VPS, but rather on a shared hosting account with webfaction. As shared hosting providers go, webfaction is quite unique in that it allows, perhaps even encourages you, to compile your own tools and run them. It doesn’t give you as much freedom as a VPS, but it’s actually good enough. This post doubles-up as a mini-guide on how to set-up php-fpm, nginx and varnish specifically on webfaction (unlike most guides, which assume you have root access and can use some package manager).

Original Setup

The baseline was a (real!) wordpress site running on webfaction, and already using W3TC with disk-enhanced page caching using .htaccess rules. W3TC is clever enough to manage those rewrite rules for you and also to detect whether your instance is running behind Apache or Nginx. My initial thinking was that since those static pages generated by W3TC are served by Apache, and are saved on-disk (rather than in-memory), I could benefit by introducing some in-memory caching (Varnish), and remove Apache out of the equation. It’s important to note that webfaction already use nginx as a front-end proxy. However, with a wordpress app, they forward all requests via Apache.

BASELINE:

{Internet} -> Webfaction Nginx -> Webfaction Apache -> W3TC / WordPress

This is how the baseline setup looks like (pretty, isn’t it?)

Nginx Setup

The first variation was to install my own nginx instance, and serve the wordpress pages via it, instead of Apache. In order to also serve php pages effectively, I opted for php-fpm, which seems like the most recommended option with nginx. php-fpm (fastcgi process manager) means you need to run a mini server that listens to requests to serve php files… Nginx uses it when it needs to serve a php page. I include some installation instructions for php-fpm here too. I couldn’t find any guide online on how to do it with webfaction.

NGINX:

{Internet} -> Webfaction Nginx -> My Nginx / php-fpm -> W3TC / WordPress

and this is what the Nginx setup will look like (once we install it, it’s not THAT easy)

Installing Nginx on webfaction

The easiest option to run your own nginx instance on webfaction is to use the ready-made passenger application. This is usually used to host a ruby-on-rails app, but there’s nothing stopping us from removing all the passenger stuff, and just using it as an nginx instance. In the webfaction control panel go to Applications, Add a new app, lets call it engine, and then choose Passenger for app category and then Passenger 3.0.11 (nginx 1.0.10/Ruby 1.9.3) in the App type. This will install it under ~/webapps/engine.

Installing PHP-FPM

Next up is getting php-fpm compiled, installed and running. This is done by compiling php with the right option to include php-fpm. SSH to your webfaction account and run these commands

    mkdir ~/src
    cd ~/src
    wget http://www.php.net/get/php-5.3.10.tar.gz/from/sg2.php.net/mirror
    tar -zxvf php-5.3.10.tar.gz
    cd php-5.3.10
    ./configure --prefix=$HOME --with-pdo-mysql --with-pdo-pgsql=/usr/pgsql-9.1 --enable-fpm --enable-bcmath --enable-calendar --enable-exif --enable-ftp --enable-mbstring --enable-soap --enable-zip --with-curl --with-freetype-dir --with-gd --with-gettext --with-gmp --with-iconv --with-jpeg-dir --with-kerberos --with-mhash --with-mysql --with-mysqli --with-openssl --with-pgsql=/usr/pgsql-9.1 --with-png-dir --with-regex --with-xmlrpc --with-xsl --with-zlib-dir --without-pear --enable-sockets --enable-intl --with-mysql-sock=/var/lib/mysql/mysql.sock
    make
    make install
    

We should now have php-fpm installed under ~/sbin and a default configuration file created in ~/etc/php-fpm.conf.default.

PHP-FPM configuration

Follow these steps to copy the configuration file and creating a folder where our Unix socket will live

    cp ~/etc/php-fpm.conf.default ~/etc/php-fpm.conf
    mkdir -p ~/var/spool
    

Then edit our php-fpm.conf file. We only need to replace the line that says listen = 127.0.0.1:9000 with

    listen = /home/WEBFACTION_USER/var/spool/phpfpm.sock
    

Replace the WEBFACTION_USER with your webfaction username

Now we can launch our php-fpm and it will listen on the unix socket. To launch it, simply run

    ~/sbin/php-fpm
    

Note that if you plan to use this for your server you will need to create a cron job that checks whether it’s running and launches it. Follow this webfaction guide about custom apps for more info.

One last thing is to create the fastcgi_params parameters file required by nginx to the nginx conf folder (~/webapps/engine/nginx/conf/fastcgi_params)

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

Nginx Configuration

This part was getting a little tricky, since I wasn’t sure which guide to follow for configuring nginx in the best way for using with W3TC. It also seemed to me that all the online guides miss an important part. W3TC actually generates a nginx configuration for you. This is not a complete configuration, but you need to make sure it is included within your nginx.conf in order to really get the most out of the W3TC plugin.

First step was to create a folder where we’ll get our W3TC-generated nginx.conf file into

    mkdir -p ~/webapps/engine/nginx/conf/sites-enabled
    

Then edit your ~/webapps/engine/nginx/conf/nginx.conf file, so it looks something like this:

    worker_processes  1;

    events {
        worker_connections  1024;
    }

    http {
        access_log  /home/WEBFACTION_USER/logs/user/access_engine.log  combined;
        error_log   /home/WEBFACTION_USER/logs/user/error_engine.log   crit;

        include         mime.types;
        sendfile        on;
        tcp_nodelay on;
        tcp_nopush on;
        port_in_redirect off;

        server {
            listen             99999; # make sure the port is the same as configured on your webfaction `engine` app
            server_name        localhost;
            root               /home/WEBFACTION_USER/webapps/wptst;  # point this to your wordpress folder
            index              index.php;
            include /home/WEBFACTION_USER/webapps/engine/nginx/conf/sites-enabled/*;

            location / {
                try_files $uri $uri/ /index.php?q=$uri&$args;
            }
            # Deny access to hidden files
            location ~* /\.ht {
                deny            all;
                access_log      off;
                log_not_found   off;
            }

            # Pass PHP scripts on to PHP-FPM
            location ~* \.php$ {
                try_files       $uri /index.php;
                fastcgi_index   index.php;
                fastcgi_pass    unix:/home/WEBFACTION_USER/var/spool/phpfpm.sock;
                include         fastcgi_params;
                fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
            }

        }
    }
    

Make sure to replace WEBFACTION_USER with your username, so the folders are correct. Also update the listen port from 99999 to the port that was given to your app.

(Re)start your nginx process by running

    ~/webapps/engine/bin/restart
    

Configuring wordpress via nginx

Now that we have both our php-fpm and nginx processes running, lets plug our site to use this new configuration. This is quite easy for those familiar with the webfaction control panel. Either change your existing website instance to point to the engine app, or create a new website, and point it to engine. Note that the domain name should match that of your wordpress instance. This is all done under the Domains / websites -> Websites menu.

Give webfaction a couple of minutes to sync, and you should be ready to access your wordpress, now running under nginx. Next step is to configure W3TC to generate the configuration for nginx

W3TC configuration

Login to your wordpress admin and go to the W3TC Performance menu. Ignore any errors or warnings you might see for now.

Scroll down to the Miscellaneous section. You should see a Nginx server configuration file path option. You should enter this path

    /home/WEBFACTION_USER/webapps/engine/nginx/conf/sites-enabled/nginx.conf
    

(replace with your own webfaction username).

Then click Save all settinegs.

Now you can click auto-install on all the warning messages that W3TC spits out. This will generate a custom nginx.conf file for you in the right folder. W3TC will probably keep complaining with an error that says

It appears Page Cache URL rewriting is not working. If using apache, verify that the server configuration allows .htaccess or if using nginx verify all configuration files are included in the configuration..

You now need to restart your nginx to pick up the new configuration file generated by W3TC.

~/webapps/engine/bin/restart

Varnish

The second variation was to use the previous nginx configuration, but also place Varnish cache in-front of it.

VARNISH:

{Internet} -> Webfaction Nginx -> My Varnish -> My Nginx / php-fpm -> W3TC / WordPress

and this is what it would look like with Varnish. It’s like the cherry on the cake. Probably not as sweet though.

Installing Varnish on Webfaction

Compiling and installing varnish is quite similar to php-fpm and nginx

    cd ~/src
    wget wget http://repo.varnish-cache.org/source/varnish-3.0.2.tar.gz
    tar -zxvf varnish-3.0.2.tar.gz
    cd varnish-3.0.2
    ./autogen.sh
    ./configure --prefix=$HOME
    make
    # this small hack was also required... (from http://community.webfaction.com/questions/5470/easy-reverse-proxy-cache-with-webfaction/5514)
    mv ./libtool ./libtool_old
    ln -s /usr/bin/libtool ./libtool
    make install
    

Unlike php-fpm, which can use unix sockets, varnish needs a TCP port to listen to. Luckily, webfaction makes it relatively easy. To do that, create a varnish app on webfaction. We’ll call it varnish, and choose Custom for the App category and Custom app (listening on port) for the App type. Click Create and notice the port number. We’ll use this port to listen to on our varnish configuration.

The configuration lives under ~/etc/varnish/default.vcl. You can simply overwrite the sample file already created there with this configuration:

 backend default {
  .host = "localhost";
# Replace this with your NGINX (Engine app) port !
  .port = "99999";
}
acl purge {
        "localhost";
}
sub vcl_recv {
        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                return(lookup);
        }
if (req.url ~ "^/$") {
               unset req.http.cookie;
            }
}
sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}
sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
if (!(req.url ~ "wp-(login|admin)")) {
                        unset req.http.cookie;
                }
    if (req.url ~ "^/[^?]+.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.$", "");
    }
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }
}
sub vcl_fetch {
        if (req.url ~ "^/$") {
                unset beresp.http.set-cookie;
        }
        if (!(req.url ~ "wp-(login|admin)")) {
                unset beresp.http.set-cookie;
        }
}

Notice that the port number in this configuration file is of the NGINX server (which we created earlier and called it engine)!

Now to run varnish:

    ~/sbin/varnishd -f ~/etc/varnish/default.vcl -s malloc,64M -a 127.0.0.1:55555
    

The port here is of the newly created varnish app (replace 55555 with your own port). You can tweak this to use more or less memory. webfaction gives each account 256Mb of application-usable memory, so it depends on how much stuff you have running already.

Benchmark results

To carry out those tests, I used ApacheBench from a near-by location with good internet connection and low latency (ping test showed around 4ms). I repeated each test several times and checked that there were no configuration problems or issues that might skew the results. The tests were carried out against the exact same wordpress site and testing both static and dynamic pages against each of the configurations (BASELINE, NGINX, VARNISH). I also used httperf for sanity-testing, to make sure the apache bench results were accurate, and on the varnish testing made sure varnishhits show realistic information about cache hits/misses.

The command I used was

    ab -kc 10 -n 1000 {url}
    

Static Pages

Static pages are not just plain html, but rather pages that W3TC caches and converts into static files. Ideally most, if not all of the pages on a typical wordpress setup can be cached this way. I have tested a couple of pages of different sizes.

Pretty Big Page (~100kb)

VARNISH

Document Path:          /category/dinosaurs/
Document Length:        113742 bytes

Concurrency Level:      10
Time taken for tests:   9.779 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      114047000 bytes
HTML transferred:       113742000 bytes
Requests per second:    102.26 [#/sec] (mean)
Time per request:       97.793 [ms] (mean)
Time per request:       9.779 [ms] (mean, across all concurrent requests)
Transfer rate:          11388.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   10   2.1     10      33
Processing:    57   87   6.2     87     112
Waiting:        3   11   2.3     11      34
Total:         67   97   6.4     97     124

Percentage of the requests served within a certain time (ms)
  50%     97
  66%    100
  75%    101
  80%    103
  90%    105
  95%    107
  98%    112
  99%    118
 100%    124 (longest request)

NGINX

Document Path:          /category/dinosaurs/
Document Length:        113742 bytes

Concurrency Level:      10
Time taken for tests:   9.723 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    993
Total transferred:      114059965 bytes
HTML transferred:       113742000 bytes
Requests per second:    102.85 [#/sec] (mean)
Time per request:       97.232 [ms] (mean)
Time per request:       9.723 [ms] (mean, across all concurrent requests)
Transfer rate:          11455.74 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   5.5      0      87
Processing:    43   96  17.5     97     310
Waiting:        3   66  22.5     74      88
Total:         43   97  20.9     97     397

Percentage of the requests served within a certain time (ms)
  50%     97
  66%     97
  75%     97
  80%     98
  90%    105
  95%    117
  98%    129
  99%    155
 100%    397 (longest request)

BASELINE

Document Path:          /category/dinosaurs/
Document Length:        113753 bytes

Concurrency Level:      10
Time taken for tests:   9.736 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    994
Total transferred:      114099970 bytes
HTML transferred:       113753000 bytes
Requests per second:    102.72 [#/sec] (mean)
Time per request:       97.356 [ms] (mean)
Time per request:       9.736 [ms] (mean, across all concurrent requests)
Transfer rate:          11445.21 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   3.6      0      63
Processing:    29   97  51.3     91     643
Waiting:        3   50  14.8     50     116
Total:         29   97  51.8     92     643

Percentage of the requests served within a certain time (ms)
  50%     92
  66%     98
  75%    101
  80%    106
  90%    126
  95%    149
  98%    213
  99%    407
 100%    643 (longest request)

Notice that nginx and our baseline setup are able to use HTTP keep-alive, whereas varnish didn’t. I’m not sure why this happens.

Smaller Page (~35kb)

VARNISH

Document Path:          /category/elephants/
Document Length:        38304 bytes

Concurrency Level:      10
Time taken for tests:   3.317 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      38607989 bytes
HTML transferred:       38304000 bytes
Requests per second:    301.50 [#/sec] (mean)
Time per request:       33.167 [ms] (mean)
Time per request:       3.317 [ms] (mean, across all concurrent requests)
Transfer rate:          11367.56 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    8   1.5      8      11
Processing:    17   25   2.2     26      35
Waiting:        2    8   1.5      8      13
Total:         23   33   2.2     33      40

Percentage of the requests served within a certain time (ms)
  50%     33
  66%     34
  75%     34
  80%     35
  90%     36
  95%     37
  98%     38
  99%     38
 100%     40 (longest request)

NGINX

Document Path:          /category/elephants/
Document Length:        38304 bytes

Concurrency Level:      10
Time taken for tests:   3.308 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    993
Total transferred:      38620965 bytes
HTML transferred:       38304000 bytes
Requests per second:    302.30 [#/sec] (mean)
Time per request:       33.080 [ms] (mean)
Time per request:       3.308 [ms] (mean, across all concurrent requests)
Transfer rate:          11401.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.4      0      23
Processing:    10   33  13.5     31     117
Waiting:        3   18   4.2     18      28
Total:         10   33  13.7     31     117

Percentage of the requests served within a certain time (ms)
  50%     31
  66%     36
  75%     40
  80%     42
  90%     46
  95%     51
  98%     79
  99%     95
 100%    117 (longest request)

BASELINE

Document Path:          /category/elephants/
Document Length:        38315 bytes

Concurrency Level:      10
Time taken for tests:   3.333 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    994
Total transferred:      38660970 bytes
HTML transferred:       38315000 bytes
Requests per second:    300.05 [#/sec] (mean)
Time per request:       33.327 [ms] (mean)
Time per request:       3.333 [ms] (mean, across all concurrent requests)
Transfer rate:          11328.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.5      0      23
Processing:     9   33  13.8     29     131
Waiting:        3   20   4.6     20      45
Total:          9   33  14.1     29     131

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     34
  75%     39
  80%     43
  90%     50
  95%     52
  98%     76
  99%     93
 100%    131 (longest request)

Even smaller (~15kb)

VARNISH

Document Path:          /cateogry/kittens/
Document Length:        16986 bytes

Concurrency Level:      10
Time taken for tests:   1.594 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      17290000 bytes
HTML transferred:       16986000 bytes
Requests per second:    627.16 [#/sec] (mean)
Time per request:       15.945 [ms] (mean)
Time per request:       1.594 [ms] (mean, across all concurrent requests)
Transfer rate:          10589.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    5   1.7      4      11
Processing:     5   11   2.2     11      17
Waiting:        2    6   1.7      5      13
Total:          9   16   2.2     16      25

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     17
  75%     17
  80%     18
  90%     19
  95%     19
  98%     21
  99%     21
 100%     25 (longest request)

NGINX

Document Path:          /cateogry/kittens/
Document Length:        16986 bytes

Concurrency Level:      10
Time taken for tests:   1.482 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    990
Total transferred:      17302950 bytes
HTML transferred:       16986000 bytes
Requests per second:    674.94 [#/sec] (mean)
Time per request:       14.816 [ms] (mean)
Time per request:       1.482 [ms] (mean, across all concurrent requests)
Transfer rate:          11404.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:     5   15   1.0     15      24
Waiting:        3   13   1.1     13      14
Total:          5   15   1.0     15      24

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     15
  75%     15
  80%     15
  90%     15
  95%     15
  98%     16
  99%     18
 100%     24 (longest request)

BASELINE

Document Path:          /cateogry/kittens/
Document Length:        16997 bytes

Concurrency Level:      10
Time taken for tests:   1.488 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    992
Total transferred:      17342960 bytes
HTML transferred:       16997000 bytes
Requests per second:    671.83 [#/sec] (mean)
Time per request:       14.885 [ms] (mean)
Time per request:       1.488 [ms] (mean, across all concurrent requests)
Transfer rate:          11378.42 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      13
Processing:     6   15   0.9     15      26
Waiting:        3   13   1.2     13      15
Total:          6   15   1.1     15      26

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     15
  75%     15
  80%     15
  90%     15
  95%     15
  98%     16
  99%     19
 100%     26 (longest request)

Dynamic Pages

I chose the wp-login.php for the test. Ideally most pages will be cached anyway. Nevertheless, it’s interesting to see what overhead (if any) is added by varnish or nginx, and also to compare PHP-FPM with the standard php-cgi provided by default.

VARNISH

Document Path:          /wp-login.php
Document Length:        3643 bytes

Concurrency Level:      10
Time taken for tests:   32.551 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    993
Total transferred:      4309921 bytes
HTML transferred:       3643000 bytes
Requests per second:    30.72 [#/sec] (mean)
Time per request:       325.514 [ms] (mean)
Time per request:       32.551 [ms] (mean, across all concurrent requests)
Transfer rate:          129.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:   137  324  42.2    316     545
Waiting:      137  323  42.1    314     544
Total:        139  324  42.3    316     548

Percentage of the requests served within a certain time (ms)
  50%    316
  66%    339
  75%    355
  80%    360
  90%    380
  95%    396
  98%    408
  99%    413
 100%    548 (longest request)

NGINX

Document Path:          /wp-login.php
Document Length:        3643 bytes

Concurrency Level:      10
Time taken for tests:   32.085 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      4213000 bytes
HTML transferred:       3643000 bytes
Requests per second:    31.17 [#/sec] (mean)
Time per request:       320.852 [ms] (mean)
Time per request:       32.085 [ms] (mean, across all concurrent requests)
Transfer rate:          128.23 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    3   0.5      3       6
Processing:   142  317  40.9    309     427
Waiting:      141  315  41.0    307     427
Total:        144  319  40.9    312     431

Percentage of the requests served within a certain time (ms)
  50%    312
  66%    332
  75%    348
  80%    357
  90%    380
  95%    397
  98%    412
  99%    416
 100%    431 (longest request)

BASELINE

Document Path:          /wp-login.php
Document Length:        3654 bytes

Concurrency Level:      10
Time taken for tests:   41.740 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      4201000 bytes
HTML transferred:       3654000 bytes
Requests per second:    23.96 [#/sec] (mean)
Time per request:       417.396 [ms] (mean)
Time per request:       41.740 [ms] (mean, across all concurrent requests)
Transfer rate:          98.29 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    3   0.5      3       4
Processing:   236  414  56.0    404     607
Waiting:      215  380  53.0    371     555
Total:        238  416  56.0    407     610

Percentage of the requests served within a certain time (ms)
  50%    407
  66%    434
  75%    452
  80%    465
  90%    494
  95%    519
  98%    554
  99%    577
 100%    610 (longest request)

Analysis

If I’m reading the numbers from apache-bench correctly, then nginx does offer some (relatively modest) improvement in performance. This is actually a little more noticable on dynamic php pages. This is not a huge surprise, since my nginx setup was configured with php-fpm, whereas the baseline setup had to use php-cgi. Adding Varnish on top does not boost performance that much further though. Perhaps my configuration wasn’t optimal, or maybe you really start to benefit when you have more memory. I don’t know. Considering the added complexity of two proxying layers, I’d say it’s not really worth it for me. I even wonder if managing my own nginx is worth the hassle. This is because with nginx, there’s a bigger risk of some weirdness in compatibility with wordpress and the myriad of plugins it supports, most of which are running on Apache.

It’s worth remembering that those benchmark don’t really simulate real-life scenarios. They hit the server with 1,000 request in a short span. Sure, it’s great for simulating your site getting slashdotted, but it’s not that realistic. Furthermore, browsing experience is affected by many other factors and elements on the page. Images, javascript, browser caching. The location of the user in relation to the server also makes a huge difference of course. Those tests were running from a close-proximity location with low latency. Doing the same from across the globe might produce much different results.

And of course, I also might have made some blatant configuration mistakes or used sub-optimal settings. I’d be happy to hear some ideas on how to work things even better!

Insights

I was reading a fascinating post recently, talking about why the web is slow. I know it’s not strictly related to my optimization, but there are many things to consider when trying to improve your site’s performance. Oddly, one of the things that really hit me. hard. The thing that I ignored completely before starting this process, was how much the actual size of the page matters. The few performance benchmarks which I came across online failed to even mention which pages were tested, or used really tiny out-of-the-box wordpress pages. If you really want to boost your site’s performance – make your web pages as small as possible. I am now starting to experiment with minifying my html using W3TC (A feature which isn’t enabled by-default), as well as trying to reduce page size and moving across unnecessary stuff to be fetched via ajax.

Thanks

Special thanks to webfaction support. It seems to me like they went beyond their call of duty to help me install stuff on the host on a couple of occasions. Considering it’s a shared-hosting provider, they really do a fantastic job. I hope this post might give some pointers to people who want to install nginx, php-fpm, varnish or anything else on webfaction. Perhaps it’s not as easy as using apt-get install, but it’s definitely possible.

Update

Looking at the results I realised that apache-bench does not use gzip compression by default. However, it can be used with gzip compression, which makes page sizes considerably smaller, and hence the results much faster. I only did a very quick cursary test, and so pretty big page became rather skinny, dropping from around 100kb to only around 15kb (and the response times accordingly). To test your site with gzip switched-on, use

ab -kc 10 -n 1000 -H 'Accept-Encoding: gzip'

WEP

There must be plenty of examples of when encryption was used incorrectly, and how it lead to unforeseen consequences. The most famous example I am aware of is WEP. I won’t go into the details. There are plenty of resources online describing the flaws in great detail. The essence of it however was that WEP used relatively good ciphers incorrectly, or insufficiently. The encryption algorithm it used (RC4), was not ‘broken’. The algorithm itself is quite robust (No algorithm is perfect, but this wasn’t the major flaw in WEP). It’s the design of WEP, not the core algorithm, that were broken.

Some illustration

Lets walk through a simplified, imaginary scenario, to illustrate when NOT to use encryption. It goes something like this:

Developer: “We need to allow access to the X section of the website to more people”
Me: “Ok, then ask them to login before they can have access”
Developer: “No. We want to send them a secure link with expiry date and not all users will have an account”
Me: “In that case, we should use something like HMAC or OAUTH. It’s designed for this purpose”
Developer: “No. I don’t have time to read about Oauth, it’s too confusing. But I’ve written a real cool function that encrypts the part of the URL, and only if we can decrypt it properly, we allow access… It uses Triple-DES, so it’s super-secure. Wikipedia says that The algorithm is believed to be practically secure in the form of Triple DES, although there are theoretical attacks”
Me: “Lets have a look at your ‘secure’ solution then…”

The solution was something along those lines (hugely simplified for the sake of illustration):

http://site.com/secure-access/{encrypted_string}

where encrypted_string contained the expiry date of the page, e.g. 20120315 (15th March 2012).
Can you see where this is flawed??


There are at least a couple of possible attacks here, that don’t involve anything with breaking the encryption or trying to get the key:

1. Randomly change the encrypted string – perhaps not very sophisticated, but since all we need is a date larger than today, there is still a fair bit of chance we might be lucky. Even without calculating the entire search space, it’s quite easy to see it is quite narrow.
2. Replace it with another encrypted string – this is even easier. All you need as another URL string with a known-to-work encrypted string. Copy this string and paste it, and you’re good to go!

Another flaw the developer hasn’t worked out was that he was using triple-DES in ECB mode. This is the most basic mode, which makes the attacks I described much easier. So even if the algorithm itself is robust, the way it is applied can be much more important. In addition to that, under some circumstances, if the encryption process failed (which it could easily, since there was no problem manipulating the encrypted string), the code was very kind as to output the decrypted string inside the error message… Back to the user (or wannabe hacker as the case may be).

Of course, the solution was a little more complicated. It contained not just an encrypted date, but some other data. Nevertheless, the same principles of attack applies. There are many bit-flipping attacks, that can alter encrypted data and generate a predictable output. The bottom line is this: Encrypting data does not prevent modification.

The alternative

In this case, all we want to achieve is authentication. We want to verify that the request was legitimate, and that nobody else can fake such a request. There’s nothing in particular that needs hiding. The expiry date of the access is not such a big secret. All we need is some kind of a signature, to validate the legitimacy of the request. This is where hash functions, and HMAC / Oauth come into play. Those mechanisms, for some strange reason, are less appealing to many developers. I’m not entirely sure why, but maybe it’s just not as fun to see an extra hash at the end of the url as it is to encrypt a string. These mechanisms are much more effective in this case. This is what they’re designed to do.


So how does this work? Again, for the sake of simplicity I’m not going to cover the detailed aspects of those algorithms. But the principle is quite simple: You take a string you want to ‘sign’, plus a secret key, and generate a unique hash value from both of them. This value will be totally different even with the slightest modification to the original string (or the key). The key will never be published or shown, only the string with the generated ‘signature’ (this unique hash value we produced). How is this more secure? As I mentioned earlier, even the slightest modification of the string will produce a completely different signature. This will prevent any undetected modification of the url. Producing the unique hash signature is virtually impossible without the knowledge of the secret key. Voila! Simple. Secure. Elegant. Of course, as always, god is in the details, so even these algorithms can be used wrongly, producing an insecure outcome. However, from my experience, following the guidelines and using the oauth/hmac libraries is far easier and less error-prone than using any encryption algorithm.

Size doesn’t matter, it’s how you use it

The Security of the WEP algorithm page sums it up quite nicely: “The [WEP] protocol’s problems are a result of misunderstanding of some cryptographic primitives and therefore combining them in insecure ways”. Even if you pick the best and most secure encryption algorithm, it might not be enough to make your solution secure. In fact, as I tried to illustrate, encryption might not be necessary at all.

Dynamic Goal Conversion Values

I was trying to get some dynamic goal conversion values into Analytics. I ended up reading about Ecommerce tracking and it seemed like the way to go. Not only would I be able to pick the goal conversion value dynamically, it gives you a breakdown of each and every transaction. Very nice. After implementing it, I was quite impressed to see each transaction, product, sku etc appear neatly on the ecommerce reports. So far so good. But somehow, goals – which were set on the very same page as the ecommerce tracking code – failed to add the transaction value. The goals were tracked just fine, I could see them adding up, but not the goal value. grrrr…

I was particularly frustrated after meticulously reading through the Ecommerce Transaction Page Goals, which is very clear about this being possible, as long as you set the conversion value to zero.

The solution

There’s a small thing that doesn’t seem to get mentioned anywhere, and I’m still not sure why it causes a problem. If it doesn’t work for you, I have a few alternative approaches worth trying too…

I discovered that in my case, the google analytics code was included on the goal conversion page twice. Once, as in every page, the standard tracking code on the header. And again on the body, together with the ecommerce tracking values. Both were running fine, and as I mentioned the results appeared. But somehow it seems to create some conflict. Removing the ‘standard’ tracking section seemed to have solved this mysterious problem.

Here’s what the HTML looked like – Remove the first part

<head>
...
<!-- FIRST SECTION ON THE HEADER : TO REMOVE -->
<script>

  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-xxxxxxxx-x']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>
<!-- DON'T REMOVE ANY FURTHER -->
</head>
<body>
...
<!-- SECOND SECTION - KEEP THIS ! -->
<script>
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXX-X']);
  _gaq.push(['_trackPageview']); // this tracks the page view
  _gaq.push(['_addTrans',
    '1234',           // order ID - required
    'Acme Clothing',  // affiliation or store name
    '11.99',          // total - required
    '1.29',           // tax
    '5',              // shipping
    'San Jose',       // city
    'California',     // state or province
    'USA'             // country
  ]);

   // add item might be called for every item in the shopping cart
   // where your ecommerce engine loops through each item in the cart and
   // prints out _addItem for each
  _gaq.push(['_addItem',
    '1234',           // order ID - required
    'DD44',           // SKU/code - required
    'T-Shirt',        // product name
    'Green Medium',   // category or variation
    '11.99',          // unit price - required
    '1'               // quantity - required
  ]);
  _gaq.push(['_trackTrans']); //submits transaction to the Analytics servers

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
</script>
...
</body>

Alternative Method

Since analytics seem to encompass lots of black magic, I’m still not 100% sure that’s the only possible solution. However, along the way I discovered some other interesting things…

Event Tracking

Instead of goal tracking by URL, you can actually track an event. If you’ve already coded the ecommerce tracking section anyway, adding event tracking to it is a doddle. Simply add this line somewhere before or after the ecommerce tracking code

_gaq.push(['_trackEvent', 'Cart', 'Checkout', 'Success', 39.99]);

In this case I defined the event category to be ‘Cart’, the action to be ‘Checkout’, the option to ‘Success’, and the value to 39.99. Once you do that, you can set your goal to use an Event instead of a URL. Here’s more on info on setting the event goals and choosing the value as the conversion value.

If your main concern is tracking conversion values, and you’re not too fussed about the whole ecommerce tracking, then this is actually a much easier way. All you need is one line of code. It can however work perfectly well together with your existing ecommerce tracking.

Wet paint – slow updates

Analytics seems to update rather slowly. It also seems to update and un-update some data if you keep refreshing. You see a new transaction. You refresh. It’s gone. Refresh again, it’s back… This is normally not a huge problem, just confusing, but when you’re trying to debug something, can make things much more difficult. For example, if you update your Goal configuration. Say you change the URL, or the tracking event, or added a new goal. It might take a good few minutes for it to update. If you trigger the goal, e.g. by going to the URL page, it might not trigger – because Analytics still uses the old goal settings. You simply have no way to find out whether the goal is active or not. You must wait for the paint to dry before you try anything new.

Other stuff

A couple of more things to watch out for:

• The Ecommerce page says that
  

  For URL: Supply the URL for your shopping cart. For example: http://www.we-sell-for-you.com/mysite/myCart.asp

  In most cases, you only need to supply the relative path, e.g. /mysite/myCart.asp

• I’m not sure whether it makes a difference, but if you’re using _gaq.push(['_setDomainName', '.domain.com']);, this might also cause some issues? (didn’t test this explicitly though)

Final grunt

Hope this is helping someone. I was pulling my hair over this so hope I can save some other people’s hairline from receding. The time it takes for the goal/ecommerce tracking to appear on analytics also makes testing it rather long and much more difficult to resolve. Having tested a few too many different combos, I’m still not 100% sure this is a bullet-proof solution, but I’m definitely seeing some goal conversion values now.

UPDATE: looks like it will only work if you also use event tracking. With URL tracking alone it doesn’t seem to pick the goal values.

This is not normally a problem, unless the URL is already encoded. Since all unicode-based urls use this encoding, they are more prone to these errors. What happens then is that a URL that looks like this:
http://www.frau-vintage.com/2011/%E3%81%95%E3%81%8F%E3%82%89 …

will be encoded again to this:
http://www.frau-vintage.com/2011/%25E3%2581%2595%25E3%25 …

So clicking on such a double-encoded link will unfortunately lead to a 404 page (don’t try it with the links above, because the workaround was already applied there).

A workaround

This workaround is specific to wordpress 404.php, but can be applied quite easily in other frameworks like django, drupal, and maybe even using apache htaccess rule(?).

<?php
/* detecting 'double-encoded' urls
 *  if the request uri contain %25 (the urlncoded form of '%' symbol)
 *  within the first few characeters, we try to decode the url and redirect
 */
$pos = strpos($_SERVER['REQUEST_URI'],'%25');
if ($pos!==false && $pos < 10) :
    header("Status: 301 Moved Permanently");
    header("Location:" . urldecode($_SERVER['REQUEST_URI']));
else:
    get_header(); ?>
    <h2>Error 404 - Page Not Found</h2>
    <?php get_sidebar(); ?>
    <?php get_footer();
endif; ?>

This is placed only in the 404 page. It then grabs the request URI and checks if it contains the string ‘%25′ within the first 10 characters (you can modify the check to suit your needs). If it finds it, it redirects to a urldecoded version of the same page…

Efficient code

Django makes a lot of things very easy. One of its most prominent features is its ORM (Object Relational Mapper). The django ORM really makes it easy to write complex database queries without a single line of SQL. This however might come at a price of sub-optimal queries, which can take a very long time to process, and also consume more memory. If you hit django memory issues, it’s very likely related to some heavy data being loaded from the database, consuming lots and lots of memory. Luckily, there are a few rules that if followed can hugely improve both the queries and memory usage. Most of those are documented clearly on the Database access optimization page. I won’t repeat everything that’s being documented there, but rather focus on a few key points.

Where do you start?

First of all, you might already see crashes on your django, get reports from users about pages loading slowly, and you’ve hopefully checked your server (e.g. using top/htop or free commands) and noticed high memory utilization. The recommendations on part I should give a little stability until you manage to figure out where the problem lies, but it is by no means a sufficient solution.

Monitor log files

The easiest place to start is your log files. If you’re not logging requests already, you seriously should. One helpful method that doesn’t add much overhead to django logging is adding the time that the request took to the logs. This can be achieved with a simple logging middleware. The time each request takes usually gives a fair indication of what’s going on, and from my experience there’s usually a close link between execution time and memory footprint. The same logging middleware can be used with DEBUG=True to view how many SQL queries are executed for each request. This can give another indication of ‘hot-spots’ to look for.

Profiling

The next step is to profile django and see which requests consume high amount of memory. I would suggest following the instructions on Using Guppy to debug Django memory leaks. This should really help pin-pointing areas of code that consume large amounts of memory. Once those are identified, it is easier to try to optimize the code there. Don’t try to do everything at once, but from optimizing one area of code you would learn a lot and could easily apply the same methods across the entire codebase.

Quick-n-dirty improvements

There are a few recommendations that generally help with reducing memory footprint. Be aware not to use them blindly though. They can have a knock-on effect on performance in other areas. So exercise some judgment. The most useful pointers are:

• Make sure DEBUG=False
• .iterator() – In many cases, there’s simply a need to retrieve a list of objects from the database, and return those in a certain format. If you go over the list more-or-less sequentially and use each object only once, it makes a lot of sense to use iterator() – this will eliminate django internal caching, which can save on memory. If, however, you might go back to the same object – this caching could really speed things up, so be aware of what you’re doing
• Using .update()/.delete() – can be a huge saver if you perform a simple update of many objects, or delete a bunch of them. Rather than walking through objects one by one, you can perform this operation with one query
• Using .values() – As above, if all you need is a bunch of values from the database, you can save a lot of memory by fetching them as values instead of as complex queryset objects
• Using .only() – can also save a lot of memory, if you only use a portion of data from each object.

These are just a few pointers to the same page on the django documentation. Please spend time reading through the entire page, as it contains many very useful techniques and other information to help you better understand how the ORM works and how to use it more effectively.

External Monitoring

Even if you manage to optimize your code, tweak the maxrequests parameter to ensure memory is cleared regularly, and brush your teeth morning and night, there are still cases when it simply won’t be enough. One user will make too many requests, the data-set will just be too big, or some query will stay sub-optimal, and you’d still end up with bloated processes eating through your memory. If that happens, you’re almost back to where we started. This is where some external tools can work wonders. They can:

1. Let you know when this happens, perhaps even before memory is completely depleted.
2. Take (automatic) action, and more gracefully restart django, to avoid a full-scale crash.

Monit

My weapon of choice in this case (and many others) is Monit. It’s lightweight, powerful and has a very easy and intuitive configuration syntax, which makes it a snap to use. There are so many uses of this little devil, but in this case I will focus on monitoring process memory. It only takes a few lines to let monit watch over django, and make sure things are running smoothly:

check process your-django-process with pidfile /path/to/your/django.pid
    start program "/etc/init.d/your-django reload"
    stop program "/etc/init.d/your-django stop"
    if totalmem > 70% then exec "/usr/local/bin/highmem" # more about this highmem later...
    if totalmem > 85% for 2 cycles then restart

Lets go over this monit config snippet quickly. What it does is very simple:

1. Monit checks our django process. Make sure you specify the correct pid file (which we set when invoking django using manage.py)
2. start and stop program directives should point to your django daemon script
3. If the total memory of the django process (including children) exceeds 70% of available memory, then it executes a custom ‘highmem’ command. The highmem command will get django to clear some memory by restarting its internal processes. We will cover this command shortly. This will also automatically send an email to alert you (make sure you configure your monit alert settings correctly)
4. The second check is a “safety-belt”, to restart django if memory stays high for too long despite all our efforts

highmem

The highmem command is a very simple 1 line bash script:

#!/bin/bash
kill -SIGUSR1 `cat /path/to/your/django.pid`

All it does is send a SIGUSR1 to our django process. So what does SIGUSR1 do? When django runs in fastcgi, it uses flup for process handling. Since version 1.0.3, flup uses the SIGUSR1 signal to safely respawn those django processes. Popping those balloons. If a request is in-progress, it will wait until it finishes, which is a very nice feature. Just hope that this process waiting to complete won’t take out ALL memory left… You can read more about it on How to Gracefully Restart Django Running FastCGI (look for the comments section in particular). Please note that you’d need flup version 1.0.3. If you’re using an older version, it might just kill your django processes instead of respawning them safely. It’s easy to get flup, simply use sudo pip install flup (or sudo easy_install flup), and you’re done.

The last resort

The last thing on our arsenal of tools, the last resort, is if some process is eating our memory so fast, that even the highmem command didn’t manage to release it, and we simply have no choice but to restart django. However, even then, we’d probably want to do it as gracefully as possible. I’m not going to repeat it, but the page on How to Gracefully Restart Django Running FastCGI covers what you need to do to modify your manage.py in order to give django just a few seconds before it’s restarted. Then just make sure that the scripts starting/restarting django send a KILL -HUP to your django process to restart it nicely.

Does django leak memory?

In actual fact, No. It doesn’t. The title is therefore misleading. I know. However, if you’re not careful, your memory usage or configuration can easily lead to exhausting all memory and crashing django. So whilst django itself doesn’t leak memory, the end result is very similar.

Memory management in Django – with (bad) illustrations

Lets start with the basics. Lets look at a django process. A django process is a basic unit that handles requests from users. We have several of those on the server, to allow handling more than one request at the time. Each process however handles one request at any given time.

But lets look at just one.

cute, isn’t it? it’s a little like a balloon actually (and balloons are generally cute). The balloon has a certain initial size to allow the process to do all the stuff it needs to. Lets say this is balloon size 1.

Now every request that comes to the server gets sent to one of those (cute) django processes. Then to serve the request, the process loads objects into memory. Like this

Those little bubbles are the objects loaded into memory. Once the process finishes processing a request it will clear all the objects from memory and go back to being ‘empty’. It is still size 1 since all the objects fitted within the space.

But some time the request is a bit heavier. It needs to load more objects than its size.

So the process simply inflates itself and grows a little. Easy. Now it’s size 2. More space for bubbles.

and of course, once the request finishes, it clears all those bubbles and there’s space for the next ones.

An import thing to note: The balloon (process) never shrinks. It can only grow. But this is (kind-of) ok, since it will never grow bigger than the biggest request we can get. So even a very big request (lets say one that uses 1Gb memory), we can probably handle. Right??

Not quite. So what’s the problem?

Well, like this little cute process we have other processes. Remember we have to serve more than one user at a time. So we must keep a few of those balloons running. So if more than one BIG request come at roughly the same time, they will inflate not just one balloon, but a few of those. And these balloons compete for space on the server (which is like a big room that contains the balloons, but the room does not grow).
This is our room:

Of course we can clear the room and start empty – this is what we do when we reboot the server or even just restart django. This is what we have to do when the server crashes. When in fact the balloons grew so big that other balloons couldn’t grow any more. So rebooting all the time is not an option. When we do that, everything stops. Including requests that are being processed. Even if they’re half-way through.

So – how about we ‘pop’ those balloons every now and then – when they’re NOT processing a request (other balloons do it), and start from a small balloon? That’s actually possible. However, there are two limitations to be aware of:

• To create a new balloon takes some effort. We have to ‘make’ the balloon. While we make a balloon the others are responding slower.
• We cannot just ‘pop’ a balloon based on its size. Instead we can only create an ‘automatic balloon popper’ that pops the balloon after X requests.

Our degree of control is as follows:

1. minspare – How many empty balloons do we start with. This will potentially save us effort later by having a few ready. The ‘cost’ in term of memory is the {number of balloons} X {balloon initial size}. The benefit, is saving time creating a new process for simultaneous requests. However, This parameter is not very helpful to our problem.
2. maxchildren/maxspare – What is the maximum number of balloons/processes we want to have on the system. This determines the maximum number of simultaneous requests we can deal with. The ‘cost’ is the {number of balloons} X {balloon size}. The balloon size can obviously grow over time!
3. maxrequests – this is the ‘auto-popper’. We can decide after how many requests we ‘pop’ a balloon and start a new one.

So if we set maxrequests too low, say 1 – then the system will work very hard to create a new process/balloon for every request. This is silly if the request is very small and doesn’t need a big balloon. With too high value however, the balloons might grow too much before they’re popped. Even if the maxrequests is 1, if we get a few requests at the same time, each causing our balloons to grow too much, we might still run out of space!

Our worse-case scenario is calculated by : {number of simultaneous requests} X {size of the request}. Lets say our server have 4Gb memory in total, which probably leaves about 3Gb memory for django itself. However, with requests that might take ~1Gb in memory (worst-case-scenario), we can only serve a maximum of 3 such requests. Not even simultaneously. Just in proximity to each other, before the server runs out of memory…

Conclusion

One of the core issues I wasn’t addressing here is obviously how to prevent high-memory usage within the django process. I hope to cover this on the next part. There are certainly some recommendations and best-practices when it comes to memory usage. However, with some types of requests, it might be impossible to avoid high-memory usage. Given enough simultaneous requests, even with optimization that leads to ‘reasonable’ memory utilization, django might still run out of memory. The minspare, maxchildren, maxspare, and most importantly maxrequests parameters are therefore crucial to having a more stable django service. It’s not a bullet-proof solution, but from my experience it helps a lot.

Sweet-Spot settings

So what are my recommended settings? I found that setting maxrequests=100 seems to give a reasonably good performance overall. Simply run django in prefork mode with something like this:

manage.py runfcgi method=prefork host=$DAEMON_HOST port=$BACKUP_DAEMON_PORT pidfile=$PIDFILE maxrequests=100

I didn’t see any need to change the default minspare, maxchildren, or maxspare parameters however.

What’s next?

On Part II I am going to cover some more advanced tweaks. Those are designed to detect and recover from situations where django runs out of memory. Using the balloon popping analogy, those tweaks/methods allow ‘popping balloons’ when memory runs out, rather than only after 100 requests. This gives another layer of protection against memory-related crashes. However, these require monitoring tools outside django. In addition, I hope to give at least some pointers on how to better utilize memory within the code.

Luckily no websites I know or maintain were affected, possibly since the htaccess change I used shouldn’t allow using remote URLs in the first place (and also it renamed timthumb.php from the url string, making it slightly obfuscated). I still very strongly advise anybody using timthumb to upgrade to the latest version to avoid risks.

Ajaxize

The plugin allows you to take any wordpress function and ‘ajaxize’ it into a special div. Typically, all plugins and core wordpress functionality boils down to a number of php functions. If you are able to figure out which function your plugin uses to output content, you can ajaxize it. How do you find the function name? This is not that complicated. Many plugins will actually tell you which function to use. For example, if you want to embed the output in one of your templates. They will instruct you to use something like this:

<?php echo plugin_function_name(); ?>

So all you have to do is take this ‘plugin_function_name’, and ajaxize it using my plugin. The output is a div which looks like this:

<div id="ajaxize_this:plugin_function_name:68e46660f7ce3bc77a51465219df5743879544bc"></div>

Then place this div inside your page, post, widget, header, anywhere really. The ajaxize plugin adds a small javascript that will find this div, and convert it automatically into an ajax call for you!


There are a few limitations though:

• Functions must return valid HTML – this will be called in php and returned via the Ajax call
• Functions cannot accept any parameters (at least at the moment)
• Functions that work within a context (e.g. of a post, page, category), will most likely lose the context information

UPDATE: version 1.1 of the plugin now handles context much better. Ajaxize is now hooking in the right place, so the ajax call is made exactly where the div element is placed. This means plugins that use a post/category/taxonomy context information can now also be ajaxized. Special thanks to One Trick Pony for helping me figure out how to hook this correctly.


This was a perfect solution for mixing caching with dynamic content. I can convert almost any plugin or widget into a div. I can also write very simple PHP functions that will show dynamic content on my pages, with zero extra javascript code. The div itself can be cached, but the content will be pulled automatically by the browser when the page loads. I also found it useful for loading plugin buttons like Facebook like and Twitter tweet. Those can take a while to load and slow the page. When converted via ajaxize, they still take a while to load, but don’t seem to hold the page content from loading first.

What about security?

Some of you may have already started thinking… “but hang on a minute. If you can ajaxize one function, what stops somebody from calling other functions on my wordpress??!!”. Very true. This is why ajaxize was built-in with security in mind. It uses a very powerful algorithm called HMAC, with a secret key, so you can use ajax on any function you like, but only those functions and not others. This also means zero-configuration. The plugin only stores one value in the database – this is your secret key! I might cover the security aspects of the plugin on a future post. I encourage people to look through the code and validate the security I’ve implemented.

Feel free to try it out and let me know what you think!