Getting Started with ELB

We recently started using Amazon's Elastic Load Balancing (ELB) service for our webapp (which is hosted in AWS). We're pretty happy with it so far; although we initially tried using Cherokee as our web server, that didn't seem to work quite right with ELB: About every few hours or so, we'd get a blank page back from the load balancer. I suspect it's a version of a similar issue between ELB and Apache that Amazon fixed back in May 2010. We switched over to using Apache for our web server, and that seemed to fix it, so we didn't dig any further.

One other little disappointing thing about ELB is that it can't serve "naked" domains (ie example.com instead of www.example.com), so we still have a single EC2 instance with a fixed ip address to which we point the DNS record for our naked domain name (and which our web server redirects to our "www" subdomain). Shlomo Swidler has a real nice blog post on why this is, and generally how ELB works (and how to test it).

Quick Start

The AWS docs already have a pretty quick, concise set of steps to setup an ELB for your web servers, so here's a little bit of a higher-level overview of what you have to do:

1 Create the Load Balancer

The only thing that's set in stone when you create the balancer is its name, which is the id you use to reference it, and also becomes the first segment of its dns name; and the region it's in. However, when you create the balancer, you also have to specify at least one port to listen on and at least one availability-zones among which to balance (although you can change both any time you want).

A load balancer can balance among all the availability zones of the region in which you create it. You do have to specifically enable each zone, however. If you don't register with the balancer any of your app's instances from any one particular availability-zone, the balancer will just ignore that zone (ie it won't try to balance to empty zones). You'll want to make sure you have an equivalent number of instances in each zone that you have any instances in, though, since the balancer doesn't adjust its balancing policy to account for the number of instances you have registered in a zone — it will just assume that it can balance to each zone equally.

If you're serving straight HTTP, you'll want to configure the balancer's listener to use the HTTP protocol (the other option is TCP, for everything else). You can configure the balancer to listen on one port and route to another port, if you want (like to listen on port 80, but route to port 8080 on your app instances).

2 Configure the Ping URL

Once you've create the balancer, the next step is to configure it with the URL of your app that the balancer should ping for each instance, to check if the instance is still healthy. You can actually specify a different port for the ping URL, so you could theoretically have the ELB ping a completely different application than the app is balancing (it just has to be some process listening on your instance). But you probably just want it to be some URL in your app that serves a stripped-down (or even empty) HTML page.

Also note that, as long as you specify the HTTP protocol for the ping URL, the ping URL must return an HTTP response status of 200 for the balancer to consider it to be OK. Anything else — including a 30x redirect to some other page — and the balancer will consider it to be unhealthy.

And when you configure the ping URL, you also configure the interval between pings (in seconds), and the max number of seconds in which your server must respond before the balancer times-out the ping. Also, you must set a threshold for the number of good pings before the instance is added to the group balanced by the ELB, and the number of bad pings before it is removed; AWS requires these threshold values to be at least 2. A pretty standard setting for this is to make the interval 30 seconds, the timeout 10 seconds, and the healthy and unhealthy thresholds 2 — so if an instance starts going sideways, the ELB will continue to balance to it for at most a minute and a half before it recognizes that it's gone bad.

3 Configure Sticky Sessions (or Not)

If you need the balancer to do "sticky" sessions (balance to the same instance over the life of your application's sesssion — for example because you have per-session state that's not shared among instances), you need to configure the ELB to use sticky sessions keyed off your application's session cookie. For example, if you're using a standard java servlet engine (like tomcat or jetty), you'd configure the ELB to use the JSESSIONID cookie to route all requests for the same session to the same instance. And note that with the ELB API, this is a two step process — the first step is to create a stickyness "policy", and the second step is to apply it (individually to each listener that needs to use the policy).

If you don't need sticky sessions, then you don't need to do anything special (ELB will do round-robin by default).

4 Register Web-Server Instances

The last step to get going is simply to add your app instances to the load-balancer. The balancer will balance only among instances in the availability zones you specified when you created the balancer (unless/until you separately enable/disable those zones). Note that you should add an equivalent number of instances to each zone to which you've added any instances, as the ELB will try to balance equally among all zones in which you have healthy instances.

5 Update Your DNS Records

The real last step, of course, is to point actual live traffic to the ELB. AWS gives you a unique (and unpredictable) DNS name for the balancer (like my-elb-name-123456789.us-east-1.elb.amazonaws.com; the DescribeLoadBalancers API response gives you it). The DNS name never changes over the life of the instance, though. So to direct your traffic through the ELB, you create/update CNAME records for all the subdomains you want the balancer to serve (ie www.example.com, www.example.org, etc), pointing each record to the balancer's AWS DNS name. As the DNS update propagates across the tubes, your users will start using the ELB.

Cherokee Django

The Cherokee Cookbook and Yann Malet's Blog Roll post have pretty helpful directions for setting up the Cherokee webserver with Django. I found it a little tricky to set it up on a virgin Ubuntu Lucid Lynx system, though, so I thought I'd share my steps to get it going:

Install Cherokee and Django

You can get a fairly recent version of each from the Lucid repositories. Because you will compile uWSGI in a sec, you will also need to get python-dev and libxml2-dev:

$ sudo apt-get install cherokee libcherokee-mod-libssl python-django python-dev libxml2-dev

Install uWSGI

According to the cherokee docs, the best way to run django is via uWSGI. On the uwsgi home page, they do have a link to an ubuntu ppa for uwsgi. It looks pretty well maintained, and it's nice of Steve Crozz to maintain it, but I don't feel great about using someone's personal-package archive in a production environment. So I figured I would just get the code and build it myself.

Unfortunately uwsgi doesn't quite have the "forehead install" kind of make file. As I mentioned before, on ubuntu you need the python-dev and libxml2-dev debs, and you also need to specify the version of python you will use it with. These are the exact steps I took:

$ wget -nd -O /tmp/uwsgi.tar.gz http://projects.unbit.it/downloads/uwsgi-0.9.4.4.tar.gz $ tar xz -C /tmp -f /tmp/uwsgi.tar.gz $ cd /tmp/uwsgi-0.9.4.4 $ VERSION_TO_MAKE=`python --version 2>&1 | perl -ne '/(\d)\.(\d)/; print "$1$2"'` $ make -f Makefile.Py$VERSION_TO_MAKE $ sudo mv uwsgi$VERSION_TO_MAKE /usr/local/bin/uwsgi

Create the django_wsgi File

You need to have a wsgi script just like Yann described in his Blog Roll post; since I'm not all that hip to django, though, I didn't know exactly what to call it or where to put it — this is how I eventually ended up organizing my django project:

/srv sites MyProject __init__.py django_wsgi.py manage.py settings.py urls.py

That is, I called the wsgi script django_wsgi.py, and put it in the root of my django project, at the same level as the settings.py file. The django_wsgi.py script contains exactly what Yann described (where MyProject is the name of your project):

import os import django.core.handlers.wsgi # init django settings os.environ['DJANGO_SETTINGS_MODULE'] = 'MyProject.settings' # define wsgi app application = django.core.handlers.wsgi.WSGIHandler() # mount this application at the webroot applications = { '/': 'application' }

Configure Cherokee

Since I'm a complete cherokee noob, the cherokee docs were a little bit thin on this point (the screencast included in the docs might have been exactly what I needed, but I can't stand to sit around and watch screencasts, so I skipped it); this is what you do:

1. Run the cherokee admin server: $ sudo cherokee-admin
2. Open up your web browser to http://127.0.0.1:9090 and enter the credentials that cherokee-admin spit out.
3. Click the Virtual Servers navigation link.
4. Click the Wizards button at the bottom of the virtual servers list.
5. Select Platforms from the Category list, and click the Run Wizard link next to uWSGI.
6. On the first page of the wizard, enter the full domain name of your server in the New Host Name (once you get everything working with the new virtual server added by the wizard, you may want to delete it and just re-add the uwsgi handler manually in the default virtual server).
7. The Document Root field is irrelevant for now; I just set it to my django project's static directory.
8. For the Configuration File field, enter the path to your wsgi script (you'll change this setting in a sec anyway, so don't sweat it too much), like /srv/sites/MyProject/django_wsgi.py.
9. For the Same logs as vserver field, I have no idea what this does, so I chose the default.
10. Click the Submit button to finish the wizard, then go back and click the Information Sources navigation link, and select the new uWSGI 1 source you just created.
11. The Interpreter field will contain something like this:

    /usr/local/bin/uwsgi -s 127.0.0.1:41058 -t 10 -M -p 1 -C -w /srv/sites/MyProject/django_wsgi.py

    — change it to this:

    /usr/local/bin/uwsgi -s 127.0.0.1:41058 -t 10 -M -p 1 -C --pythonpath /srv/sites -w MyProject.django_wsgi

    (where /srv/sites is the directory which contains your django project-directory (MyProject), and MyProject is the name of your project).
12. Save it, and try navigating to your server in a browser.

Troubleshooting

Getting the right Interperter field setting in cherokee took me a while, and the logs were of no help. What you need to do is just run the uwsgi command (/usr/local/bin/uwsgi ...) yourself in a shell (killing the existing uwsgi instances if necessary); as long as you run it with the same socket as specified in cherokee's Connection field, cherokee will try to use the uwsgi instance you spun up, instead of spinning up its own instance. When you run uwsgi from a shell, you do get all the regular logging and error messages you would expect to see from django (from the shell stdout/stderr).

The other thing I found (which is probably pretty obvious if you know what you're doing) is that I had to give the uwsgi user (by default the same as the cherokee user, www-data) write rights on the django project. So I ended up creating a new copy of the django source from my working copy to /srv/sites/MyProject, and then making www-data the owner of that copy (sudo chown -R www-data:www-data /srv/sites/MyProject). There's probably a better way to do that, but this way worked.