IPv6 for Private AWS Subnets

Instead of setting up "private" subnets in AWS with the use of moderately-expensive AWS NAT Gateways, I've been experimenting with free Egress-Only Internet Gateways (EIGW). The downside with EIGW is that it's IPv6 only — so you can communicate to the outside world from instances in your private subnet only through IPv6.

In theory, that shouldn't be a problem — it's 2020 and IPv6 just works everywhere, right? Well, actually, in practice it does work pretty well — but there are still a few gotchas. Here are a few hurdles I had to work around when setting up a fleet of Ubuntu Linux EC2 instances with Ansible:

APT Repos

The base AMIs that Ubuntu provides are configured to use the Ubuntu APT repos hosted by AWS (like us-east-1.ec2.archive.ubuntu.com); however, these repos only support IPv4. So the first thing you need to do is change the repos listed in /etc/apt/sources.list to use external repos that support IPv6 (like us.archive.ubuntu.com, or other Ubuntu mirrors you might find).

And since you won't be able to use IPv4 to access the repos, you can speed up APT updates by configuring APT to try only IPv6. To do so, add a file in your /etc/apt/apt.conf.d/ directory (call it something like 99force-ipv6) with the following content:

Acquire::ForceIPv6 "true";

Also don't forget that if you do set up a restrictive Network ACL for your private subnet, you'll need to allow inbound TCP access to the standard Linux ethereal port range (32768-61000) from whatever APT repos you use.

NTP Pools

The NTP pools used by the base AMIs also don't support IPv6. I use the traditional NTP daemon provided by the Ubuntu ntp package, rather the default systemd-tymesyncd service. To configure the NTP daemon, I remove all the default pools from the /etc/ntp.conf file, and instead just use the 2.us.pool.ntp.org pool (the convention with NTP is that for domains that have numbered pools, like 0.us.pool.ntp.org, 1.us.pool.ntp.org, 2.us.pool.ntp.org, etc, the pool numbered 2 is the one that supports IPv6).

Specifically, this is how I configure the 2.us.pool.ntp.org pool in /etc/ntp.conf:

pool -6 2.us.pool.ntp.org iburst minpoll 10 maxpoll 12

The -6 flag means to use IPv6; the iburst part is supposed to help speed up initial synchronization; the minpoll 10 part means to poll no more often than every 2^10 seconds (around 17 minutes); and the maxpoll 12 part means to poll no less often than every 2^12 seconds (around 68 minutes).

Also, if you set up a restrictive Network ACL for your private subnet, you'll need to allow inbound access to UDP port 123.

AWS APIs

If you are planning to directly call AWS APIs (either through the various per-language SDKs, or the CLI), a huge gotcha is that very few AWS services as of yet provide IPv6 endpoints. This means that you won't be able to use most AWS services at all from within your private IPv6 subnet (with the exception of services that consist of instances that themselves reside within your VPCs, like RDS; rather than endpoints hosted outside of your VPCs, like DynamoDB).

The only major AWS service I've tried that does support IPv6 through its APIs is S3. When connecting to it via CLI, you can get it to use IPv6 by explicitly specifying the "dualstack" endpoint via command-line flag, like this:

aws --endpoint-url https://s3.dualstack.us-east-1.amazonaws.com --region us-east-1 s3 ls

Or, alternately, you can enable IPv6 usage via the AWS config file (~/.aws/config), like this:

[default] region = us-east-1 s3 = use_dualstack_endpoint = true addressing_style = virtual

Ansible Inventory

To access EC2 instances in a private subnet, typically you'd use a VPN running in a public subnet of the same (or bridged) VPC, with the VPN client set to route the VPC's private IPv4 block through the VPN. For IPv6, I also have my VPN set to route the VPC's IPv6 block through the VPN, too.

Using Ansible through a VPN with IPv4 is pretty much as simple as configuring Ansible's ec2.ini file to set its destination_variable and vpc_destination_variable settings to private_ip_address. But since I decided to disallow any IPv4 access to my private subnets (even from other subnets within the same VPC), I had to jump through a few extra hoops:

1. Custom Internal Domain Names

I use a custom internal domain name for all my servers (I'll use example.net as the custom domain in the following examples), and assign each server its own domain name (like db1.example.net or mail2.example.net, etc). When I launch a new EC2 server, I create a DNS AAAA record for it (via Route53), pointing the DNS record to the IPv6 address of the newly-launched server. In this way I can use the DNS name to refer to the same server throughout the its lifetime.

I also tag my EC2 instances as soon as I launch them with tags from which the DNS name can be constructed. For example, I'd assign the server with the DNS name of fe3.example.net a "node" tag of fe and a "number" tag of 3.

2. SSH Config

In my SSH config file (~/.ssh/config), I have an entry like the following, to make sure SSH (and Ansible) only tries to access my EC2 instances through IPv6:

Host *.example.net AddressFamily inet6

3. Ansible EC2 Config

With the above two elements in place, I can then enable the destination_format (and destination_format_tags) settings in the Ansible ec2.ini configuration file to direct Ansible to use DNS names instead of IP address for EC2 inventory. With the "node" and "number" tags described above, I can use the following configuration in my ec2.ini file:

destination_format = {0}{1}.example.net destination_format_tags = node,number

When the above is set up correctly, you can run the ec2.py script (eg as ./ec2.py), and see your DNS names in its output (like db1.example.net or mail2.example.net, etc), instead of IPv4 addresses. And when you run an ad-hoc Ansible module (like ansible fe3.example.com -i ec2.py -m setup) everything should "just work".

OpenDKIM Key Retrieval Failed

I set up OpenDKIM on my mailserver years ago, but while I got it working for signing just fine, I could never get it working for verification. Whenever I'd receive a message signed with DKIM, I'd see an error message like this in my mailserver logs:

May  4 01:20:20 mail opendkim[24874]: 7EE5982132: key retrieval failed (s=20161025, d=gmail.com): '20161025._domainkey.gmail.com' query timed out

When I tried dig on the DNS record listed in the logs, however, I was able to retrieve it just fine:

dig TXT 20161025._domainkey.gmail.com

Recently I set aside some time to "dig" into it further. I found I could use the opendkim-testkey command to at least reproduce the issue (instead of having to keep sending test emails from other accounts to myself). For example, the following command tries to retrieve the DKIM key from the 20161025._domainkey.gmail.com DNS TXT record (20161025 is the DKIM selector and gmail.com is the signing domain):

opendkim-testkey -s 20161025 -d gmail.com

This command gave me the same "query timed out" error message that I saw in my logs. Through a lot of trial and error, I figured out that I could avoid the error by setting the Nameservers property in my /etc/opendkim.conf file to an external DNS server (any external one will do), and restarting the OpenDKIM daemon. I've been running my mailserver on an Ubuntu EC2 instance, and apparently OpenDKIM does not like something about the combination of the Ubuntu DNS resolver and the internal EC2 DNS servers.

So, I added this line to my /etc/opendkim.conf, restarted the OpenDKIM daemon, and now I no longer see the "key retrieval failed" error message in my logs (instead I get a nice Authentication-Results header added by OpenDKIM to my incoming mail)!:

Nameservers 1.1.1.1

1.1.1.1 is Cloudflare's DNS servers — but any external DNS servers should work. One thing to keep in mind with using external DNS servers is that if your network has a stateless firewall, you need to allow inbound access to UDP in the "ephemeral" port range. If you're using EC2's Network ACLs (and not just using the defaults), this means adding a rule like the following to the ACL for the subnet in which your mailserver lives (32768-61000 is Ubuntu's ephemeral port range):

Rule #: [any number lower than your DENY rules]
Type: Custom UDP Rule
Protocol: UDP (17)
Port Range: 32768-61000
Source: 1.1.1.1/32
Allow/Deny: ALLOW

Jenkins EC2 Slave Plugin

Just finished configuring our Jenkins build machine to use the Jenkins EC2 Plugin (currently at version 1.18), which allows the Jenkins server to spin up AWS EC2 instances on demand to use as slaves for selected build jobs. It's very cool, and requires only a little bit of configuration to get it running your existing jobs automatically on EC2 slaves. These were the steps I had to take:

Create an IAM user for Jenkins

The Jenkins server needs access to your AWS account in order to run and kill EC2 instances; the best way to enable this is to create a separate IAM user that is used only by Jenkins, and has only the minimum permissions required. I used the IAM section of the AWS Console for this (although you can also do it via command line). When you create the new IAM user, create an access key for the user, and make sure you save the secret key part of it, since AWS does not store the secret key (you have to generate a new access key if you lose the secret part of it). You will use this access key when you configure the Jenkins EC2 plugin.

Once you create the user, you need to attach a policy to it that will allow the user to run and kill EC2 instances. Via trial-and-error, I found that these were the minimum permissions currently required by the Jenkins EC2 plugin (I limited it to a single region, us-west-2; if you want to allow Jenkins to manage instances in all regions, remove the ec2:Region condition):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeRegions"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:GetConsoleOutput",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:Region": "us-west-2"
                }
            }
        }
    ]
}

Create an SSH key pair for Jenkins

Once the Jenkins server starts a new EC2 slave, it will connect to it via SSH, using public-key authentication. At minimum, you need to provide Jenkins with the private key of a user account that has password-less sudo privileges on the slave; the easiest way to do this with a stock AMI image is to just let Jenkins use the image's default account via the EC2-registered key-pair used to boot it.

I just used the stock Ubuntu AMI (12.04, 64-bit intance-store variety), where the default user (ubuntu) has password-less sudo privileges. And I created an SSH key-pair for Jenkins to use via the EC2 section of the AWS Console (although you can also do it via the command line). As with the secret part of the IAM user's access key, make sure you save the private key for the key pair, since AWS does not store it (you'll have to generate a new key pair if you lose the private key).

Create an EC2 security-group for the slave

Whenever you launch an EC2 instance, you must specify the "security group" to which the instance belongs. This is basically just a set of firewall rules for the built-in EC2 firewall that restrict inbound access to a limited set of ports (optionally from a limited set of hosts). You can change the firewall rules for a group even while that group has running instances, but you can't switch an instance to use a different group once the instance has been launched. So I recommend creating a separate group just for your Jenkins slaves, even if the group has the same rules as you use for other security groups (so that you can change the rules for the different groups independently, should the need ever arise).

The only port you'll need inbound access to on the slaves is 22 (SSH). Optionally, you can set the source for that rule to the IP address of the Jenkins master server (if you want to disallow any inbound access to the slave other than from the master).

Create an init script for the slave

When Jenkins starts a new EC2 slave (and once the slave has booted and Jenkins connects to it), Jenkins will run a script that you specify to prepare the slave to run builds. Jenkins automatically will bootstrap the Jenkins-specific environment on the slave (once your init script has run), so you really only need to setup a few things, like java and git (or svn, hg, etc).

Following is the init script I used with the stock Ubuntu AMI. It installs a few things via Apt; installs a few particular versions of Grails that we use; and then installs and runs the latest version of PhantomJS (which we need to have running with some specific arguments for our functional tests). It also sets-up a working directory for Jenkins to use as the ubuntu user at /srv/jenkins; creates a bigger swap file than comes with the Ubuntu AMIs; and moves the /srv, /var, and /tmp directories to the faster "ephemeral" drive on the EC2 instance (mounted on the smaller EC2 instances at /mnt):

#!/bin/sh

# keep trying to install until it works
for I in 1 2 3; do
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q update
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q upgrade
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q install \
        git-core \
        groovy \
        openjdk-6-jdk \
        vim \
        zip
    sleep 1
done

# fix missing java links
if [ ! -e /usr/lib/jvm/default-java ]; then
    sudo ln -s /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/default-java
fi
if [ ! -e /usr/lib/jvm/java-6-openjdk ]; then
    sudo ln -s /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/java-6-openjdk
fi

# download grails if necessary
if [ ! -e /opt/grails-2.1.0 ]; then
    # download grails zip
    if [ ! -e "/tmp/grails.zip" ]; then
        wget -nd -O /tmp/grails.zip http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-1.3.7.zip
    fi
    # unzip grails
    unzip /tmp/grails.zip
    # move to /opt
    sudo mv grails-1.3.7 /opt/.

    # download grails zip
    if [ ! -e "/tmp/grails2.zip" ]; then
        wget -nd -O /tmp/grails2.zip http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-2.1.0.zip
    fi
    # unzip grails
    unzip /tmp/grails2.zip
    # move to /opt
    sudo mv grails-2.1.0 /opt/.

    # expand max-memory size for grails
    sudo perl -pli -e 's/Xmx\d+/Xmx2048/; s/MaxPermSize=\d+/MaxPermSize=1024/' /opt/grails*/bin/startGrails
fi

# download phantomjs if necessary
HAS_PHANTOMJS=`whereis phantomjs | grep bin`
if [ -z "$HAS_PHANTOMJS" ]; then
    # download phantomjs binary
    if [ ! -e "/tmp/phantomjs.tar.bz2" ]; then
        wget -nd -O /tmp/phantomjs.tar.bz2 https://phantomjs.googlecode.com/files/phantomjs-1.9.1-linux-x86_64.tar.bz2
    fi
    # unzip phantomjs
    tar xf /tmp/phantomjs.tar.bz2
    # move to /opt
    sudo mv phantomjs-1.9.1-linux-x86_64 /opt/.
    sudo ln -s /opt/phantomjs-1.9.1-linux-x86_64 /opt/phantomjs
    # run in background
    nohup /opt/phantomjs/bin/phantomjs \
        --webdriver=7483 \
        --webdriver-logfile=webdriver.log \
        --ignore-ssl-errors=true \
        &>phantomjs.log &
fi

# creating jenkins working directory
sudo mkdir /srv/jenkins && sudo chown ubuntu:ubuntu /srv/jenkins

# add more swap
SWAPFILE=/mnt/swap1
if [ ! -f $SWAPFILE ]; then
    # creates 2G (1M * 2K) /mnt/swap1 file
    sudo dd if=/dev/zero of=$SWAPFILE bs=1M count=2K
    sudo chmod 600 $SWAPFILE
    sudo mkswap $SWAPFILE

    # add new swap to config and start using it
    echo "$SWAPFILE none swap defaults 0 0" | sudo tee -a /etc/fstab
    sudo swapon -a
fi

# move /tmp to larger/faster/transient /mnt volume
if [ ! -e /mnt/tmp ]; then
    # move each directory, then create link from old location to new
    for DIR in /srv /tmp /var; do
        echo $DIR
        sudo mv $DIR /mnt$DIR
        sudo mkdir $DIR
        echo "/mnt$DIR  $DIR    none    bind    0   0" | sudo tee -a /etc/fstab
        sudo mount $DIR
    done
fi

Among a few quirky things that the script does is re-try apt-get three times — I found that at least half the time the first run of apt-get would fail to update/install any packages without any comprehensible error messages (I think maybe because the system was still lazy-initializing some components?). Re-running it after a second seems to solve the issue, however.

Also, while installing only the openjdk-6-jdk package installs enough of java 6 for our build purposes, it doesn't install some of the links in the /usr/lib/jvm directory through which we reference the jdk or java home in our build scripts (these links usually are created by some other unknown java packages); so the script manually creates these java links.

And finally, the default JVM memory settings used by grails aren't sufficient for running or testing our apps; the quick fix for this is just to overwrite the Xmx and MaxPermSize settings in the default GRAILS_OPTS of the startGrails script that comes with each version of grails.

Configure the Jenkins EC2 plugin

Now, finally, you're ready to configure the EC2 plugin itself. Once you've installed the plugin, you navigate to the main "Manage Jenkins" > "Configure System" page, and scroll down near the bottom to the "Cloud" section. There, you click the "Add a new cloud" button, and select the "Amazon EC2" option. This will display the UI for configuring the EC2 plugin. The first things you configure are the access key that you created for the Jenkins IAM user (via the "Access Key ID" and "Secret Access Key" fields). If you've configured the permissions for the Jenkins IAM user correctly, this will populate the "Region" dropdown, and allow you to select the AWS region to use. Next, paste in the text from the secret-key pem file for the SSH key pair that you created for Jenkins in the "EC2 Key Pair's Private Key" field (this text will start with the line "-----BEGIN RSA PRIVATE KEY-----").

In the "AMIs" section of the configuration UI, you'll configure the per-slave settings. If you want, you can generate multiple slave profiles (from different AMI images, or different instance sizes, or different initialization parameters, etc). But you can start with just one, which you can add by clicking the "Add" button at the bottom of the "AMIs" section.

For each slave profile, you configure the profile with a description ("Standard EC2 Slave"), as well as the ID of the AMI to use (I used the ami-5168f861 AMI, the current official Ubuntu 12.04, 64-bit instance-store AMI). Next, select the instance type; a "micro" instance is probably too small for just about anything other than a one-line shell job; a "small" instance may be fine for some jobs; but with most of our builds (which all include at least one grails build step), we get a 3-4x improvement over "small" instance times with a "medium" instance (which is 2x the price of a "small").

You can optionally select a specific availability-zone within your selected region in which to launch the slave; this matters a lot if you have an existing EBS volume to which you will attach to the slave, and it matters a little if you have other EC2 instances which the slave is going to access (like if the master Jenkins server is in EC2, or if you have a version-control repo in EC2 from which the slave is going to download source code, etc); otherwise you can just leave the "Availability Zone" field blank, and AWS automatically will launch the slave in whatever zone of your configured region that is least active when the slave is launched.

Enter the name of the EC2 security-group that you created for the slave in the "Security group names" field. Enter the path to the directory that Jenkins should use as its working directory (similar to the /var/lib/jenkins directory on the master); you probably created this in your slave init script (it should be writable by the user Jenkins uses on the slave; the directory that my init script created for this was /srv/jenkins). Specify the user Jeknins should run as on the slave in the "Remote user" field; on a stock Ubuntu AMI, this is ubuntu. And unless you use root for this user, enter sudo in the "Root command prefix" field.

Enter one or more labels (separated by spaces) in the "Labels" field. You can assign jobs to specific slave profiles via labels, so if you have multiple slave profiles, you may want to include a label for each distinguishing feature of the slave. For example, if you have one profile for a small instance with no DB, you might label it just "small"; and for another profile using a medium instance with a MySQL DB, you might label as "medium mysql". Then for a job that only needs a small instance, you can set the job to use the slave labeled "small"; and for a job that requires a medium instance, you can set it to use the slave labeled "medium"; and for a job that requires MySQL, you can set the job to use the slave with the "mysql" label. If you have only one slave profile, you can just use a simple label like "slave" or "ec2" (and then configure any job that you want to run on the slave with the "slave" or "ec2" label).

The default "Usage" setting for both the master Jenkins server and each slave server is to "Utilize this slave as much as possible". This means that Jenkins will not boot a slave for a job unless the job specifically has been configured (via label) to use a slave that currently is not running, or unless all the build-executors on the master currently are in use. If you instead change the slave's "Usage" setting to "Leave this machine for tied jobs only", Jenkins will use the slave only if it can't run the job on the master or any other slaves. See the "Usage" settings matrix table below for a clearer description of the interaction of the "Usage" setting between slaves and the master.

Set "Idle termination time" to the number of minutes a slave must sit at idle before Jenkins shuts it down (default is 30 minutes — if you use the slaves only for scheduled jobs you might want to cut this down to 0). Paste your init script into the "Init script" field. Jenkins executes the init script by writing your script to a file to the slave, setting execute permissions on it, and then running the file as the user you specified in the "Remote user" field — so make sure that you include #!/bin/sh at the top of the script (if it's a shell script, or the appropriate "sha-bang" if you you a different scripting language).

Click the "Advanced..." button at the bottom of the AMI's configuration section to access a few more options. A couple of settings that you may want to customize are "Number of executors" (you may want to set this to 1 — unless you intend to run multiple jobs on the same slave at the same time); and "Instance Cap", which is the maximum number of slaves from this profile that Jenkins can have running at the same time.

Click the "Save" button at the bottom of the page to save and apply your changes.

Configure the job-selecting behavior of the master

Once you've saved your first slave profile, go back to the same "Manage Jenkins" > "Configure System" page, and at the top of the page you'll find a new "Labels" and a new "Usage" field (just below the "# of executors" field). The "Usage" field determines how Jenkins utilizes the master for jobs. If you leave the "Utilize this slave as much as possible" option selected, whenever a job is triggered that either doesn't have a label, or is labeled so that it could be executed either by the master or by another slave, Jenkins will run the job on an already-running slave only if the slave has a free executor; but otherwise it will try to run the job on the master, and only boot a new slave if the master has no free executors. If you change the "Usage" field to "Leave this machine for tied jobs only", whenever a job is triggered that either doesn't have a label, or is labeled so that it could be executed either by the master or by another slave, Jenkins will first try a running slave, and then try booting a slave; and only try using the master if it can't boot any more slaves.

Here's a description of the interaction between the master and slave "Usage" settings in tabular form:

"Usage" setting matrix

		Master
		Utilize this slave as much as possible	Leave this machine for tied jobs only
Slave	Utilize this slave as much as possible	use executor on running slave if free use executor on master if free boot new slave if below instance cap wait	use executor on running slave if free boot new slave if below instance cap use executor on master if free wait
	Leave this machine for tied jobs only	use executor on master if free use executor on running slave if free boot new slave if below instance cap wait	use executor on running slave if free use executor on master if free boot new slave if below instance cap wait

Note that if a job is labeled with a label that the master doesn't have, Jenkins will not run it on the master regardless of the "Usage" setting — it will wait until it can boot or otherwise free up an executor on a slave that does have that label. So if there are jobs that you do want to run on the master, do give the master a label or two via the "Labels" field. For example, if you want to allow only small jobs and one or two master-specific jobs to run on the master, you might label the master "small master".

Earmark jobs for slaves with labels

The last step is to re-configure individual jobs to run on your new slaves. You can skip this step entirely if any slave can take any job — just configure the "Usage" setting of the master and slaves to indicate how Jenkins should utilize the slaves (if it should boot slaves to run jobs on them even if the master is free, or if it should max out the master before booting slaves).

Otherwise, navigate to the configuration page of each job for which you want to specify the type of slave to run the job, and check the "Restrict where this project can be run" checkbox at the bottom of the first section of the page. This will reveal the "Label expression" field; enter the label (or space-separated labels) that defines what kind of slave the job requires. For example, if the job requires a MySQL DB, you might enter "mysql" as the label (requiring a slave with the "mysql" label); or if the job can run on any small instance, you might enter "small" as the label (requiring a slave with the "small" label).

The next time you trigger a job that is labeled for a slave, Jenkins automatically will boot the slave (if no slaves of that type are currently running and have free executors), and run the job. The display of Jenkins' leftnav will also change, to include the list of executors on the slave in the "Build Executor" box (and when the slave is terminated, the slave's executors will be removed from this box).

EC2 Reserved Instance "Break Even" Points

Updated July 21, 2013 with the latest EC2 and RDS prices:

While trying to figure out the best deal for some EC2 instances we have running continuously (and inspired by Jordan Sissel's EC2 Reserved vs. On-Demand Chart), I charted the EC2 pricing curves for the various reserve options (light/medium/heavy utilization and 1- or 3-year terms). With reserve pricing, you're basically buying the right to run a specific instance type at a reduced hourly rate for a 1-year or 3-year term. For example, if you buy one "light-utilization" 1-year term for a small EC2 instance in your us-east-1a availability-zone, you pay $61 dollars now; and whenever you run at least one small instance in us-east-1a during the next 365 days, Amazon will automatically apply the reduced hourly rate to your usage for one of those instances (billing you at $0.034 an hour instead of $0.060 an hour for the instance).

Note that the "heavy" plans are a little bit different than the "light" and "medium" plans — Amazon will bill you each month for the full compliment of hours under the "heavy" plan, regardless of whether or not you actually use all the hours. In contrast, with the "light" and "medium" plans, Amazon will bill you only for the hours during which you had an applicable instance running. For example, if you've bought a "light" 1 year plan, and run an applicable instance for 100 hours in a 30-day month, Amazon will bill you $3.40 ($0.034 x 100 hours). But if you've bought a "heavy" 1 year plan, and run the same instance the same 100 hours, Amazon will bill you $10.08 ($0.014 x 24 hours x 30 days) instead.

Amazon's characterization of their reserve options as "light utilization", "medium utilization", and "heavy utilization" is also confusing, since you can use a "light utilization" plan heavily, and a "heavy utilization" plan lightly — you just have to think of the names as suggestions for how the plans might be most useful to you, rather than as restrictions on their use. And when you're using a number of instances of a certain type all the time, what really matters is how long you expect to continue to use that same number of instances in the future: one month, two months, a year, etc.

The Chart

So this is how I use the chart I made (although it shows the "small-instance" costs for the us-east-1 zone, all the EC2 reserve pricing follows the same basic curves): I figure out how likely we are to continue using an instance type (either in its current role, or in some future role) for which we're currently paying on-demand prices, try to put that likelihood into months (like "there's a good chance I'll still be using it in 2 months, and probably will be in 6 months, but I might not be in a year"), and then take a look at where those months fall on the chart:

The X axis is months, and the Y axis is dollars. Here's the legend:

Option Name	Term Years	$ Upfront	$ Hourly
On Demand	0	000.00	0.060
Light 1yr	1	061.00	0.034
Light 3yr	3	096.00	0.027
Medium 1yr	1	139.00	0.021
Medium 3yr	3	215.00	0.017
Heavy 1yr	1	169.00	0.014
Heavy 3yr	3	257.00	0.012

You can view the full chart separately (it's a javascript chart with fancy tooltips built using elycharts). The chart shows the cumulative cost after each month; for example, the first light-green dot shows that after one month of running a small instance 24/7, you'll have spent a total of $85.82 under the light-utilization 1-year term plan (from the upfront cost of $61, plus the hourly cost of $0.034 for one month). I modeled the "heavy" options as having a fixed price over their term — while technically their price isn't all upfront (Amazon bills you for the hourly component of those plans month-by-month), you'll eventually have to pay for all their hours, regardless of whether you use them or not (so your total cost is the same whether you use 1 hour or 10,000).

The Bottom Line

The chart shows that your best deal for an instance that you use 24/7 is "on demand" if you use it less than three months — but if you use it any longer, you would have been better off buying a "light utilization" plan. Here's a table of the best deal for a each length of time:

Months	Best Option
00 - 03	On Demand
03 - 07	Light 1yr
07 - 10	Light 3yr
10 - 12	Heavy 1yr
12 - 16	Light 3yr
16 - 29	Medium 3yr
29 - 36	Heavy 3yr

As an even rougher rule of thumb: less than half-a-year, "on demand"; half-a-year to a year-and-a-half, "light 3yr"; and over a year-and-a-half, "medium 3yr".

If you do actually use the heavy-utilization 3-year term plan for the full three years, it's a nice deal that can save you almost two thirds of what you'd pay for "on demand" over that time period (and not much more than what you'd pay a low-end VPS-hosting provider for a VPS with similar specs over the same time period) — but if you use it for less than half that time, you'll wish you had chosen any other option. Also, keep in mind that each reserved instance you buy is applicable only to a specific instance-type in a specific availability-zone — so if halfway through your term you stop using so many instances of that type, or move your usage of those instances to a different region, you'll be stuck with reserved instances you never use.

Other AWS Reserved Instances

I also included a chart for RDS instances on the same page as the full chart of EC2 instance pricing (below the EC2 chart). It looks quite similar to the EC2 chart, but if you look closely you'll see that the "break-even" points are slightly different: "light 3yr" becomes a better option than "on demand" around 3 months instead of 4, and as good as "light 1yr" in just 4 months; and "medium 3yr" becomes the best deal at just 14 months. I imagine the reserve options for other AWS services will be similar — same curves, slightly different deals.

EC2 Sudo: Unable to Resolve Host

Sometimes after changing elastic-IP settings or stopping/starting instances on EC2, I get an irritating error like this when I execute a command with sudo:

sudo: unable to resolve host domU-12-34-ab-cd-56-78

The fix is to lookup the instance's private dns name (via ec2-describe-instances or the AWS console ui) and update the hostname on the instance with the first segment of that DNS name (which is something that looks like ip-12-34-56-78 or domU-12-34-ab-cd-56-78). On ubuntu, this is what you need to do (assuming ip-12-34-56-78 is the new hostname):

$ sudo hostname ip-12-34-56-78 $ echo 'ip-12-34-56-78' | sudo tee /etc/hostname

The first line will set the hostname until you reboot; and the second line will configure the hostname to use once you do reboot.

Getting Started with ELB

We recently started using Amazon's Elastic Load Balancing (ELB) service for our webapp (which is hosted in AWS). We're pretty happy with it so far; although we initially tried using Cherokee as our web server, that didn't seem to work quite right with ELB: About every few hours or so, we'd get a blank page back from the load balancer. I suspect it's a version of a similar issue between ELB and Apache that Amazon fixed back in May 2010. We switched over to using Apache for our web server, and that seemed to fix it, so we didn't dig any further.

One other little disappointing thing about ELB is that it can't serve "naked" domains (ie example.com instead of www.example.com), so we still have a single EC2 instance with a fixed ip address to which we point the DNS record for our naked domain name (and which our web server redirects to our "www" subdomain). Shlomo Swidler has a real nice blog post on why this is, and generally how ELB works (and how to test it).

Quick Start

The AWS docs already have a pretty quick, concise set of steps to setup an ELB for your web servers, so here's a little bit of a higher-level overview of what you have to do:

1 Create the Load Balancer

The only thing that's set in stone when you create the balancer is its name, which is the id you use to reference it, and also becomes the first segment of its dns name; and the region it's in. However, when you create the balancer, you also have to specify at least one port to listen on and at least one availability-zones among which to balance (although you can change both any time you want).

A load balancer can balance among all the availability zones of the region in which you create it. You do have to specifically enable each zone, however. If you don't register with the balancer any of your app's instances from any one particular availability-zone, the balancer will just ignore that zone (ie it won't try to balance to empty zones). You'll want to make sure you have an equivalent number of instances in each zone that you have any instances in, though, since the balancer doesn't adjust its balancing policy to account for the number of instances you have registered in a zone — it will just assume that it can balance to each zone equally.

If you're serving straight HTTP, you'll want to configure the balancer's listener to use the HTTP protocol (the other option is TCP, for everything else). You can configure the balancer to listen on one port and route to another port, if you want (like to listen on port 80, but route to port 8080 on your app instances).

2 Configure the Ping URL

Once you've create the balancer, the next step is to configure it with the URL of your app that the balancer should ping for each instance, to check if the instance is still healthy. You can actually specify a different port for the ping URL, so you could theoretically have the ELB ping a completely different application than the app is balancing (it just has to be some process listening on your instance). But you probably just want it to be some URL in your app that serves a stripped-down (or even empty) HTML page.

Also note that, as long as you specify the HTTP protocol for the ping URL, the ping URL must return an HTTP response status of 200 for the balancer to consider it to be OK. Anything else — including a 30x redirect to some other page — and the balancer will consider it to be unhealthy.

And when you configure the ping URL, you also configure the interval between pings (in seconds), and the max number of seconds in which your server must respond before the balancer times-out the ping. Also, you must set a threshold for the number of good pings before the instance is added to the group balanced by the ELB, and the number of bad pings before it is removed; AWS requires these threshold values to be at least 2. A pretty standard setting for this is to make the interval 30 seconds, the timeout 10 seconds, and the healthy and unhealthy thresholds 2 — so if an instance starts going sideways, the ELB will continue to balance to it for at most a minute and a half before it recognizes that it's gone bad.

3 Configure Sticky Sessions (or Not)

If you need the balancer to do "sticky" sessions (balance to the same instance over the life of your application's sesssion — for example because you have per-session state that's not shared among instances), you need to configure the ELB to use sticky sessions keyed off your application's session cookie. For example, if you're using a standard java servlet engine (like tomcat or jetty), you'd configure the ELB to use the JSESSIONID cookie to route all requests for the same session to the same instance. And note that with the ELB API, this is a two step process — the first step is to create a stickyness "policy", and the second step is to apply it (individually to each listener that needs to use the policy).

If you don't need sticky sessions, then you don't need to do anything special (ELB will do round-robin by default).

4 Register Web-Server Instances

The last step to get going is simply to add your app instances to the load-balancer. The balancer will balance only among instances in the availability zones you specified when you created the balancer (unless/until you separately enable/disable those zones). Note that you should add an equivalent number of instances to each zone to which you've added any instances, as the ELB will try to balance equally among all zones in which you have healthy instances.

5 Update Your DNS Records

The real last step, of course, is to point actual live traffic to the ELB. AWS gives you a unique (and unpredictable) DNS name for the balancer (like my-elb-name-123456789.us-east-1.elb.amazonaws.com; the DescribeLoadBalancers API response gives you it). The DNS name never changes over the life of the instance, though. So to direct your traffic through the ELB, you create/update CNAME records for all the subdomains you want the balancer to serve (ie www.example.com, www.example.org, etc), pointing each record to the balancer's AWS DNS name. As the DNS update propagates across the tubes, your users will start using the ELB.