Send Journald to CloudWatch Logs with Vector

Timber's Vector log collection tool is a nifty Swiss Army knife for collecting and shipping logs and metrics from one system to another. In particular, I think it's the best tool for shipping structured journald events to CloudWatch Logs.

Here's how to start using Vector to send journald log events to CloudWatch:

Grant Permissions to EC2 Roles

In order to push logs (or metrics) from your EC2 instances to CloudWatch, you first need to grant those EC2 instances some CloudWatch permissions. The permissions you need are basically the same as the AWS CloudWatch Agent needs, so just follow the Create IAM roles and users for use with the CloudWatch agent tutorial to assign the AWS-managed CloudWatchAgentServerPolicy to the IAM roles of the EC2 instances from which you plan on shipping journald logs.

The current version of the CloudWatchAgentServerPolicy looks like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData", "ec2:DescribeVolumes", "ec2:DescribeTags", "logs:PutLogEvents", "logs:DescribeLogStreams", "logs:DescribeLogGroups", "logs:CreateLogStream", "logs:CreateLogGroup" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ssm:GetParameter" ], "Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*" } ] }

With the Vector configuration described below, however, you actually only need to grant the logs:PutLogEvents, logs:DescribeLogStreams, logs:DescribeLogGroups, logs:CreateLogStream, and logs:CreateLogGroup permissions to your EC2 roles.

Install Vector

Installing Vector is easy on Linux. Timber maintains their own deb repo for Vector, so on a Debian-based distro like Ubuntu, you can just update the system's APT package manager with the Vector signing-key and repo, and install the Vector package:

$ wget https://repositories.timber.io/public/vector/gpg.3543DB2D0A2BC4B8.key -O - | sudo apt-key add - $ cat <<EOF | sudo tee /etc/apt/sources.list.d/timber-vector.list deb https://repositories.timber.io/public/vector/deb/ubuntu focal main deb-src https://repositories.timber.io/public/vector/deb/ubuntu focal main EOF $ sudo apt update $ sudo apt install vector

Configure Vector

The default Vector config file, located at /etc/vector/vector.toml, just includes a sample source and sink, so you can replace it entirely with your own config settings. This is the minimum you need to ship journald logs to CloudWatch:

[sources.my_journald_source] type = "journald" [sinks.my_cloudwatch_sink] type = "aws_cloudwatch_logs" inputs = ["my_journald_source"] compression = "gzip" encoding.codec = "json" region = "us-east-1" group_name = "myenv" stream_name = "mysite/myhost"

Replace the CloudWatch region, group_name, and stream_name settings above with whatever's appropriate for your EC2 instances.

Restart Vector

In one terminal screen, watch for errors by tailing Vector's own log entries with the journalctl -u vector -f command, and in another terminal restart Vector with the sudo systemctl restart vector command. If everything works, this is what you'll see in Vector's own logs:

$ journalctl -u vector -f Jun 11 19:54:02 myhost systemd[1]: Started Vector. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.008 INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info" Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.008 INFO vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.008 INFO vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.010 INFO vector::app: Loading configs. path=[("/etc/vector/vector.toml", Some(Toml))] Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.060 INFO vector::topology: Running healthchecks. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.060 INFO vector::topology: Starting source. name="journald" Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.061 INFO vector::topology: Starting sink. name="aws_cloudwatch_logs" Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.061 INFO vector: Vector has started. version="0.14.0" arch="x86_64" build_id="5f3a319 2021-06-03" Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.062 INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.063 INFO journald-server: vector::sources::journald: Starting journalctl. Jun 11 19:54:03 myhost vector[686208]: Jun 11 19:54:03.128 INFO vector::sinks::aws_cloudwatch_logs: Skipping healthcheck log group check: `group_name` will be created if missing. Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.430 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs: Sending events. events=4 Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.453 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs::request: Log group provided does not exist; creating a new one. Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.489 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs::request: Group created. name=myenv Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.507 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs::request: Stream created. name=mysite/myhost Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.523 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs::request: Putting logs. token=None Jun 11 19:55:04 myhost vector[686208]: Jun 11 19:55:04.560 INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=0}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49610241853835534178700884863462197886393926766970915618")

If something went wrong, Vector will output some error messages (these are especially helpful as you add transformation steps to your basic Vector configuration).

Check Your CloudWatch Logs

Vector will have also shipped some logs to CloudWatch, so check them now. If you use a command-line tool like Saw, you'll see some log events like this:

$ saw watch myenv --expand --prefix mysite [2021-06-11T13:00:27-07:00] (myhost) { "PRIORITY": "6", "SYSLOG_FACILITY": "3", "SYSLOG_IDENTIFIER": "uwsgi", "_BOOT_ID": "6cb87d254d3742728b4fe20e746bcbe6", "_CAP_EFFECTIVE": "0", "_CMDLINE": "/usr/bin/uwsgi /etc/myapp/uwsgi.ini", "_COMM": "uwsgi", "_EXE": "/usr/bin/uwsgi-core", "_GID": "33", "_MACHINE_ID": "ec2aff1204bfae2781faf97e68afb1d4", "_PID": "363", "_SELINUX_CONTEXT": "unconfined\n", "_STREAM_ID": "aa261772c2e74663a7bb122c24b92e64", "_SYSTEMD_CGROUP": "/system.slice/myapp.service", "_SYSTEMD_INVOCATION_ID": "b5e117501bbb43428ab7565659022c20", "_SYSTEMD_SLICE": "system.slice", "_SYSTEMD_UNIT": "myapp.service", "_TRANSPORT": "stdout", "_UID": "33", "__MONOTONIC_TIMESTAMP": "511441719050", "__REALTIME_TIMESTAMP": "1623441627906124", "host": "myhost", "message": "[pid: 363|app: 0|req: 501/501] 203.0.113.2 () {34 vars in 377 bytes} [Fri Jun 11 20:00:27 2021] HEAD / =< generated 0 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 78 bytes (0 switches on core 0)", "source_type": "journald" }

With Saw, use the saw watch command to tail log events as they come in, and use the saw get command to get historical events. For example, this command will print the last 10 minutes of events using the mysite log stream prefix from the myenv log group:

$ saw get myenv --expand --pretty --prefix mysite --start -10m

Filter and Remap Your Logs

With that working, you can tune your Vector configuration to filter out log events you don't care about, and remap certain log fields into a more useful format. Let's add two "transform" steps to our /etc/vector/vector.toml file between the Journald Source and the AWS CloudWatch Logs Sink: a Filter transform, and a Remap transform:

[sources.my_journald_source] type = "journald" [transforms.my_journald_filter] type = "filter" inputs = ["my_journald_source"] condition = ''' (includes(["0", "1", "2", "3", "4"], .PRIORITY) || includes(["systemd", "uwsgi"], .SYSLOG_IDENTIFIER)) ''' [transforms.my_journald_remap] type = "remap" inputs = ["my_journald_filter"] source = ''' .app = .SYSLOG_IDENTIFIER .datetime = to_timestamp(round((to_int(.__REALTIME_TIMESTAMP) ?? 0) / 1000000 ?? 0)) .facility = to_syslog_facility(to_int(.SYSLOG_FACILITY) ?? 0) ?? "" .severity = to_int(.PRIORITY) ?? 0 .level = to_syslog_level(.severity) ?? "" ''' [sinks.my_cloudwatch_sink] type = "aws_cloudwatch_logs" inputs = ["my_journald_filter"] compression = "gzip" encoding.codec = "json" region = "us-east-1" group_name = "myenv" stream_name = "mysite/myhost"

In the above pipeline, the my_journald_source step pipes to the my_journald_transform step, which pipes to the my_journald_transform step, which pipes to the my_cloudwatch_sink step (configured via the inputs setting of each receiving step). The condition VRL expression in the filter step drops entries unless the entry's PRIORITY field is less than 5 (aka "emerg", "alert", "crit", "err", and "warning"), or unless the entry's SYSLOG_IDENTITY field is "systemd" or "uwsgi". And the source VRL program in the remap step adds some additional conveniently-formatted fields (app datetime facility severity and level) to each log entry (the ?? operator in the source coerces "fallible" expressions to a default value when they would otherwise throw an error).

Now if you restart Vector and check your CloudWatch logs, you'll see fewer unimportant entries (those with lower priorities or uninteresting sources that we filtered), plus some additional fields that we added:

$ saw watch myenv --expand --prefix mysite [2021-06-11T13:00:27-07:00] (myhost) { "PRIORITY": "6", "SYSLOG_FACILITY": "3", "SYSLOG_IDENTIFIER": "uwsgi", "_BOOT_ID": "6cb87d254d3742728b4fe20e746bcbe6", "_CAP_EFFECTIVE": "0", "_CMDLINE": "/usr/bin/uwsgi /etc/myapp/uwsgi.ini", "_COMM": "uwsgi", "_EXE": "/usr/bin/uwsgi-core", "_GID": "33", "_MACHINE_ID": "ec2aff1204bfae2781faf97e68afb1d4", "_PID": "363", "_SELINUX_CONTEXT": "unconfined\n", "_STREAM_ID": "aa261772c2e74663a7bb122c24b92e64", "_SYSTEMD_CGROUP": "/system.slice/myapp.service", "_SYSTEMD_INVOCATION_ID": "b5e117501bbb43428ab7565659022c20", "_SYSTEMD_SLICE": "system.slice", "_SYSTEMD_UNIT": "myapp.service", "_TRANSPORT": "stdout", "_UID": "33", "__MONOTONIC_TIMESTAMP": "511441719050", "__REALTIME_TIMESTAMP": "1623441627906124", "app": "uwsgi", "datetime": "2021-06-11T20:00:27Z", "facility": "daemon", "host": "myhost", "level": "info", "message": "[pid: 363|app: 0|req: 501/501] 203.0.113.2 () {34 vars in 377 bytes} [Fri Jun 11 20:00:27 2021] HEAD / =< generated 0 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 78 bytes (0 switches on core 0)", "severity": 6, "source_type": "journald" }

And we can use the new fields we added to further filter our output from Saw, as well as print compact log lines with jq:

$ saw watch myenv --raw --prefix mysite --filter '{ $.severity < 4 || $.app = "uwsgi" }' | jq --unbuffered -r '[.datetime, .level, .host, .app, .message] | join(" ")' 2021-06-11T20:00:27Z info myhost uwsgi [pid: 363|app: 0|req: 501/501] 203.0.113.2 () {34 vars in 377 bytes} [Fri Jun 11 20:00:27 2021] HEAD / =< generated 0 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 78 bytes (0 switches on core 0)

Remove Irrelevant Fields

You can also use Vector's remap filter to remove extraneous fields that you don't want to ship to and store in CloudWatch. You can use the del function to delete specific fields from each event — for example, to skip the journald fields which duplicate the custom fields we added:

source = ''' .app = .SYSLOG_IDENTIFIER .datetime = to_timestamp(round((to_int(.__REALTIME_TIMESTAMP) ?? 0) / 1000000 ?? 0)) .facility = to_syslog_facility(to_int(.SYSLOG_FACILITY) ?? 0) ?? "" .severity = to_int(.PRIORITY) ?? 0 .level = to_syslog_level(.severity) ?? "" del(.PRIORITY) del(.SYSLOG_IDENTIFIER) del(.SYSLOG_FACILITY) '''

Or you could replace the original event entirely with a new object that contains just your desired fields:

source = ''' e = {} e.app = .SYSLOG_IDENTIFIER e.cgroup = ._SYSTEMD_CGROUP e.cmd = ._CMDLINE e.facility = to_int(.SYSLOG_FACILITY) ?? 0 e.gid = to_int(._GID) ?? 0 e.host = .host e.message = .message e.monotime = to_int(.__MONOTONIC_TIMESTAMP) ?? 0 e.pid = to_int(._PID) ?? 0 e.realtime = to_int(.__REALTIME_TIMESTAMP) ?? 0 e.datetime = to_timestamp(round(e.realtime / 1000000 ?? 0)) e.severity = to_int(.PRIORITY) ?? 0 e.level = to_syslog_level(e.severity) ?? "" e.uid = to_int(._UID) ?? 0 . = [e] '''

If you change your Vector pipeline to remap events like the above and restart it, you'll now see log events with only the following fields shipped to CloudWatch:

$ saw watch myenv --expand --prefix mysite [2021-06-11T13:00:27-07:00] (myhost) { "app": "uwsgi", "cgroup": "/system.slice/myapp.service", "cmd": "/usr/bin/uwsgi /etc/myapp/uwsgi.ini", "datetime": "2021-06-11T20:00:27Z", "facility": 3, "gid": 33, "host": "myhost", "level": "info", "message": "[pid: 363|app: 0|req: 501/501] 203.0.113.2 () {34 vars in 377 bytes} [Fri Jun 11 20:00:27 2021] HEAD / =< generated 0 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 78 bytes (0 switches on core 0)", "monotime": 511441719050, "pid": 363, "realtime": 1623441627906124, "severity": 6, "uid": 33 }

---

Edit 4/23/2022: As of Vector 0.21.1, the rounding shown in the to_timestamp examples is no longer fallible — but the to_timestamp function itself is. So the to_timestamp examples should now look like the following:

e.datetime = to_timestamp(round(e.realtime / 1000000)) ?? now()

Elixir AWS SDK

While AWS doesn't provide an SDK directly for Erlang or Elixir, the AWS for the BEAM project has built a nice solution for this — a code generator that uses the JSON API definitions from the official AWS Go SDK to create native Erlang and Elixir AWS SDK bindings. The result for Elixir is the nifty aws-elixir library.

The aws-elixir library itself doesn't have the automagic functionality from other AWS SDKs of being able to pull AWS credentials from various sources like environment variables, profile files, IAM roles for tasks or EC2, etc. However, the AWS for the BEAM project has another library you can use for that: aws_credentials. Here's how to use aws-elixir in combination with aws_credentials for a standard Mix project:

1. Add aws dependencies

First, add the aws, aws_credentials, and hackney libraries as dependencies to your mix.exs file:

# mix.exs defp deps do [ {:aws, "~> 0.8.0"}, {:aws_credentials, git: "https://github.com/aws-beam/aws_credentials", ref: "0.1.1"}, {:hackney, "~> 1.17"}, ] end

2. Set up AWS.Client struct

Next, set up aws-elixir's AWS.Client struct with the AWS credentials found by the :aws_credentials.get_credentials/0 function. In this example, I'm going to create a simple MyApp.AwsUtils module, with a client/0 function that I can call from anywhere else in my app to initialize the AWS.Client struct:

# lib/my_app/aws_utils.ex defmodule MyApp.AwsUtils do @doc """ Creates a new AWS.Client with default settings. """ @spec client() :: AWS.Client.t() def client, do: :aws_credentials.get_credentials() |> build_client() defp build_client(%{access_key_id: id, secret_access_key: key, token: "", region: region}) do AWS.Client.create(id, key, region) end defp build_client(%{access_key_id: id, secret_access_key: key, token: token, region: region}) do AWS.Client.create(id, key, token, region) end defp build_client(credentials), do: struct(AWS.Client, credentials) end

The aws_credentials library will handle caching for you, so you don't need to separately cache the credentials it returns — just call get_credentials/0 every time you need them. By default, it will first check for the standard AWS environment variables (AWS_ACCESS_KEY_ID etc), then for the standard credentials file (~/.aws/credentials), then for ECS task credentials, and then for credentials from the EC2 metadata service.

So the above example will work if on one system you configure the environment variables for your Elixir program like this:

# .env AWS_DEFAULT_REGION=us-east-1 AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPQRST AWS_SECRET_ACCESS_KEY=01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ/+a AWS_SESSION_TOKEN=

And on another system you configure the user account running your Elixir program with a ~/.aws/credentials file like this:

# ~/.aws/credentials [default] aws_access_key_id = ABCDEFGHIJKLMNOPQRST aws_secret_access_key = 01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ/+a

And when running the Elixir program in an ECS task or EC2 instance, it will automatically pick up the credentials configured for the ECS task or EC2 instance under which the program is running.

If you do use a credentials file, you can customize the path to the credentials file, or profile within the file, via the :provider_options configuration parameter, like so:

# config/config.exs config :aws_credentials, :provider_options, %{ credential_path: "/home/me/.aws/config", profile: "myprofile" }

Some caveats with the current aws_credentials implementation are:

1. With environment variables, you can specify the region (via the AWS_DEFAULT_REGION or AWS_REGION variable) only if you also specify the session token (via the AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN variable).
2. With credential files, the region and aws_session_token settings won't be included.

3. Call AWS.* module functions

Now you can go ahead and call any AWS SDK function. In this example, I'm going to create a get_my_special_file/0 function to get the contents of a file from S3:

# lib/my_app/my_files.ex defmodule MyApp.MyFiles do @doc """ Gets the content of my special file from S3. """ @spec get_my_special_file() :: binary def get_my_special_file do client = MyApp.AwsUtils.client() bucket = "my-bucket" key = "my/special/file.txt" {:ok, %{"Body" => body}, %{status_code: 200}} = AWS.S3.get_object(client, bucket, key) body end

For any AWS SDK function, you can use the Hex docs to guide you as to the Elixir function signature, the Go docs for any structs not explained in the Hex docs, and the AWS docs for more details and examples. For example, here are the docs for the get_object function used above:

1. Hex docs for AWS.S3.get_object/22
2. Go docs for S3.GetObject
3. AWS docs for S3 GetObject

The general response format form each aws-elixir SDK function is this:

# successful response { :ok, map_of_parsed_response_body_with_string_keys, %{body: body_binary, headers: list_of_string_header_tuples, status_code: integer} } # error response { :error, { :unexpected_response, %{body: body_binary, headers: list_of_string_header_tuples, status_code: integer} } }

With the AWS.S3.get_object/22 example above, a successful response will look like this:

iex> AWS.S3.get_object(MyApp.AwsUtils.client(), "my-bucket", "my/special/file.txt") {:ok, %{ "Body" => "my special file content\n", "ContentLength" => "24", "ContentType" => "text/plain", "ETag" => "\"00733c197e5877adf705a2ec6d881d44\"", "LastModified" => "Wed, 14 Apr 2021 19:05:34 GMT" }, %{ body: "my special file content\n", headers: [ {"x-amz-id-2", "ouJJOzsesw0m24Y6SCxtnDquPbo4rg0BwSORyMn3lOJ8PIeptboR8ozKgIwuPGRAtRPyRIPi6Dk="}, {"x-amz-request-id", "P9ZVDJ2L378Q3EGX"}, {"Date", "Wed, 14 Apr 2021 20:40:46 GMT"}, {"Last-Modified", "Wed, 14 Apr 2021 19:05:34 GMT"}, {"ETag", "\"00733c197e59877ad705a2ec6d881d44\""}, {"Accept-Ranges", "bytes"}, {"Content-Type", "text/plain"}, {"Content-Length", "24"}, {"Server", "AmazonS3"} ], status_code: 200 }}

And an error response will look like this:

iex> AWS.S3.get_object(MyApp.AwsUtils.client(), "my-bucket", "not/my/special/file.txt") {:error, {:unexpected_response, %{ body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>FJWGFYKL44AB4XZK</RequestId><HostId>G4mzxVPQdjFsHpErTWZhG7djVLks1Vu7RLLYS37XA38c6JsAaJs+QMp3bR3Vm9aKhoWBuS/Mk6Y=</HostId></Error>", headers: [ {"x-amz-request-id", "FJWGFYKL44AB4XZK"}, {"x-amz-id-2", "G4mzxVPQdjFsHpErTWZhG7djVLks1Vu7RLLYS37XA38c6JsAaJs+QMp3bR3Vm9aKhoWBuS/Mk6Y="}, {"Content-Type", "application/xml"}, {"Transfer-Encoding", "chunked"}, {"Date", "Wed, 14 Apr 2021 19:25:01 GMT"}, {"Server", "AmazonS3"} ], status_code: 403 }}}

Ecto RDS SSL Connection with Certificate Verification

It's nice and easy to connect to an AWS RDS instance with Elixir Ecto over SSL/TLS, as long as you're not worried about verifying the database server's certificate. You just add a ssl: true setting when your configure the Ecto Repo, like this snippet from a config/releases.exs file for a hypothetical "myapp":

# config/releases.exs config :myapp, MyApp.Repo, hostname: System.get_env("DB_HOSTNAME"), database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: true

That's probably good enough for most cloud environments; but if you want to defend against a sophisticated attacker eavesdropping on or manipulating the SSL connections between your DB client and the RDS server, you also need to configure your Ecto Repo's ssl_opts setting to verify the server's certificate.

Unfortunately, this is not so straightforward. You need to either write your own certificate verification function (not trivial), or use one supplied by another library — like the ssl_verify_fun.erl library.

To use the :ssl_verify_hostname verification function from the ssl_verify_fun.erl library, first add the library as a dependency to your mix.exs file:

# mix.exs defp deps do [ {:ecto_sql, "~> 3.5"}, {:ssl_verify_fun, ">= 0.0.0"} ] end

Then add the following ssl_opts setting to your Ecto Repo config:

# config/releases.exs check_hostname = String.to_charlist(System.get_env("DB_HOSTNAME")) config :myapp, MyApp.Repo, hostname: System.get_env("DB_HOSTNAME"), database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: true, ssl_opts: [ cacertfile: "/etc/ssl/certs/rds-ca-2019-root.pem", server_name_indication: check_hostname, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: check_hostname]} ]

Note the RDS server hostname (which would be something like my-rds-cluster.cluster-abcd1234efgh.us-east-1.rds.amazonaws.com) needs to be passed to the server_name_indication and check_hostname options as a charlist. The above example also assumes that you have downloaded the root RDS SSL certificate to /etc/ssl/certs/rds-ca-2019-root.pem on your DB client hosts.

I'd also suggest pulling out the generation of ssl_opts into a function, to make it easy to set up multiple repos. This is the way I'd do it with out our hypothetical "myapp" repo: I'd add one environment variable (DB_SSL) to trigger the Ecto ssl setting (with or without verifying the server cert), and another environment variable (DB_SSL_CA_CERT) to specify the path for the cacertfile option (triggering cert verification):

# config/releases.exs make_ssl_opts = fn "", _hostname -> [] cacertfile, hostname -> check_hostname = String.to_charlist(hostname) [ cacertfile: cacertfile, server_name_indication: check_hostname, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: check_hostname]} ] end db_ssl_ca_cert = System.get_env("DB_SSL_CA_CERT", "") db_ssl = db_ssl_ca_cert != "" or System.get_env("DB_SSL", "") != "" db_hostname = System.get_env("DB_HOSTNAME") config :myapp, MyApp.Repo, hostname: db_hostname, database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: db_ssl, ssl_opts: make_ssl_opts.(db_ssl_ca_cert, db_hostname)

With this verification in place, you'd see an error like the following if your DB client tries to connect to a server with a SSL certificate signed by a CA other than the one you configured:

{:tls_alert, {:unknown_ca, 'TLS client: In state certify at ssl_handshake.erl:1950 generated CLIENT ALERT: Fatal - Unknown CA\n'}}

And you'd see an error like the following if the certificate was signed by the expected CA, but for a different hostname:

{bad_cert,unable_to_match_altnames} - {:tls_alert, {:handshake_failure, 'TLS client: In state certify at ssl_handshake.erl:1952 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,unable_to_match_altnames}'}}

Using Logstash to Ingest CloudFront Logs Into Elasticsearch

Elasticsearch can be a good way of monitoring usage of your AWS CloudFront websites. There are some fairly straightforward paths to shipping CloudFront logs to hosted Elasticsearch services like Logz.io or Amazon Elasticsearch. Here's how to do it with your own self-hosted Elasticsearch and Logstash instances:

1. Set up CloudFront logging
2. Set up SQS notifications
3. Set up test Logstash pipeline
4. Set up main Logstash pipeline
5. View logs in Kibana

Set up CloudFront logging

First, you need an S3 bucket to store your CloudFront logs. You can use an existing bucket, or create a new one. You don't need to set up any special permissions for the bucket — but you probably will want to make sure the bucket denies public access to its content by default. In this example, we'll use an S3 bucket for logs called my-log-bucket, and we'll store our CloudFront logs under a directory of the bucket called my-cloudfront-logs. Also, we'll store each CloudFront distribution's logs in their own subdirectory of that directory; so for the distribution serving the www.example.com domain, we'll store the distributions logs under the my-cloudfront-logs/www.example.com subdirectory.

With the S3 logging bucket created and available, update each of your CloudFront distributions to log to it. You can do this via the AWS console by editing the distribution, turning the "Standard Logging" setting on, setting the "S3 Bucket for Logs" to your S3 logging bucket (my-log-bucket.s3.amazonaws.com), and setting the "Log Prefix" to the directory path of the subdirectory of the S3 bucket under which you'll store the logs (my-cloudfront-logs/www.example.com/). Save your changes, and every few minutes CloudFront will save a new .gz file to the my-cloudfront-logs/www.example.com/ subdirectory of the my-log-bucket (see the CloudFront access logs docs for details).

Set up SQS notifications

Next, create a new SQS queue. We'll call ours my-cloudfront-log-notifications, and we'll create it in the us-east-1 AWS region. When you create the queue, configure its "Receive message wait time" setting to 10 seconds or so; this will ensure the SQS client doesn't make way more SQS requests than needed (a setting of 10 seconds should keep the cost of this queue down to less than $1/month).

The only other thing special you need to do when you create the queue is add an access policy to it that allows S3 to send messages to it. The policy should look like this (replace my-cloudfront-log-notifications with the name of your queue, us-east-1 with your queue's region, my-log-bucket with the name of your log bucket, and 123456789012 with your AWS account ID):

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" }, "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-log-bucket" } } } ] }

With the SQS queue created, update the S3 bucket to send all object-create events to the queue. You can do this via the AWS console by selecting the bucket and opening the "Events" block in the "Advanced Settings" section of the "Properties" tab of the bucket. There you can add a notification; name it my-cloudfront-log-configuration, check the "All object create events" checkbox, set the "Prefix" to my-cloudfront-logs/, and send it to your SQS queue my-cloudfront-log-notifications.

Alternately, you can add a notification with the same settings as above via the put-bucket-notification-configuration command of the s3api CLI, using a notification-configuration JSON file like the following:

{ "QueueConfigurations": [ { "Id": "my-cloudfront-log-configuration", "QueueArn": "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications", "Events": [ "s3:ObjectCreated:*" ], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "my-cloudfront-logs/" } ] } } } ] }

Now that you've hooked up S3 bucket notifications to the SQS queue, if you look in the AWS console for the SQS queue, under the Monitoring tab's charts you'll start to see messages received every few minutes.

Set up test Logstash pipeline

Download a sample .gz log file from your S3 logging bucket, and copy it over to the machine you have Logstash running on. Move the file to a directory that Logstash can access, and make sure it has read permissions on the file. Our sample file will live at /var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz.

Copy the following my-cloudfront-pipeline.conf file into the /etc/logstash/conf.d directory on your Logstash machine (replacing the input path with your sample .gz log file), tail the Logstash logs (journalctl -u logstash -f if managed with systemd), and restart the Logstash service (sudo systemctl restart logstash):

# /etc/logstash/conf.d/my-cloudfront-pipeline.conf input { file { file_completed_action => "log" file_completed_log_path => "/var/lib/logstash/cloudfront-completed.log" mode => "read" path => "/var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz" sincedb_path => "/var/lib/logstash/cloudfront-since.db" type => "cloudfront" } } filter { if [type] == "cloudfront" { if (("#Version: 1.0" in [message]) or ("#Fields: date" in [message])) { drop {} } mutate { rename => { "type" => "[@metadata][type]" } # strip dashes that indicate empty fields gsub => ["message", "\t-(?=\t)", " "] # literal tab } #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end csv { separator => " " # literal tab columns => [ "date", "time", "x_edge_location", "sc_bytes", "c_ip", "cs_method", "cs_host", "cs_uri_stem", "sc_status", "cs_referer", "cs_user_agent", "cs_uri_query", "cs_cookie", "x_edge_result_type", "x_edge_request_id", "x_host_header", "cs_protocol", "cs_bytes", "time_taken", "x_forwarded_for", "ssl_protocol", "ssl_cipher", "x_edge_response_result_type", "cs_protocol_version", "fle_status", "fle_encrypted_fields", "c_port", "time_to_first_byte", "x_edge_detailed_result_type", "sc_content_type", "sc_content_len", "sc_range_start", "sc_range_end" ] convert => { "c_port" => "integer" "cs_bytes" => "integer" "sc_bytes" => "integer" "sc_content_len" => "integer" "sc_range_end" => "integer" "sc_range_start" => "integer" "sc_status" => "integer" "time_taken" => "float" "time_to_first_byte" => "float" } add_field => { "datetime" => "%{date} %{time}" "[@metadata][document_id]" => "%{x_edge_request_id}" } remove_field => ["cloudfront_fields", "cloudfront_version", "message"] } # parse datetime date { match => ["datetime", "yy-MM-dd HH:mm:ss"] remove_field => ["datetime", "date", "time"] } # lookup geolocation of client ip address geoip { source => "c_ip" target => "geo" } # parse user-agent into subfields urldecode { field => "cs_user_agent" } useragent { source => "cs_user_agent" target => "ua" add_field => { "user_agent.name" => "%{[ua][name]}" "user_agent.version" => "%{[ua][major]}" "user_agent.device.name" => "%{[ua][device]}" "user_agent.os.name" => "%{[ua][os_name]}" "user_agent.os.version" => "%{[ua][os_major]}" } remove_field => ["cs_user_agent", "ua"] } # pull logfile path from s3 metadata, if present if [@metadata][s3][object_key] { mutate { add_field => { "path" => "%{[@metadata][s3][object_key]}" } } } # strip directory path from logfile path, and canonicalize field name mutate { rename => { "path" => "log.file.path" } gsub => ["log.file.path", ".*/", ""] remove_field => "host" } # canonicalize field names, and drop unwanted fields mutate { rename => { "c_ip" => "client.ip" "cs_bytes" => "http.request.bytes" "sc_content_len" => "http.response.body.bytes" "sc_content_type" => "http.response.body.type" "cs_method" => "http.request.method" "cs_protocol" => "url.scheme" "cs_protocol_version" => "http.version" "cs_referer" => "http.request.referrer" "cs_uri_query" => "url.query" "cs_uri_stem" => "url.path" "sc_bytes" => "http.response.bytes" "sc_status" => "http.response.status_code" "ssl_cipher" => "tls.cipher" "ssl_protocol" => "tls.protocol_version" "x_host_header" => "url.domain" } gsub => [ "http.version", "HTTP/", "", "tls.protocol_version", "TLSv", "" ] remove_field => [ "c_port", "cs_cookie", "cs_host", "fle_encrypted_fields", "fle_status", "sc_range_end", "sc_range_start", "x_forwarded_for" ] } } } output { stdout { codec => "rubydebug" } }

You should see a bunch of entries in the Logstash logs like the following, one for each entry from your sample log file (note the fields will appear in a different order every time you run this):

Jan 02 03:04:05 logs1 logstash[12345]: { Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_detailed_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "@timestamp" => 2020-01-02T03:01:02.000Z, Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.device.name" => "EML-AL00", Jan 02 03:04:05 logs1 logstash[12345]: "time_taken" => 0.001, Jan 02 03:04:05 logs1 logstash[12345]: "http.version" => "2.0", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.os.version" => "8", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.body.bytes" => nil, Jan 02 03:04:05 logs1 logstash[12345]: "tls.cipher" => "ECDHE-RSA-AES128-GCM-SHA256", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.bytes" => 2318, Jan 02 03:04:05 logs1 logstash[12345]: "@version" => "1", Jan 02 03:04:05 logs1 logstash[12345]: "time_to_first_byte" => 0.001, Jan 02 03:04:05 logs1 logstash[12345]: "http.request.method" => "GET", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_request_id" => "s7lmJasUXiAm7w2oR34Gfg5zTgeQSTkYwiYV1pnz5Hzv8mRmBzyGrw==", Jan 02 03:04:05 logs1 logstash[12345]: "log.file.path" => "EML9FBPJY2494.2020-01-02-03.abcd1234.gz", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "http.request.bytes" => 388, Jan 02 03:04:05 logs1 logstash[12345]: "http.request.referrer" => "http://baidu.com/", Jan 02 03:04:05 logs1 logstash[12345]: "client.ip" => "192.0.2.0", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.name" => "UC Browser", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.version" => "11", Jan 02 03:04:05 logs1 logstash[12345]: "url.query" => nil, Jan 02 03:04:05 logs1 logstash[12345]: "http.response.body.type" => "text/html", Jan 02 03:04:05 logs1 logstash[12345]: "url.domain" => "www.example.com", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_location" => "LAX50-C3", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.status_code" => 200, Jan 02 03:04:05 logs1 logstash[12345]: "geo" => { Jan 02 03:04:05 logs1 logstash[12345]: "ip" => "192.0.2.0", Jan 02 03:04:05 logs1 logstash[12345]: "region_name" => "Shanghai", Jan 02 03:04:05 logs1 logstash[12345]: "country_name" => "China", Jan 02 03:04:05 logs1 logstash[12345]: "timezone" => "Asia/Shanghai", Jan 02 03:04:05 logs1 logstash[12345]: "longitude" => 121.4012, Jan 02 03:04:05 logs1 logstash[12345]: "country_code3" => "CN", Jan 02 03:04:05 logs1 logstash[12345]: "location" => { Jan 02 03:04:05 logs1 logstash[12345]: "lon" => 121.4012, Jan 02 03:04:05 logs1 logstash[12345]: "lat" => 31.0449 Jan 02 03:04:05 logs1 logstash[12345]: }, Jan 02 03:04:05 logs1 logstash[12345]: "region_code" => "SH", Jan 02 03:04:05 logs1 logstash[12345]: "country_code2" => "CN", Jan 02 03:04:05 logs1 logstash[12345]: "continent_code" => "AS", Jan 02 03:04:05 logs1 logstash[12345]: "latitude" => 31.0449 Jan 02 03:04:05 logs1 logstash[12345]: }, Jan 02 03:04:05 logs1 logstash[12345]: "url.scheme" => "https", Jan 02 03:04:05 logs1 logstash[12345]: "tls.protocol_version" => "1.2", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.os.name" => "Android", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_response_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "url.path" => "/" Jan 02 03:04:05 logs1 logstash[12345]: }

These entries show you what Logstash will push to Elasticsearch, once you hook it up. You can adjust this my-cloudfront-pipeline.conf file and restart Logstash again and again until you get the exact field names and values that you want to push to Elasticsearch.

Let's look at each part of the pipeline individually.

In the input section, we're using the file input to read just our one sample file:

input { file { file_completed_action => "log" file_completed_log_path => "/var/lib/logstash/cloudfront-completed.log" mode => "read" path => "/var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz" sincedb_path => "/var/lib/logstash/cloudfront-since.db" type => "cloudfront" } }

The key bit here is that we set the type field to cloudfront, which we'll use in the filter section below to apply our filtering logic only to entries of this type. If you're only going to process CloudFront log files in this pipeline, you can omit all the bits of the pipeline that deal with "type", which would simplify it some.

In the filter section, the first step is to check if the type field was set to "cloudfront", and only execute the rest of the filter block if so:

filter { if [type] == "cloudfront" {

Then the next step in filter section is to drop the two header lines in each CloudFront log file, the first beginning with #Version, and the second beginning with #Fields:

if (("#Version: 1.0" in [message]) or ("#Fields: date" in [message])) { drop {} }

After that, the next step renames the type field to [@metadata][type], so that it won't be pushed to the Elasticsearch index. I've opted to use Elasticsearch indexes that are for my CloudFront logs only; however, if you want to push your CloudFront logs into indexes that are shared with other data, you may want to keep the type field.

mutate { rename => { "type" => "[@metadata][type]" }

The second half of this mutate filter strips out the - characters that indicate empty field values from all the columns in the log entry. Note that the last argument of this gsub function is a literal tab character — make sure your text editor does not convert it to spaces!

# strip dashes that indicate empty fields gsub => ["message", "\t-(?=\t)", " "] # literal tab }

For example, it will convert a entry like this:

2020-01-02 03:03:03 HIO50-C1 6564 192.0.2.0 GET d2c4n4ttot8c65.cloudfront.net / 200 - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:40.0)%20Gecko/20100101%20Firefox/40.1 - - Miss nY0knXse4vDxS5uOBe3YAhDpH809bqhsILUUFAtE_4ZLlfXCiYcD0A== www.example.com https 170 0.164 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/1.1 - - 62684 0.164 Miss text/html 6111 - -

Into this (removing the dashes that indicate empty values, but not the dashes in non-empty values like the date or ciphersuite):

2020-01-02 03:03:03 HIO50-C1 6564 192.0.2.0 GET d2c4n4ttot8c65.cloudfront.net / 200 Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:40.0)%20Gecko/20100101%20Firefox/40.1 Miss nY0knXse4vDxS5uOBe3YAhDpH809bqhsILUUFAtE_4ZLlfXCiYcD0A== www.example.com https 170 0.164 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/1.1 62684 0.164 Miss text/html 6111

The next step is the meat of the process, using the csv filter to convert each tab-separated log line into named fields. Note that the separator property value is also a literal tab character:

#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end csv { separator => " " # literal tab columns => [ "date", "time", "x_edge_location", "sc_bytes", "c_ip", "cs_method", "cs_host", "cs_uri_stem", "sc_status", "cs_referer", "cs_user_agent", "cs_uri_query", "cs_cookie", "x_edge_result_type", "x_edge_request_id", "x_host_header", "cs_protocol", "cs_bytes", "time_taken", "x_forwarded_for", "ssl_protocol", "ssl_cipher", "x_edge_response_result_type", "cs_protocol_version", "fle_status", "fle_encrypted_fields", "c_port", "time_to_first_byte", "x_edge_detailed_result_type", "sc_content_type", "sc_content_len", "sc_range_start", "sc_range_end" ] }

The columns property lists out each field name, in order. Later on in this pipeline, we'll rename many of these fields to use the ECS nomenclature, but this step uses the field names as defined by CloudFront, for clarity.

The middle part of the csv filter converts the numeric fields to actual numbers, via the convert property mapping:

convert => { "c_port" => "integer" "cs_bytes" => "integer" "sc_bytes" => "integer" "sc_content_len" => "integer" "sc_range_end" => "integer" "sc_range_start" => "integer" "sc_status" => "integer" "time_taken" => "float" "time_to_first_byte" => "float" }

The add_field part of the csv filter combines the individual date and time fields into a combined datetime field (to be converted to a timestamp object later); and also copies the x_edge_request_id field value as the [@metadata][document_id] field:

add_field => { "datetime" => "%{date} %{time}" "[@metadata][document_id]" => "%{x_edge_request_id}" }

The [@metadata][document_id] field will be used later on when we push the record to Elasticsearch (to be used as the record's ID). Like with the [@metadata][type] field, this is another case where if you're only going to process CloudFront log files in this pipeline, you could omit this extra metadata field, and just use the x_edge_request_id directly when configuring the Elasticsearch record ID.

The final part of the csv filter removes some fields that are redundant once the log entry has been parsed: message (the full log entry text itself), and cloudfront_fields and cloudfront_version (which the s3snssqs input we'll add later automatically includes):

remove_field => ["cloudfront_fields", "cloudfront_version", "message"] }

The next filter step is to convert the datetime field (created from the date and time fields above) into a proper datetime object:

# parse datetime date { match => ["datetime", "yy-MM-dd HH:mm:ss"] remove_field => ["datetime", "date", "time"] }

This sets the datetime as the value of the @timestamp field. We'll also remove the datetime, date, and time fields, since we won't need them now that we have the parsed datetime in the @timestamp field.

The next filter uses the client IP address to lookup a probable physical location for the client:

# lookup geolocation of client ip address geoip { source => "c_ip" target => "geo" }

This creates a geo field with a bunch of subfields (like [geo][country_name], [geo][city_name], etc) containing the probable location details. Note that many IP address won't have a mapping value for many of the subfields; see the Geoip filter docs for more details.

The next filter decodes the user-agent field, and the filter after that parses it. The useragent filter parses the cs_user_agent field into the ua field, which, like the geo field, will contain a bunch of subfields. We'll pull out a few of those subfields, and add fields with ECS names for them:

# parse user-agent into subfields urldecode { field => "cs_user_agent" } useragent { source => "cs_user_agent" target => "ua" add_field => { "user_agent.name" => "%{[ua][name]}" "user_agent.version" => "%{[ua][major]}" "user_agent.device.name" => "%{[ua][device]}" "user_agent.os.name" => "%{[ua][os_name]}" "user_agent.os.version" => "%{[ua][os_major]}" } remove_field => ["cs_user_agent", "ua"] }

Since the user-agent info we want are now in those newly added user_agent.* fields, the last part of the useragent filter removes the cs_user_agent field and intermediate ua field.

When using the file input, like we are while testing this pipeline, the file input will add a path field to each record, containing the path to the file its reading. Later on, when we use the s3snssqs input, the s3snssqs input will pass the same path as the [@metadata][s3][object_key] field. So that we can access this value uniformly, regardless of which input we used, we have this next filter step, where if the [@metadata][s3][object_key] field is present, we set the path field to the [@metadata][s3][object_key] field's value:

# pull logfile path from s3 metadata, if present if [@metadata][s3][object_key] { mutate { add_field => { "path" => "%{[@metadata][s3][object_key]}" } } }

With the path field now containing the file path, regardless of input, we use the next filter to chop the path down to just the log file name (like E123456789ABCD.2020-01-02-03.abcd1234.gz):

# strip directory path from logfile path, and canonicalize field name mutate { rename => { "path" => "log.file.path" } gsub => ["log.file.path", ".*/", ""] remove_field => "host" }

We also have the filter rename the path field to log.file.path (the canonical ECS name for it); and have the filter remove the host field (added by the file input along with the path field, based on the host Logstash is running on — which we don't really care to have as part of our log record in Elasticsearch).

The last filter in our pipeline renames all CloudFront fields that have equivalent ECS (Elastic Common Schema) field names:

# canonicalize field names, and drop unwanted fields mutate { rename => { "c_ip" => "client.ip" "cs_bytes" => "http.request.bytes" "sc_content_len" => "http.response.body.bytes" "sc_content_type" => "http.response.body.type" "cs_method" => "http.request.method" "cs_protocol" => "url.scheme" "cs_protocol_version" => "http.version" "cs_referer" => "http.request.referrer" "cs_uri_query" => "url.query" "cs_uri_stem" => "url.path" "sc_bytes" => "http.response.bytes" "sc_status" => "http.response.status_code" "ssl_cipher" => "tls.cipher" "ssl_protocol" => "tls.protocol_version" "x_host_header" => "url.domain" }

To match the ECS field specs, the middle part of the filter removes the HTTP/ prefix from the http.version field values (converting values like HTTP/2.0 to just 2.0); and removes the TLSv prefix from the tls.protocol_version field values (converting values like TLSv1.2 to just 1.2):

gsub => [ "http.version", "HTTP/", "", "tls.protocol_version", "TLSv", "" ]

And finally, the last part of the filter removes miscellaneous CloudFront fields that we don't care about:

remove_field => [ "c_port", "cs_cookie", "cs_host", "fle_encrypted_fields", "fle_status", "sc_range_end", "sc_range_start", "x_forwarded_for" ] } } }

The output section of the pipeline simply outputs each log record to Logstash's own log output — which is what you see when you tail Logstash's logs:

output { stdout { codec => "rubydebug" } }

Set up main Logstash pipeline

Once you have this test pipeline working to your satisfaction, it's time to change the output section of the pipeline to push the output to Elasticsearch. Replace the output block of the /etc/logstash/conf.d/my-cloudfront-pipeline.conf file with this block (substituting your own host, user, and password settings, as well as any custom SSL settings you need — see the Elasticsearch output plugin docs for details):

output { # don't try to index anything that didn't get a document_id if [@metadata][document_id] { elasticsearch { hosts => ["https://elasticsearch.example.com:9243"] user => "elastic" password => "password123" document_id => "%{[@metadata][document_id]}" ecs_compatibility => "v1" index => "ecs-logstash-%{[@metadata][type]}-%{+YYYY.MM.dd}" } } }

This following line in this block serves as one more guard to avoid indexing anything that didn't get parsed properly (you may want to send such log entries to a dedicated errors index, to keep an eye on entries that failed to parse):

if [@metadata][document_id] {

And this line uses the [@metadata][document_id] field to set the record ID for each entry (recall in the pipeline filters, we copied the value of the CloudFront x_edge_request_id, which should be unique for each request, to the [@metadata][document_id] field):

document_id => "%{[@metadata][document_id]}"

And since our output block includes setting ecs_compatibility to v1, which directs Logstash to use ECS-compatible index templates, this line directs Logstash to create a separate index for each day and type of log entry we process:

index => "ecs-logstash-%{[@metadata][type]}-%{+YYYY.MM.dd}"

For example, Logstash will create an index named ecs-logstash-cloudfront-2020.01.02 if we process a CloudFront log entry for January 2, 2020 (or use the existing index with that name, if it already exists).

Restart Logstash once you change the output block. In Logstash's own log output, you should see entries indicating succesful connections to your Elasticsearch host, as well as a ginormous entry for the index template it installs in Elasticsearch. Once you see that, check your Elasticsearch instance — you should see a new ecs-logstash-cloudfront-YYYY.MM.DD index created, with entries from your sample CloudFront log file.

You can use this same mechanism to backfill your existing CloudFront log files to Elastic search — manually download the log files to backfill to your Logstash machine (like via the sync command of the s3 CLI), and customize the file input block's path property (with wildcards) to direct Logstash to read them in.

For future CloudFront log files, however, we're going to make one more change to our pipeline, and use the S3 via SNS/SQS input (aka s3snssqs) to pull CloudFront log files from S3 as soon as CloudFront publishes them.

First, create a new IAM policy for your Logstash machine to use that will allow it to both read from your logging bucket, and to read and delete items from the SQS queue we set up above. The policy should look like this (change the Resource elements to point to your own S3 log bucket and SQS log queue, set up in the first two sections of this article):

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-log-bucket" }, { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-log-bucket/my-cloudfront-logs/*" }, { "Effect": "Allow", "Action": [ "sqs:Get*", "sqs:List*", "sqs:ReceiveMessage", "sqs:ChangeMessageVisibility", "sqs:DeleteMessage" ], "Resource": [ "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications" ] } ] }

Then install the logstash-input-s3-sns-sqs plugin on your Logstash machine:

cd /usr/share/logstash sudo -u logstash bin/logstash-plugin install logstash-input-s3-sns-sqs

Then update the input section of your pipeline to be the following (substituting your own SQS queue name and its AWS region):

input { # pull new logfiles from s3 when notified s3snssqs { region => "us-east-1" queue => "my-cloudfront-log-notifications" from_sns => false type => "cloudfront" } }

If you're running the Logstash machine in AWS, you can use the usual EC2 instance profiles or IAM roles for tasks to grant the machine access to the policy you created above. Otherwise, you'll need to add some AWS credential settings to the s3snssqs input as well; consult the S3 input plugins docs for options (the s3snssqs input allows for the same AWS credential options as the s3 input does, but the s3 input has better documentation for them).

Now restart Logstash. You should see the same output in Logstash's own log as before; but if you check Elasticsearch, you should see new records being added.

View logs in Kibana

Eventually you'll want to create fancy dashboards in Kibana for your new CloudFront data; but for now we'll just get started by setting up a listing where you can view them in the "Discover" section of Kibana.

First log into Kibana, and navigate to the "Management" > "Stack Management" section of Kibana. Within the "Stack Management" section, if you navigate to the "Data" > "Index management" subsection, you should see a bunch of new indexes named in the form of ecs-logstash-cloudfront-YYYY.MM.DD (like ecs-logstash-cloudfront-2020.01.01 and so on):

Once you've verified Kibana is seeing the indexes, navigate to the "Kibana" > "Index Patterns" subsection, and click the "Create index pattern" button. Specify ecs-logstash-cloudfront-* as the pattern, and select @timestamp as the time field:

With the new index pattern created, navigate out of the "Stack Management" section of Kibana into the main "Kibana" > "Discover" section. This will show your most recent "Discover" search. On the left side of the page, change the selected index pattern to the pattern you just created (ecs-logstash-cloudfront-*). You should now see your most recent CloudFront entries listed (if not, use the time window selector in the top right of the page to expand the time window to include a range you know should include some entries). You can use this page to create a list with custom columns and custom filter settings for your CloudFront logs:

IPv6 for Private AWS Subnets

Instead of setting up "private" subnets in AWS with the use of moderately-expensive AWS NAT Gateways, I've been experimenting with free Egress-Only Internet Gateways (EIGW). The downside with EIGW is that it's IPv6 only — so you can communicate to the outside world from instances in your private subnet only through IPv6.

In theory, that shouldn't be a problem — it's 2020 and IPv6 just works everywhere, right? Well, actually, in practice it does work pretty well — but there are still a few gotchas. Here are a few hurdles I had to work around when setting up a fleet of Ubuntu Linux EC2 instances with Ansible:

APT Repos

The base AMIs that Ubuntu provides are configured to use the Ubuntu APT repos hosted by AWS (like us-east-1.ec2.archive.ubuntu.com); however, these repos only support IPv4. So the first thing you need to do is change the repos listed in /etc/apt/sources.list to use external repos that support IPv6 (like us.archive.ubuntu.com, or other Ubuntu mirrors you might find).

And since you won't be able to use IPv4 to access the repos, you can speed up APT updates by configuring APT to try only IPv6. To do so, add a file in your /etc/apt/apt.conf.d/ directory (call it something like 99force-ipv6) with the following content:

Acquire::ForceIPv6 "true";

Also don't forget that if you do set up a restrictive Network ACL for your private subnet, you'll need to allow inbound TCP access to the standard Linux ethereal port range (32768-61000) from whatever APT repos you use.

NTP Pools

The NTP pools used by the base AMIs also don't support IPv6. I use the traditional NTP daemon provided by the Ubuntu ntp package, rather the default systemd-tymesyncd service. To configure the NTP daemon, I remove all the default pools from the /etc/ntp.conf file, and instead just use the 2.us.pool.ntp.org pool (the convention with NTP is that for domains that have numbered pools, like 0.us.pool.ntp.org, 1.us.pool.ntp.org, 2.us.pool.ntp.org, etc, the pool numbered 2 is the one that supports IPv6).

Specifically, this is how I configure the 2.us.pool.ntp.org pool in /etc/ntp.conf:

pool -6 2.us.pool.ntp.org iburst minpoll 10 maxpoll 12

The -6 flag means to use IPv6; the iburst part is supposed to help speed up initial synchronization; the minpoll 10 part means to poll no more often than every 2^10 seconds (around 17 minutes); and the maxpoll 12 part means to poll no less often than every 2^12 seconds (around 68 minutes).

Also, if you set up a restrictive Network ACL for your private subnet, you'll need to allow inbound access to UDP port 123.

AWS APIs

If you are planning to directly call AWS APIs (either through the various per-language SDKs, or the CLI), a huge gotcha is that very few AWS services as of yet provide IPv6 endpoints. This means that you won't be able to use most AWS services at all from within your private IPv6 subnet (with the exception of services that consist of instances that themselves reside within your VPCs, like RDS; rather than endpoints hosted outside of your VPCs, like DynamoDB).

The only major AWS service I've tried that does support IPv6 through its APIs is S3. When connecting to it via CLI, you can get it to use IPv6 by explicitly specifying the "dualstack" endpoint via command-line flag, like this:

aws --endpoint-url https://s3.dualstack.us-east-1.amazonaws.com --region us-east-1 s3 ls

Or, alternately, you can enable IPv6 usage via the AWS config file (~/.aws/config), like this:

[default] region = us-east-1 s3 = use_dualstack_endpoint = true addressing_style = virtual

Ansible Inventory

To access EC2 instances in a private subnet, typically you'd use a VPN running in a public subnet of the same (or bridged) VPC, with the VPN client set to route the VPC's private IPv4 block through the VPN. For IPv6, I also have my VPN set to route the VPC's IPv6 block through the VPN, too.

Using Ansible through a VPN with IPv4 is pretty much as simple as configuring Ansible's ec2.ini file to set its destination_variable and vpc_destination_variable settings to private_ip_address. But since I decided to disallow any IPv4 access to my private subnets (even from other subnets within the same VPC), I had to jump through a few extra hoops:

1. Custom Internal Domain Names

I use a custom internal domain name for all my servers (I'll use example.net as the custom domain in the following examples), and assign each server its own domain name (like db1.example.net or mail2.example.net, etc). When I launch a new EC2 server, I create a DNS AAAA record for it (via Route53), pointing the DNS record to the IPv6 address of the newly-launched server. In this way I can use the DNS name to refer to the same server throughout the its lifetime.

I also tag my EC2 instances as soon as I launch them with tags from which the DNS name can be constructed. For example, I'd assign the server with the DNS name of fe3.example.net a "node" tag of fe and a "number" tag of 3.

2. SSH Config

In my SSH config file (~/.ssh/config), I have an entry like the following, to make sure SSH (and Ansible) only tries to access my EC2 instances through IPv6:

Host *.example.net AddressFamily inet6

3. Ansible EC2 Config

With the above two elements in place, I can then enable the destination_format (and destination_format_tags) settings in the Ansible ec2.ini configuration file to direct Ansible to use DNS names instead of IP address for EC2 inventory. With the "node" and "number" tags described above, I can use the following configuration in my ec2.ini file:

destination_format = {0}{1}.example.net destination_format_tags = node,number

When the above is set up correctly, you can run the ec2.py script (eg as ./ec2.py), and see your DNS names in its output (like db1.example.net or mail2.example.net, etc), instead of IPv4 addresses. And when you run an ad-hoc Ansible module (like ansible fe3.example.com -i ec2.py -m setup) everything should "just work".

Let's Encrypt DNS Validation with Lego

Let's Encrypt recently enabled support for DNS challenges, but only a few clients yet support it. Lego is one of these clients, and already features integration with a number of popular DNS management APIs, including AWS Route 53, CloudFlare, DigitalOcean, and DNSimple. Lego also makes it really easy to use DNS challenges even without a supported API — if you run it in "manual" DNS challenge mode, it will print out the TXT record you need to add to your zone file, wait for you to add it, and then continue on to complete the challenge.

Lego is a neat Go project that can also itself be used as an API by other Go projects. However, it's quite easy to simply install and run Lego as a command-line tool. These are the steps I took to install it from scratch on Ubuntu 15.10, and use it with Route 53 to generate a SAN SSL certificate (a single certificate covering multiple domain names):

0. Install Go

If you don't already have Go installed on your system, you need to install it first. I don't know exactly what version Lego requires — but something newer than 1.2.1 (the version packaged in Ubuntu 14.04). It does work with version 1.5.1, the version packaged in Ubuntu 15.10, so if you have Ubuntu 15.10 or newer, you can simply install Go via apt-get:

sudo apt-get install golang

Otherwise, the Go Version Manager (GVM) provides a convenient command-line installer, allowing you to install multiple different versions of Go on the same machine, and switch between them as necessary.

1. Install Lego

You can install Lego with the go get command — but if you're not a Go developer, you probably don't have your GOPATH environment variable set, and you need to have it set first. I'd suggest just creating a lego directory somewhere convenient (like in your home folder), and using it for your GOPATH:

mkdir $HOME/lego
export GOPATH=$HOME/lego
go get -u github.com/xenolf/lego

The above will create a lego directory in your home folder, and install the lego executable in its bin subdirectory.

2. Run Lego

Now you can run the lego executable. Specify each domain you want the cert to cover via a separate --domain argument (the example below covers mail.example.com, www.example.com, and example.com). Specify your email address (for renewal reminders) via the --email flag. To use Route 53 DNS validation, include the --dns=route53 flag (for "manual" DNS validation, where you create the challenge DNS records manually, specify --dns=manual instead).

If you use Route 53, first specify your API key and secret as environment variables, like this:

export AWS_ACCESS_KEY_ID=ABC123
export AWS_SECRET_ACCESS_KEY=ABC+def/123

Alternatively, Lego supports the same credential/configuration files that the standard AWS command-line tools support (and uses the Golang Amazon Library specifically, so check out its aws.GetAuth function for the details of exactly what environment variables and configuration files are supported, and in what order they're checked). So if you've got those files set up with your AWS API credentials already, you don't need to mess around with any additional environment variables.

I'd also suggest explicitly specifying the path to the certificates/account info that Lego will generate, via the --path flag; if you created a lego directory in your home folder, just use that:

$HOME/lego/bin/lego \
    --accept-tos \
    --dns=route53 \
    --path=$HOME/lego \
    --email=me@example.com \
    --domains=mail.example.com \
    --domains=www.example.com \
    --domains=example.com \
    run

3. Check Out the Results

Once run successfully, Lego will output two sets of files into the directory you specified with the --path flag: your Let's Encrypt account info in the accounts subdirectory, and your new SSL cert in the certificates directory:

/home/me/lego/accounts/acme-v01.api.letsencrypt.org/me@example.com/account.json
/home/me/lego/accounts/acme-v01.api.letsencrypt.org/me@example.com/keys/me@example.com.key
/home/me/lego/certificates/mail.example.com.crt
/home/me/lego/certificates/mail.example.com.json
/home/me/lego/certificates/mail.example.com.key

In the accounts hierarchy, the account.json file contains your Let's Encrypt account info (which you can use later to renew or revoke the certificate). The me@example.com.key contains the secret key (ie password) for that account.

In the certificates subdirectory, your new certificate file will be named with the first domain you specified (in the above example, mail.example.com), with a crt extension. This file also includes the intermediate certificate for Let's Encrypt, so it's equivalent to the "fullchain" file that the reference Let's Encrypt client generates (the so-called "all-in-one" file that you'd use with Apache's SSLCertificateFile directive or Nginx's ssl_certificate directive). The corresponding secret key for the certificate will have the same name, but with a key extension. The file with the json extension contains some (non-secret) metadata for the cert that Let's Encrypt will need later if you renew or revoke it.

If you have OpenSSL installed, you can check out the cert details with this command (which will print out the details to stdout):

openssl x509 -text -noout -in $HOME/lego/certificates/mail.example.com.crt

The different domain names the cert covers will be listed in the X509v3 Subject Alternative Name field of the output.

Importing the New RDS CA Certificate Into the Java Keytool

A few days ago (Feb 2015), Amazon released a new CA certificate bundle (rds-combined-ca-bundle.pem) for use with AWS RDS databases. In order to connect with a MySQL, PostgreSQL, or SQL Server DB over SSL, your client app has to trust the certificates in this bundle. For java apps (using JDBC) this generally requires importing the certs into your java keystore via the java keytool utility.

The first RDS cert (from 2010) was released as a single certificate in a single file; but this new release is actually 10 separate new certificates (plus the old certificate) jammed into the same file. The java keytool, however, supports importing only one certificate at a time — so you have to first split up the bundle into individual certificates, and then import the certs one by one. If you try to import the whole bundle as a single file, you'll end up trusting only the first cert listed in that file (which happens, in this case, to be the old cert).

I wrote up a quick shell script that 1) downloads the new CA bundle, 2) splits up the bundle into individial certificate files, and 3) imports each file separately into the java keytool. It assumes that you already have the keytool installed and on your path; that the keystore location is /etc/ssl/certs/java/cacerts and the password is changeit (the defaults for the OpenJDK on Ubuntu); that you're running the script as a sudoer; and that you have OpenSSL installed. You'll have to tweak those things if that's not the case:

#!/bin/sh -e

# create a temp dir in which to work
OLDDIR="$PWD"
mkdir /tmp/rds-ca && cd /tmp/rds-ca

# download the bundle
wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
# split the bundle into individual certs (prefixed with xx)
csplit -sz rds-combined-ca-bundle.pem '/-BEGIN CERTIFICATE-/' '{*}'

# import each cert individually
for CERT in xx*; do
    # extract a human-readable alias from the cert
    ALIAS=$(openssl x509 -noout -text -in $CERT |
        perl -ne 'next unless /Subject:/; s/.*CN=//; print')
    echo "importing $ALIAS"
    # import the cert into the default java keystore
    sudo keytool -import \
        -keystore /etc/ssl/certs/java/cacerts \
        -storepass changeit -noprompt \
        -alias "$ALIAS" -file $CERT
done

# back out of the temp dir and delete it
cd "$OLDDIR"
rm -r /tmp/rds-ca

# list the imported rds certs as a sanity check
keytool -list \
    -keystore /etc/ssl/certs/java/cacerts \
    -storepass changeit -noprompt |
    grep -i rds

OpenSSL, in particular, is used by the script only to extract a convenient, unique alias for each certificate; so if you don't have OpenSSL installed, you could replace the alias-creating line with something like this (which will create aliases in the form of rdsxx00, rdsxx01, etc):

    ALIAS=rds$CERT

Jenkins EC2 Slave Plugin

Just finished configuring our Jenkins build machine to use the Jenkins EC2 Plugin (currently at version 1.18), which allows the Jenkins server to spin up AWS EC2 instances on demand to use as slaves for selected build jobs. It's very cool, and requires only a little bit of configuration to get it running your existing jobs automatically on EC2 slaves. These were the steps I had to take:

Create an IAM user for Jenkins

The Jenkins server needs access to your AWS account in order to run and kill EC2 instances; the best way to enable this is to create a separate IAM user that is used only by Jenkins, and has only the minimum permissions required. I used the IAM section of the AWS Console for this (although you can also do it via command line). When you create the new IAM user, create an access key for the user, and make sure you save the secret key part of it, since AWS does not store the secret key (you have to generate a new access key if you lose the secret part of it). You will use this access key when you configure the Jenkins EC2 plugin.

Once you create the user, you need to attach a policy to it that will allow the user to run and kill EC2 instances. Via trial-and-error, I found that these were the minimum permissions currently required by the Jenkins EC2 plugin (I limited it to a single region, us-west-2; if you want to allow Jenkins to manage instances in all regions, remove the ec2:Region condition):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeRegions"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:GetConsoleOutput",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:Region": "us-west-2"
                }
            }
        }
    ]
}

Create an SSH key pair for Jenkins

Once the Jenkins server starts a new EC2 slave, it will connect to it via SSH, using public-key authentication. At minimum, you need to provide Jenkins with the private key of a user account that has password-less sudo privileges on the slave; the easiest way to do this with a stock AMI image is to just let Jenkins use the image's default account via the EC2-registered key-pair used to boot it.

I just used the stock Ubuntu AMI (12.04, 64-bit intance-store variety), where the default user (ubuntu) has password-less sudo privileges. And I created an SSH key-pair for Jenkins to use via the EC2 section of the AWS Console (although you can also do it via the command line). As with the secret part of the IAM user's access key, make sure you save the private key for the key pair, since AWS does not store it (you'll have to generate a new key pair if you lose the private key).

Create an EC2 security-group for the slave

Whenever you launch an EC2 instance, you must specify the "security group" to which the instance belongs. This is basically just a set of firewall rules for the built-in EC2 firewall that restrict inbound access to a limited set of ports (optionally from a limited set of hosts). You can change the firewall rules for a group even while that group has running instances, but you can't switch an instance to use a different group once the instance has been launched. So I recommend creating a separate group just for your Jenkins slaves, even if the group has the same rules as you use for other security groups (so that you can change the rules for the different groups independently, should the need ever arise).

The only port you'll need inbound access to on the slaves is 22 (SSH). Optionally, you can set the source for that rule to the IP address of the Jenkins master server (if you want to disallow any inbound access to the slave other than from the master).

Create an init script for the slave

When Jenkins starts a new EC2 slave (and once the slave has booted and Jenkins connects to it), Jenkins will run a script that you specify to prepare the slave to run builds. Jenkins automatically will bootstrap the Jenkins-specific environment on the slave (once your init script has run), so you really only need to setup a few things, like java and git (or svn, hg, etc).

Following is the init script I used with the stock Ubuntu AMI. It installs a few things via Apt; installs a few particular versions of Grails that we use; and then installs and runs the latest version of PhantomJS (which we need to have running with some specific arguments for our functional tests). It also sets-up a working directory for Jenkins to use as the ubuntu user at /srv/jenkins; creates a bigger swap file than comes with the Ubuntu AMIs; and moves the /srv, /var, and /tmp directories to the faster "ephemeral" drive on the EC2 instance (mounted on the smaller EC2 instances at /mnt):

#!/bin/sh

# keep trying to install until it works
for I in 1 2 3; do
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q update
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q upgrade
    sudo DEBIAN_FRONTEND=noninteractive apt-get -y -q install \
        git-core \
        groovy \
        openjdk-6-jdk \
        vim \
        zip
    sleep 1
done

# fix missing java links
if [ ! -e /usr/lib/jvm/default-java ]; then
    sudo ln -s /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/default-java
fi
if [ ! -e /usr/lib/jvm/java-6-openjdk ]; then
    sudo ln -s /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/java-6-openjdk
fi

# download grails if necessary
if [ ! -e /opt/grails-2.1.0 ]; then
    # download grails zip
    if [ ! -e "/tmp/grails.zip" ]; then
        wget -nd -O /tmp/grails.zip http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-1.3.7.zip
    fi
    # unzip grails
    unzip /tmp/grails.zip
    # move to /opt
    sudo mv grails-1.3.7 /opt/.

    # download grails zip
    if [ ! -e "/tmp/grails2.zip" ]; then
        wget -nd -O /tmp/grails2.zip http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-2.1.0.zip
    fi
    # unzip grails
    unzip /tmp/grails2.zip
    # move to /opt
    sudo mv grails-2.1.0 /opt/.

    # expand max-memory size for grails
    sudo perl -pli -e 's/Xmx\d+/Xmx2048/; s/MaxPermSize=\d+/MaxPermSize=1024/' /opt/grails*/bin/startGrails
fi

# download phantomjs if necessary
HAS_PHANTOMJS=`whereis phantomjs | grep bin`
if [ -z "$HAS_PHANTOMJS" ]; then
    # download phantomjs binary
    if [ ! -e "/tmp/phantomjs.tar.bz2" ]; then
        wget -nd -O /tmp/phantomjs.tar.bz2 https://phantomjs.googlecode.com/files/phantomjs-1.9.1-linux-x86_64.tar.bz2
    fi
    # unzip phantomjs
    tar xf /tmp/phantomjs.tar.bz2
    # move to /opt
    sudo mv phantomjs-1.9.1-linux-x86_64 /opt/.
    sudo ln -s /opt/phantomjs-1.9.1-linux-x86_64 /opt/phantomjs
    # run in background
    nohup /opt/phantomjs/bin/phantomjs \
        --webdriver=7483 \
        --webdriver-logfile=webdriver.log \
        --ignore-ssl-errors=true \
        &>phantomjs.log &
fi

# creating jenkins working directory
sudo mkdir /srv/jenkins && sudo chown ubuntu:ubuntu /srv/jenkins

# add more swap
SWAPFILE=/mnt/swap1
if [ ! -f $SWAPFILE ]; then
    # creates 2G (1M * 2K) /mnt/swap1 file
    sudo dd if=/dev/zero of=$SWAPFILE bs=1M count=2K
    sudo chmod 600 $SWAPFILE
    sudo mkswap $SWAPFILE

    # add new swap to config and start using it
    echo "$SWAPFILE none swap defaults 0 0" | sudo tee -a /etc/fstab
    sudo swapon -a
fi

# move /tmp to larger/faster/transient /mnt volume
if [ ! -e /mnt/tmp ]; then
    # move each directory, then create link from old location to new
    for DIR in /srv /tmp /var; do
        echo $DIR
        sudo mv $DIR /mnt$DIR
        sudo mkdir $DIR
        echo "/mnt$DIR  $DIR    none    bind    0   0" | sudo tee -a /etc/fstab
        sudo mount $DIR
    done
fi

Among a few quirky things that the script does is re-try apt-get three times — I found that at least half the time the first run of apt-get would fail to update/install any packages without any comprehensible error messages (I think maybe because the system was still lazy-initializing some components?). Re-running it after a second seems to solve the issue, however.

Also, while installing only the openjdk-6-jdk package installs enough of java 6 for our build purposes, it doesn't install some of the links in the /usr/lib/jvm directory through which we reference the jdk or java home in our build scripts (these links usually are created by some other unknown java packages); so the script manually creates these java links.

And finally, the default JVM memory settings used by grails aren't sufficient for running or testing our apps; the quick fix for this is just to overwrite the Xmx and MaxPermSize settings in the default GRAILS_OPTS of the startGrails script that comes with each version of grails.

Configure the Jenkins EC2 plugin

Now, finally, you're ready to configure the EC2 plugin itself. Once you've installed the plugin, you navigate to the main "Manage Jenkins" > "Configure System" page, and scroll down near the bottom to the "Cloud" section. There, you click the "Add a new cloud" button, and select the "Amazon EC2" option. This will display the UI for configuring the EC2 plugin. The first things you configure are the access key that you created for the Jenkins IAM user (via the "Access Key ID" and "Secret Access Key" fields). If you've configured the permissions for the Jenkins IAM user correctly, this will populate the "Region" dropdown, and allow you to select the AWS region to use. Next, paste in the text from the secret-key pem file for the SSH key pair that you created for Jenkins in the "EC2 Key Pair's Private Key" field (this text will start with the line "-----BEGIN RSA PRIVATE KEY-----").

In the "AMIs" section of the configuration UI, you'll configure the per-slave settings. If you want, you can generate multiple slave profiles (from different AMI images, or different instance sizes, or different initialization parameters, etc). But you can start with just one, which you can add by clicking the "Add" button at the bottom of the "AMIs" section.

For each slave profile, you configure the profile with a description ("Standard EC2 Slave"), as well as the ID of the AMI to use (I used the ami-5168f861 AMI, the current official Ubuntu 12.04, 64-bit instance-store AMI). Next, select the instance type; a "micro" instance is probably too small for just about anything other than a one-line shell job; a "small" instance may be fine for some jobs; but with most of our builds (which all include at least one grails build step), we get a 3-4x improvement over "small" instance times with a "medium" instance (which is 2x the price of a "small").

You can optionally select a specific availability-zone within your selected region in which to launch the slave; this matters a lot if you have an existing EBS volume to which you will attach to the slave, and it matters a little if you have other EC2 instances which the slave is going to access (like if the master Jenkins server is in EC2, or if you have a version-control repo in EC2 from which the slave is going to download source code, etc); otherwise you can just leave the "Availability Zone" field blank, and AWS automatically will launch the slave in whatever zone of your configured region that is least active when the slave is launched.

Enter the name of the EC2 security-group that you created for the slave in the "Security group names" field. Enter the path to the directory that Jenkins should use as its working directory (similar to the /var/lib/jenkins directory on the master); you probably created this in your slave init script (it should be writable by the user Jenkins uses on the slave; the directory that my init script created for this was /srv/jenkins). Specify the user Jeknins should run as on the slave in the "Remote user" field; on a stock Ubuntu AMI, this is ubuntu. And unless you use root for this user, enter sudo in the "Root command prefix" field.

Enter one or more labels (separated by spaces) in the "Labels" field. You can assign jobs to specific slave profiles via labels, so if you have multiple slave profiles, you may want to include a label for each distinguishing feature of the slave. For example, if you have one profile for a small instance with no DB, you might label it just "small"; and for another profile using a medium instance with a MySQL DB, you might label as "medium mysql". Then for a job that only needs a small instance, you can set the job to use the slave labeled "small"; and for a job that requires a medium instance, you can set it to use the slave labeled "medium"; and for a job that requires MySQL, you can set the job to use the slave with the "mysql" label. If you have only one slave profile, you can just use a simple label like "slave" or "ec2" (and then configure any job that you want to run on the slave with the "slave" or "ec2" label).

The default "Usage" setting for both the master Jenkins server and each slave server is to "Utilize this slave as much as possible". This means that Jenkins will not boot a slave for a job unless the job specifically has been configured (via label) to use a slave that currently is not running, or unless all the build-executors on the master currently are in use. If you instead change the slave's "Usage" setting to "Leave this machine for tied jobs only", Jenkins will use the slave only if it can't run the job on the master or any other slaves. See the "Usage" settings matrix table below for a clearer description of the interaction of the "Usage" setting between slaves and the master.

Set "Idle termination time" to the number of minutes a slave must sit at idle before Jenkins shuts it down (default is 30 minutes — if you use the slaves only for scheduled jobs you might want to cut this down to 0). Paste your init script into the "Init script" field. Jenkins executes the init script by writing your script to a file to the slave, setting execute permissions on it, and then running the file as the user you specified in the "Remote user" field — so make sure that you include #!/bin/sh at the top of the script (if it's a shell script, or the appropriate "sha-bang" if you you a different scripting language).

Click the "Advanced..." button at the bottom of the AMI's configuration section to access a few more options. A couple of settings that you may want to customize are "Number of executors" (you may want to set this to 1 — unless you intend to run multiple jobs on the same slave at the same time); and "Instance Cap", which is the maximum number of slaves from this profile that Jenkins can have running at the same time.

Click the "Save" button at the bottom of the page to save and apply your changes.

Configure the job-selecting behavior of the master

Once you've saved your first slave profile, go back to the same "Manage Jenkins" > "Configure System" page, and at the top of the page you'll find a new "Labels" and a new "Usage" field (just below the "# of executors" field). The "Usage" field determines how Jenkins utilizes the master for jobs. If you leave the "Utilize this slave as much as possible" option selected, whenever a job is triggered that either doesn't have a label, or is labeled so that it could be executed either by the master or by another slave, Jenkins will run the job on an already-running slave only if the slave has a free executor; but otherwise it will try to run the job on the master, and only boot a new slave if the master has no free executors. If you change the "Usage" field to "Leave this machine for tied jobs only", whenever a job is triggered that either doesn't have a label, or is labeled so that it could be executed either by the master or by another slave, Jenkins will first try a running slave, and then try booting a slave; and only try using the master if it can't boot any more slaves.

Here's a description of the interaction between the master and slave "Usage" settings in tabular form:

"Usage" setting matrix

		Master
		Utilize this slave as much as possible	Leave this machine for tied jobs only
Slave	Utilize this slave as much as possible	use executor on running slave if free use executor on master if free boot new slave if below instance cap wait	use executor on running slave if free boot new slave if below instance cap use executor on master if free wait
	Leave this machine for tied jobs only	use executor on master if free use executor on running slave if free boot new slave if below instance cap wait	use executor on running slave if free use executor on master if free boot new slave if below instance cap wait

Note that if a job is labeled with a label that the master doesn't have, Jenkins will not run it on the master regardless of the "Usage" setting — it will wait until it can boot or otherwise free up an executor on a slave that does have that label. So if there are jobs that you do want to run on the master, do give the master a label or two via the "Labels" field. For example, if you want to allow only small jobs and one or two master-specific jobs to run on the master, you might label the master "small master".

Earmark jobs for slaves with labels

The last step is to re-configure individual jobs to run on your new slaves. You can skip this step entirely if any slave can take any job — just configure the "Usage" setting of the master and slaves to indicate how Jenkins should utilize the slaves (if it should boot slaves to run jobs on them even if the master is free, or if it should max out the master before booting slaves).

Otherwise, navigate to the configuration page of each job for which you want to specify the type of slave to run the job, and check the "Restrict where this project can be run" checkbox at the bottom of the first section of the page. This will reveal the "Label expression" field; enter the label (or space-separated labels) that defines what kind of slave the job requires. For example, if the job requires a MySQL DB, you might enter "mysql" as the label (requiring a slave with the "mysql" label); or if the job can run on any small instance, you might enter "small" as the label (requiring a slave with the "small" label).

The next time you trigger a job that is labeled for a slave, Jenkins automatically will boot the slave (if no slaves of that type are currently running and have free executors), and run the job. The display of Jenkins' leftnav will also change, to include the list of executors on the slave in the "Build Executor" box (and when the slave is terminated, the slave's executors will be removed from this box).