Elixir Systemd Logging

If you run an Elixir application as a Linux service with systemd, you'll probably find that logging works pretty well out of the box. By default, Elixir uses the Console logger backend, which sends all log messages to stdout. And with systemd services, by default all stdout messages are sent to journald.

This means you can view your application's logs easily via the journalctl command. For example, you can "tail" your app's logs with a command like this (if the systemd unit for the app was named my_app):

journalctl -u my_app -f

You can also configure systemd to send your app's stdout to a custom log file instead of journald, using the StandardOutput directive. You can add that directive to the [Service] section of a systemd unit file (for example, to log to a custom /var/log/my_app.log):

# /etc/systemd/system/my_app.service [Service] ExecStart=/srv/my_app/bin/my_app start ExecStop=/srv/my_app/bin/my_app stop StandardOutput=append:/var/log/my_app.log

Problems

If you collect and ship your log messages off to a centralized log service (like AWS CloudWatch, Google Cloud Logging, Azure Monitor, Splunk, Sumologic, Elasticsearch, Loggly, Datadog, New Relic, etc), you'll find two problems with this, however:

1. Multi-line messages are broken up into a separate log entry for each line
2. Log level/priority is lost

You can add some steps further down your logging pipeline to try to correct this, but the easiest way to fix it is at the source: Replace the default Console logger with the ExSyslogger backend.

(Update for 2025: see my Elixir Syslog Logger blog post for an alternative with a "modern" logger handler instead of a now-deprecated logger backend.)

Here's how you'd do that with a Phoenix web app:

1. Add the ex_syslogger dependency

First, add the ex_syslogger library as a dependency to your mix.exs file:

# mix.exs defp deps do [ {:ex_syslogger, "~> 1.5"} ] end

2. Register the ex_syslogger backend

Update the root config :logger options in your config/prod.exs file to register the ExSyslogger backend under the name :ex_syslogger:

# config/prod.exs # Do not print debug messages in production config :logger, level: :info config :logger, level: :info, backends: [{ExSyslogger, :ex_syslogger}]

Note that the :ex_syslogger name isn't special — you can call it whatever you want. It just has to match the name you use in the next section:

3. Configure the ex_syslogger backend

Now add config :logger, :ex_syslogger options to your config/config.exs file to configure the backend named :ex_syslogger that you registered above. I'd suggest just duplicating the configuration you already have for the default :console backend, plus setting the syslog APP-NAME field to your app's name via the ident option:

# config/config.exs # Configures Elixir's Logger config :logger, :console, format: "$time $metadata[$level] $message\n", metadata: [:request_id] config :logger, :ex_syslogger, format: "$time $metadata[$level] $message\n", metadata: [:request_id], ident: "my_app"

Result

Now when you compile your app with MIX_ENV=prod and run it as a systemd service, journald will automatically handle multi-line messages and log levels/priorities correctly. Furthermore, you can use any generic syslog collector to ship log entries to your log service as soon as they occur — with multi-line messages and log levels intact.

For example, when using the default Console logger, an error message from a Phoenix web app would have been displayed like this by journalctl:

$ journalctl -u my_app -f Mar 26 18:21:10 foo my_app[580361]: 18:21:10.337 request_id=Fm_3dFhPMtEHARkAAALy [info] Sent 500 in 16ms Mar 26 18:21:10 foo my_app[580361]: 18:21:10.345 [error] #PID<0.4149.0> running MyAppWeb.Endpoint (connection #PID<0.4148.0>, stream id 1) terminated Mar 26 18:21:10 foo my_app[580361]: Server: foo.example.com:443 (https) Mar 26 18:21:10 foo my_app[580361]: Request: GET /test/error Mar 26 18:21:10 foo my_app[580361]: ** (exit) an exception was raised: Mar 26 18:21:10 foo my_app[580361]: ** (RuntimeError) test runtime error Mar 26 18:21:10 foo my_app[580361]: (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:9: MyAppWeb.TestController.error/2 Mar 26 18:21:10 foo my_app[580361]: (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:1: MyAppWeb.TestController.action/2 Mar 26 18:21:10 foo my_app[580361]: (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:1: MyAppWeb.TestController.phoenix_controller_pipeline/2 Mar 26 18:21:10 foo my_app[580361]: (phoenix 1.5.8) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2 Mar 26 18:21:10 foo my_app[580361]: (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2 Mar 26 18:21:10 foo my_app[580361]: (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2 Mar 26 18:21:10 foo my_app[580361]: (phoenix 1.5.8) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4 Mar 26 18:21:10 foo my_app[580361]: (cowboy 2.8.0) /srv/my_app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2

But with ExSyslogger in place, you'll now see this (where the full error message is captured as a single log entry, and is recognized as an error-level message):

$ journalctl -u my_app -f Mar 26 18:21:10 foo my_app[580361]: 18:21:10.337 request_id=Fm_3dFhPMtEHARkAAALy [info] Sent 500 in 16ms Mar 26 18:21:10 foo my_app[580361]: 18:21:10.345 [error] #PID<0.4149.0> running MyAppWeb.Endpoint (connection #PID<0.4148.0>, stream id 1) terminated Server: foo.example.com:443 (https) Request: GET /test/error ** (exit) an exception was raised: ** (RuntimeError) test runtime error (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:9: MyAppWeb.TestController.error/2 (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:1: MyAppWeb.TestController.action/2 (my_app 0.1.0) lib/my_app_web/controllers/test_controller.ex:1: MyAppWeb.TestController.phoenix_controller_pipeline/2 (phoenix 1.5.8) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2 (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2 (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2 (phoenix 1.5.8) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4 (cowboy 2.8.0) /srv/my_app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2

And as a side note, you can use journalctl to view just error-level messages and above via the --priority=err flag (-p3 for short):

journalctl -u my_app -p3

D3v6 Pan and Zoom

Since D3 version 3 it's been really easy to add panning and zooming to custom visualizations, allowing the user to scroll the SVG canvas vertically and horizontally by clicking and dragging the mouse cursor around the canvas, and to scale the canvas larger and smaller by spinning the mouse wheel.

Simplest way

For the simplest case, all you need is to apply the d3.zoom() behavior to your root svg element. This is how you do it with D3 version 6 (d3.v6.js):

<svg id="viz1" width="300" height="300" style="background:#ffc"> <circle cx="50%" cy="50%" r="25%" fill="#69c" /> </svg> <script> const svg = d3 .select('#viz1') .call(d3.zoom().on('zoom', ({ transform }) => svg.attr('transform', transform))) </script>

It'll work like the following:

Smoothest way

In most cases, however, you'll get smoother behavior by adding some group (<g>) elements to wrap your main visualization elements. If you're starting with a structure like the following, where you've got a .canvas group element containing the main content you want to pan and zoom:

<svg id="viz2" width="300" height="300"> <g class="canvas" transform="translate(150,150)"> <circle cx="0" cy="0" r="25%" fill="#69c" /> </g> </svg>

Do this: add one wrapper group element, .zoomed, around the original .canvas group; and a second group element, .bg, around .zoomed; and add a rect inside the .bg group:

<svg id="viz2" width="300" height="300"> <g class="bg"> <rect width="100%" height="100%" fill="#efc" /> <g class="zoomed"> <g class="canvas" transform="translate(150,150)"> <circle cx="0" cy="0" r="25%" fill="#69c" /> </g> </g> </g> </svg>

The rect inside the .bg group will ensure that the user's click-n-drag or mouse wheeling will be captured as long as the mouse pointer is anywhere inside the svg element (without this rect, the mouse would be captured only when the user positions the mouse over a graphical element drawn inside the .bg group — like the circle in this example). For this example, I've set the fill of the rect to a light green-yellow color; but usually you'd just set it to transparent.

Then attach the pan & zoom behavior to the .bg group — but apply the pan & zoom transform to the .zoomed group it contains. This will prevent stuttering when panning, since the .bg group will remain fixed; and it will avoid messing with any transforms or other fancy styling/positioning you already have on your inner .canvas group:

<script> const zoomed = d3.select('#viz2 .zoomed') const bg = d3 .select('#viz2 .bg') .call( d3 // base d3 pan & zoom behavior .zoom() // limit zoom to between 20% and 200% of original size .scaleExtent([0.2, 2]) // apply pan & zoom transform to 'zoomed' element .on('zoom', ({ transform }) => zoomed.attr('transform', transform)) // add 'grabbing' class to 'bg' element when panning; // add 'scaling' class to 'bg' element when zooming .on('start', ({ sourceEvent: { type } }) => { bg.classed(type === 'wheel' ? 'scaling' : 'grabbing', true) }) // remove 'grabbing' and 'scaling' classes when done panning & zooming .on('end', () => bg.classed('grabbing scaling', false)), ) </script>

Finally, set the mouse cursor via CSS when the user positions the pointer over the rect element. The grabbing and scaling classes will be added to the .bg group while the pan or zoom activity is ongoing, via the on('start') and on('end') hooks above:

<style lang="css"> .bg > rect { cursor: move; } .bg.grabbing > rect { cursor: grabbing; } .bg.scaling > rect { cursor: zoom-in; } </style>

When you put it all together, it will work like the following:

Using an Ecto Readonly Replica Repo

Elixir Ecto has excellent documentation for how to use read-only replica databases, but because I'm so dense it took me a bit of trial and error to figure out where all the changes suggested by the documentation should go in my own app. Here's a concrete example of what I had to change for my conventional Mix + Phoenix application.

(The docs describe how to add N different replicas dynamically — MyApp.Repo.Replica1, MyApp.Repo.Replica2, etc — but since I only have to worry about a single endpoint for my read replicas, I simplified things and just used a single, static MyApp.Repo.Replica instance in my Elixir configuration and code.)

Mix Environment Helper

To allow my app to determine whether it was compiled and is running with a test, dev, or prod configuration, I added a config_env setting to my app, and set it to the value of the Mix.env/0 function at compile time:

# config/config.exs config :my_app, config_env: Mix.env(), ecto_repos: [MyApp.Repo] end

Note that with Elixir 1.11 and newer, you can instead use Config.config_env/0 in place of Mix.env/0:

# config/config.exs config :my_app, config_env: Config.config_env(), ecto_repos: [MyApp.Repo] end

And in my root MyApp module, I added a helper function to access this config_env setting:

# lib/my_app.ex defmodule MyApp do def config_env, do: Application.get_env(:my_app, :config_env) end

This means that I can call MyApp.config_env/0 at runtime in various places in my app's code, and get the Mix.env/0 value with which the app was compiled (like :test, :dev, or :prod).

Replica Module

To my existing lib/my_app/repo.ex file (which already contained the MyApp.Repo module), I added the definition for my new MyApp.Repo.Replica module, like so:

# lib/my_app/repo.ex defmodule MyApp.Repo do use Ecto.Repo, otp_app: :my_app, adapter: Ecto.Adapters.Postgres def replica, do: MyApp.Repo.Replica end defmodule MyApp.Repo.Replica do use Ecto.Repo, otp_app: :my_app, adapter: Ecto.Adapters.Postgres, default_dynamic_repo: if(MyApp.config_env() != :test, do: MyApp.Repo.Replica, else: MyApp.Repo), read_only: true end

The default_dynamic_repo option in the MyApp.Repo.Replica module uses the config_env helper I added above to set up the module to use the primary MyApp.Repo's own connection pool for the read replica in the test environment, as recommended by the Ecto docs. This way the replica instance will just delegate to the primary repo instance for all of its read operations in the test environment, but will still enforce its own read-only setting. Also, this way I don't have to configure any test-env-specific settings for the read replica in my config/test.exs file (nor do I need to start up another child process for the replica, as we'll see in the next section).

Application Module

In non-test environments, the new read replica module does need to be started as a child process, alongside the primary repo. So I modified the start/2 function in my application module to start it:

# lib/my_app/application.ex defmodule MyApp.Application do use Application def start(_type, _args) do # don't start separate readonly repo in test mode repos = if MyApp.config_env() != :test do [MyApp.Repo, MyApp.Repo.Replica] else [MyApp.Repo] end children = repos ++ [ MyApp.Repo, MyAppWeb.Endpoint ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end

Dev Config

For my dev environment configuration, I updated my config/dev.exs file to simply duplicate the configuration of the primary MyApp.Reop for the MyApp.Repo.Replica (creating a separate connection pool to the same database as the primary for the replica):

# config/dev.exs config :my_app, MyApp.Repo, for repo <- [MyApp.Repo, MyApp.Repo.Replica] do config :my_app, repo, username: "myusername", password: "mypassword", database: "mydatabase", hostname: "localhost", show_sensitive_data_on_connection_error: true, pool_size: 10 end

Prod Config

For the prod environment configuration, I updated my config/releases.exs file to use a similar configuration as the primary for the replica, but have it instead pull the replica hostname from a different environment variable (DB_READONLY in this case):

# config/releases.exs config :my_app, MyApp.Repo, ssl: true, username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), database: System.get_env("DB_DATABASE"), hostname: System.get_env("DB_HOSTNAME"), pool_size: String.to_integer(System.get_env("DB_POOLSIZE") || "10") config :my_app, MyApp.Repo.Replica, ssl: true, username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), database: System.get_env("DB_DATABASE"), hostname: System.get_env("DB_READONLY"), pool_size: String.to_integer(System.get_env("DB_POOLSIZE") || "10")

Using the Replica

With all the above in place, everywhere in my Elixir code that I want to query a read replica instead the primary database, I can just replace MyApp.Repo with MyApp.Repo.replica():

# lib/my_app/users.ex import Ecto.Query alias MyApp.Repo alias MyApp.Users.User def list_usernames do from(u in User, select: u.username) |> Repo.replica().all() end

Ecto RDS SSL Connection with Certificate Verification

It's nice and easy to connect to an AWS RDS instance with Elixir Ecto over SSL/TLS, as long as you're not worried about verifying the database server's certificate. You just add a ssl: true setting when your configure the Ecto Repo, like this snippet from a config/releases.exs file for a hypothetical "myapp":

# config/releases.exs config :myapp, MyApp.Repo, hostname: System.get_env("DB_HOSTNAME"), database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: true

That's probably good enough for most cloud environments; but if you want to defend against a sophisticated attacker eavesdropping on or manipulating the SSL connections between your DB client and the RDS server, you also need to configure your Ecto Repo's ssl_opts setting to verify the server's certificate.

Unfortunately, this is not so straightforward. You need to either write your own certificate verification function (not trivial), or use one supplied by another library — like the ssl_verify_fun.erl library.

To use the :ssl_verify_hostname verification function from the ssl_verify_fun.erl library, first add the library as a dependency to your mix.exs file:

# mix.exs defp deps do [ {:ecto_sql, "~> 3.5"}, {:ssl_verify_fun, ">= 0.0.0"} ] end

Then add the following ssl_opts setting to your Ecto Repo config:

# config/releases.exs check_hostname = String.to_charlist(System.get_env("DB_HOSTNAME")) config :myapp, MyApp.Repo, hostname: System.get_env("DB_HOSTNAME"), database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: true, ssl_opts: [ cacertfile: "/etc/ssl/certs/rds-ca-2019-root.pem", server_name_indication: check_hostname, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: check_hostname]} ]

Note the RDS server hostname (which would be something like my-rds-cluster.cluster-abcd1234efgh.us-east-1.rds.amazonaws.com) needs to be passed to the server_name_indication and check_hostname options as a charlist. The above example also assumes that you have downloaded the root RDS SSL certificate to /etc/ssl/certs/rds-ca-2019-root.pem on your DB client hosts.

I'd also suggest pulling out the generation of ssl_opts into a function, to make it easy to set up multiple repos. This is the way I'd do it with out our hypothetical "myapp" repo: I'd add one environment variable (DB_SSL) to trigger the Ecto ssl setting (with or without verifying the server cert), and another environment variable (DB_SSL_CA_CERT) to specify the path for the cacertfile option (triggering cert verification):

# config/releases.exs make_ssl_opts = fn "", _hostname -> [] cacertfile, hostname -> check_hostname = String.to_charlist(hostname) [ cacertfile: cacertfile, server_name_indication: check_hostname, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: check_hostname]} ] end db_ssl_ca_cert = System.get_env("DB_SSL_CA_CERT", "") db_ssl = db_ssl_ca_cert != "" or System.get_env("DB_SSL", "") != "" db_hostname = System.get_env("DB_HOSTNAME") config :myapp, MyApp.Repo, hostname: db_hostname, database: System.get_env("DB_DATABASE"), username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), ssl: db_ssl, ssl_opts: make_ssl_opts.(db_ssl_ca_cert, db_hostname)

With this verification in place, you'd see an error like the following if your DB client tries to connect to a server with a SSL certificate signed by a CA other than the one you configured:

{:tls_alert, {:unknown_ca, 'TLS client: In state certify at ssl_handshake.erl:1950 generated CLIENT ALERT: Fatal - Unknown CA\n'}}

And you'd see an error like the following if the certificate was signed by the expected CA, but for a different hostname:

{bad_cert,unable_to_match_altnames} - {:tls_alert, {:handshake_failure, 'TLS client: In state certify at ssl_handshake.erl:1952 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,unable_to_match_altnames}'}}

Using Logstash to Ingest CloudFront Logs Into Elasticsearch

Elasticsearch can be a good way of monitoring usage of your AWS CloudFront websites. There are some fairly straightforward paths to shipping CloudFront logs to hosted Elasticsearch services like Logz.io or Amazon Elasticsearch. Here's how to do it with your own self-hosted Elasticsearch and Logstash instances:

1. Set up CloudFront logging
2. Set up SQS notifications
3. Set up test Logstash pipeline
4. Set up main Logstash pipeline
5. View logs in Kibana

Set up CloudFront logging

First, you need an S3 bucket to store your CloudFront logs. You can use an existing bucket, or create a new one. You don't need to set up any special permissions for the bucket — but you probably will want to make sure the bucket denies public access to its content by default. In this example, we'll use an S3 bucket for logs called my-log-bucket, and we'll store our CloudFront logs under a directory of the bucket called my-cloudfront-logs. Also, we'll store each CloudFront distribution's logs in their own subdirectory of that directory; so for the distribution serving the www.example.com domain, we'll store the distributions logs under the my-cloudfront-logs/www.example.com subdirectory.

With the S3 logging bucket created and available, update each of your CloudFront distributions to log to it. You can do this via the AWS console by editing the distribution, turning the "Standard Logging" setting on, setting the "S3 Bucket for Logs" to your S3 logging bucket (my-log-bucket.s3.amazonaws.com), and setting the "Log Prefix" to the directory path of the subdirectory of the S3 bucket under which you'll store the logs (my-cloudfront-logs/www.example.com/). Save your changes, and every few minutes CloudFront will save a new .gz file to the my-cloudfront-logs/www.example.com/ subdirectory of the my-log-bucket (see the CloudFront access logs docs for details).

Set up SQS notifications

Next, create a new SQS queue. We'll call ours my-cloudfront-log-notifications, and we'll create it in the us-east-1 AWS region. When you create the queue, configure its "Receive message wait time" setting to 10 seconds or so; this will ensure the SQS client doesn't make way more SQS requests than needed (a setting of 10 seconds should keep the cost of this queue down to less than $1/month).

The only other thing special you need to do when you create the queue is add an access policy to it that allows S3 to send messages to it. The policy should look like this (replace my-cloudfront-log-notifications with the name of your queue, us-east-1 with your queue's region, my-log-bucket with the name of your log bucket, and 123456789012 with your AWS account ID):

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" }, "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-log-bucket" } } } ] }

With the SQS queue created, update the S3 bucket to send all object-create events to the queue. You can do this via the AWS console by selecting the bucket and opening the "Events" block in the "Advanced Settings" section of the "Properties" tab of the bucket. There you can add a notification; name it my-cloudfront-log-configuration, check the "All object create events" checkbox, set the "Prefix" to my-cloudfront-logs/, and send it to your SQS queue my-cloudfront-log-notifications.

Alternately, you can add a notification with the same settings as above via the put-bucket-notification-configuration command of the s3api CLI, using a notification-configuration JSON file like the following:

{ "QueueConfigurations": [ { "Id": "my-cloudfront-log-configuration", "QueueArn": "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications", "Events": [ "s3:ObjectCreated:*" ], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "my-cloudfront-logs/" } ] } } } ] }

Now that you've hooked up S3 bucket notifications to the SQS queue, if you look in the AWS console for the SQS queue, under the Monitoring tab's charts you'll start to see messages received every few minutes.

Set up test Logstash pipeline

Download a sample .gz log file from your S3 logging bucket, and copy it over to the machine you have Logstash running on. Move the file to a directory that Logstash can access, and make sure it has read permissions on the file. Our sample file will live at /var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz.

Copy the following my-cloudfront-pipeline.conf file into the /etc/logstash/conf.d directory on your Logstash machine (replacing the input path with your sample .gz log file), tail the Logstash logs (journalctl -u logstash -f if managed with systemd), and restart the Logstash service (sudo systemctl restart logstash):

# /etc/logstash/conf.d/my-cloudfront-pipeline.conf input { file { file_completed_action => "log" file_completed_log_path => "/var/lib/logstash/cloudfront-completed.log" mode => "read" path => "/var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz" sincedb_path => "/var/lib/logstash/cloudfront-since.db" type => "cloudfront" } } filter { if [type] == "cloudfront" { if (("#Version: 1.0" in [message]) or ("#Fields: date" in [message])) { drop {} } mutate { rename => { "type" => "[@metadata][type]" } # strip dashes that indicate empty fields gsub => ["message", "\t-(?=\t)", " "] # literal tab } #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end csv { separator => " " # literal tab columns => [ "date", "time", "x_edge_location", "sc_bytes", "c_ip", "cs_method", "cs_host", "cs_uri_stem", "sc_status", "cs_referer", "cs_user_agent", "cs_uri_query", "cs_cookie", "x_edge_result_type", "x_edge_request_id", "x_host_header", "cs_protocol", "cs_bytes", "time_taken", "x_forwarded_for", "ssl_protocol", "ssl_cipher", "x_edge_response_result_type", "cs_protocol_version", "fle_status", "fle_encrypted_fields", "c_port", "time_to_first_byte", "x_edge_detailed_result_type", "sc_content_type", "sc_content_len", "sc_range_start", "sc_range_end" ] convert => { "c_port" => "integer" "cs_bytes" => "integer" "sc_bytes" => "integer" "sc_content_len" => "integer" "sc_range_end" => "integer" "sc_range_start" => "integer" "sc_status" => "integer" "time_taken" => "float" "time_to_first_byte" => "float" } add_field => { "datetime" => "%{date} %{time}" "[@metadata][document_id]" => "%{x_edge_request_id}" } remove_field => ["cloudfront_fields", "cloudfront_version", "message"] } # parse datetime date { match => ["datetime", "yy-MM-dd HH:mm:ss"] remove_field => ["datetime", "date", "time"] } # lookup geolocation of client ip address geoip { source => "c_ip" target => "geo" } # parse user-agent into subfields urldecode { field => "cs_user_agent" } useragent { source => "cs_user_agent" target => "ua" add_field => { "user_agent.name" => "%{[ua][name]}" "user_agent.version" => "%{[ua][major]}" "user_agent.device.name" => "%{[ua][device]}" "user_agent.os.name" => "%{[ua][os_name]}" "user_agent.os.version" => "%{[ua][os_major]}" } remove_field => ["cs_user_agent", "ua"] } # pull logfile path from s3 metadata, if present if [@metadata][s3][object_key] { mutate { add_field => { "path" => "%{[@metadata][s3][object_key]}" } } } # strip directory path from logfile path, and canonicalize field name mutate { rename => { "path" => "log.file.path" } gsub => ["log.file.path", ".*/", ""] remove_field => "host" } # canonicalize field names, and drop unwanted fields mutate { rename => { "c_ip" => "client.ip" "cs_bytes" => "http.request.bytes" "sc_content_len" => "http.response.body.bytes" "sc_content_type" => "http.response.body.type" "cs_method" => "http.request.method" "cs_protocol" => "url.scheme" "cs_protocol_version" => "http.version" "cs_referer" => "http.request.referrer" "cs_uri_query" => "url.query" "cs_uri_stem" => "url.path" "sc_bytes" => "http.response.bytes" "sc_status" => "http.response.status_code" "ssl_cipher" => "tls.cipher" "ssl_protocol" => "tls.protocol_version" "x_host_header" => "url.domain" } gsub => [ "http.version", "HTTP/", "", "tls.protocol_version", "TLSv", "" ] remove_field => [ "c_port", "cs_cookie", "cs_host", "fle_encrypted_fields", "fle_status", "sc_range_end", "sc_range_start", "x_forwarded_for" ] } } } output { stdout { codec => "rubydebug" } }

You should see a bunch of entries in the Logstash logs like the following, one for each entry from your sample log file (note the fields will appear in a different order every time you run this):

Jan 02 03:04:05 logs1 logstash[12345]: { Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_detailed_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "@timestamp" => 2020-01-02T03:01:02.000Z, Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.device.name" => "EML-AL00", Jan 02 03:04:05 logs1 logstash[12345]: "time_taken" => 0.001, Jan 02 03:04:05 logs1 logstash[12345]: "http.version" => "2.0", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.os.version" => "8", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.body.bytes" => nil, Jan 02 03:04:05 logs1 logstash[12345]: "tls.cipher" => "ECDHE-RSA-AES128-GCM-SHA256", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.bytes" => 2318, Jan 02 03:04:05 logs1 logstash[12345]: "@version" => "1", Jan 02 03:04:05 logs1 logstash[12345]: "time_to_first_byte" => 0.001, Jan 02 03:04:05 logs1 logstash[12345]: "http.request.method" => "GET", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_request_id" => "s7lmJasUXiAm7w2oR34Gfg5zTgeQSTkYwiYV1pnz5Hzv8mRmBzyGrw==", Jan 02 03:04:05 logs1 logstash[12345]: "log.file.path" => "EML9FBPJY2494.2020-01-02-03.abcd1234.gz", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "http.request.bytes" => 388, Jan 02 03:04:05 logs1 logstash[12345]: "http.request.referrer" => "http://baidu.com/", Jan 02 03:04:05 logs1 logstash[12345]: "client.ip" => "192.0.2.0", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.name" => "UC Browser", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.version" => "11", Jan 02 03:04:05 logs1 logstash[12345]: "url.query" => nil, Jan 02 03:04:05 logs1 logstash[12345]: "http.response.body.type" => "text/html", Jan 02 03:04:05 logs1 logstash[12345]: "url.domain" => "www.example.com", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_location" => "LAX50-C3", Jan 02 03:04:05 logs1 logstash[12345]: "http.response.status_code" => 200, Jan 02 03:04:05 logs1 logstash[12345]: "geo" => { Jan 02 03:04:05 logs1 logstash[12345]: "ip" => "192.0.2.0", Jan 02 03:04:05 logs1 logstash[12345]: "region_name" => "Shanghai", Jan 02 03:04:05 logs1 logstash[12345]: "country_name" => "China", Jan 02 03:04:05 logs1 logstash[12345]: "timezone" => "Asia/Shanghai", Jan 02 03:04:05 logs1 logstash[12345]: "longitude" => 121.4012, Jan 02 03:04:05 logs1 logstash[12345]: "country_code3" => "CN", Jan 02 03:04:05 logs1 logstash[12345]: "location" => { Jan 02 03:04:05 logs1 logstash[12345]: "lon" => 121.4012, Jan 02 03:04:05 logs1 logstash[12345]: "lat" => 31.0449 Jan 02 03:04:05 logs1 logstash[12345]: }, Jan 02 03:04:05 logs1 logstash[12345]: "region_code" => "SH", Jan 02 03:04:05 logs1 logstash[12345]: "country_code2" => "CN", Jan 02 03:04:05 logs1 logstash[12345]: "continent_code" => "AS", Jan 02 03:04:05 logs1 logstash[12345]: "latitude" => 31.0449 Jan 02 03:04:05 logs1 logstash[12345]: }, Jan 02 03:04:05 logs1 logstash[12345]: "url.scheme" => "https", Jan 02 03:04:05 logs1 logstash[12345]: "tls.protocol_version" => "1.2", Jan 02 03:04:05 logs1 logstash[12345]: "user_agent.os.name" => "Android", Jan 02 03:04:05 logs1 logstash[12345]: "x_edge_response_result_type" => "Hit", Jan 02 03:04:05 logs1 logstash[12345]: "url.path" => "/" Jan 02 03:04:05 logs1 logstash[12345]: }

These entries show you what Logstash will push to Elasticsearch, once you hook it up. You can adjust this my-cloudfront-pipeline.conf file and restart Logstash again and again until you get the exact field names and values that you want to push to Elasticsearch.

Let's look at each part of the pipeline individually.

In the input section, we're using the file input to read just our one sample file:

input { file { file_completed_action => "log" file_completed_log_path => "/var/lib/logstash/cloudfront-completed.log" mode => "read" path => "/var/log/my-cloudfront-logs/www.example.com/E123456789ABCD.2020-01-02-03.abcd1234.gz" sincedb_path => "/var/lib/logstash/cloudfront-since.db" type => "cloudfront" } }

The key bit here is that we set the type field to cloudfront, which we'll use in the filter section below to apply our filtering logic only to entries of this type. If you're only going to process CloudFront log files in this pipeline, you can omit all the bits of the pipeline that deal with "type", which would simplify it some.

In the filter section, the first step is to check if the type field was set to "cloudfront", and only execute the rest of the filter block if so:

filter { if [type] == "cloudfront" {

Then the next step in filter section is to drop the two header lines in each CloudFront log file, the first beginning with #Version, and the second beginning with #Fields:

if (("#Version: 1.0" in [message]) or ("#Fields: date" in [message])) { drop {} }

After that, the next step renames the type field to [@metadata][type], so that it won't be pushed to the Elasticsearch index. I've opted to use Elasticsearch indexes that are for my CloudFront logs only; however, if you want to push your CloudFront logs into indexes that are shared with other data, you may want to keep the type field.

mutate { rename => { "type" => "[@metadata][type]" }

The second half of this mutate filter strips out the - characters that indicate empty field values from all the columns in the log entry. Note that the last argument of this gsub function is a literal tab character — make sure your text editor does not convert it to spaces!

# strip dashes that indicate empty fields gsub => ["message", "\t-(?=\t)", " "] # literal tab }

For example, it will convert a entry like this:

2020-01-02 03:03:03 HIO50-C1 6564 192.0.2.0 GET d2c4n4ttot8c65.cloudfront.net / 200 - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:40.0)%20Gecko/20100101%20Firefox/40.1 - - Miss nY0knXse4vDxS5uOBe3YAhDpH809bqhsILUUFAtE_4ZLlfXCiYcD0A== www.example.com https 170 0.164 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/1.1 - - 62684 0.164 Miss text/html 6111 - -

Into this (removing the dashes that indicate empty values, but not the dashes in non-empty values like the date or ciphersuite):

2020-01-02 03:03:03 HIO50-C1 6564 192.0.2.0 GET d2c4n4ttot8c65.cloudfront.net / 200 Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:40.0)%20Gecko/20100101%20Firefox/40.1 Miss nY0knXse4vDxS5uOBe3YAhDpH809bqhsILUUFAtE_4ZLlfXCiYcD0A== www.example.com https 170 0.164 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/1.1 62684 0.164 Miss text/html 6111

The next step is the meat of the process, using the csv filter to convert each tab-separated log line into named fields. Note that the separator property value is also a literal tab character:

#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end csv { separator => " " # literal tab columns => [ "date", "time", "x_edge_location", "sc_bytes", "c_ip", "cs_method", "cs_host", "cs_uri_stem", "sc_status", "cs_referer", "cs_user_agent", "cs_uri_query", "cs_cookie", "x_edge_result_type", "x_edge_request_id", "x_host_header", "cs_protocol", "cs_bytes", "time_taken", "x_forwarded_for", "ssl_protocol", "ssl_cipher", "x_edge_response_result_type", "cs_protocol_version", "fle_status", "fle_encrypted_fields", "c_port", "time_to_first_byte", "x_edge_detailed_result_type", "sc_content_type", "sc_content_len", "sc_range_start", "sc_range_end" ] }

The columns property lists out each field name, in order. Later on in this pipeline, we'll rename many of these fields to use the ECS nomenclature, but this step uses the field names as defined by CloudFront, for clarity.

The middle part of the csv filter converts the numeric fields to actual numbers, via the convert property mapping:

convert => { "c_port" => "integer" "cs_bytes" => "integer" "sc_bytes" => "integer" "sc_content_len" => "integer" "sc_range_end" => "integer" "sc_range_start" => "integer" "sc_status" => "integer" "time_taken" => "float" "time_to_first_byte" => "float" }

The add_field part of the csv filter combines the individual date and time fields into a combined datetime field (to be converted to a timestamp object later); and also copies the x_edge_request_id field value as the [@metadata][document_id] field:

add_field => { "datetime" => "%{date} %{time}" "[@metadata][document_id]" => "%{x_edge_request_id}" }

The [@metadata][document_id] field will be used later on when we push the record to Elasticsearch (to be used as the record's ID). Like with the [@metadata][type] field, this is another case where if you're only going to process CloudFront log files in this pipeline, you could omit this extra metadata field, and just use the x_edge_request_id directly when configuring the Elasticsearch record ID.

The final part of the csv filter removes some fields that are redundant once the log entry has been parsed: message (the full log entry text itself), and cloudfront_fields and cloudfront_version (which the s3snssqs input we'll add later automatically includes):

remove_field => ["cloudfront_fields", "cloudfront_version", "message"] }

The next filter step is to convert the datetime field (created from the date and time fields above) into a proper datetime object:

# parse datetime date { match => ["datetime", "yy-MM-dd HH:mm:ss"] remove_field => ["datetime", "date", "time"] }

This sets the datetime as the value of the @timestamp field. We'll also remove the datetime, date, and time fields, since we won't need them now that we have the parsed datetime in the @timestamp field.

The next filter uses the client IP address to lookup a probable physical location for the client:

# lookup geolocation of client ip address geoip { source => "c_ip" target => "geo" }

This creates a geo field with a bunch of subfields (like [geo][country_name], [geo][city_name], etc) containing the probable location details. Note that many IP address won't have a mapping value for many of the subfields; see the Geoip filter docs for more details.

The next filter decodes the user-agent field, and the filter after that parses it. The useragent filter parses the cs_user_agent field into the ua field, which, like the geo field, will contain a bunch of subfields. We'll pull out a few of those subfields, and add fields with ECS names for them:

# parse user-agent into subfields urldecode { field => "cs_user_agent" } useragent { source => "cs_user_agent" target => "ua" add_field => { "user_agent.name" => "%{[ua][name]}" "user_agent.version" => "%{[ua][major]}" "user_agent.device.name" => "%{[ua][device]}" "user_agent.os.name" => "%{[ua][os_name]}" "user_agent.os.version" => "%{[ua][os_major]}" } remove_field => ["cs_user_agent", "ua"] }

Since the user-agent info we want are now in those newly added user_agent.* fields, the last part of the useragent filter removes the cs_user_agent field and intermediate ua field.

When using the file input, like we are while testing this pipeline, the file input will add a path field to each record, containing the path to the file its reading. Later on, when we use the s3snssqs input, the s3snssqs input will pass the same path as the [@metadata][s3][object_key] field. So that we can access this value uniformly, regardless of which input we used, we have this next filter step, where if the [@metadata][s3][object_key] field is present, we set the path field to the [@metadata][s3][object_key] field's value:

# pull logfile path from s3 metadata, if present if [@metadata][s3][object_key] { mutate { add_field => { "path" => "%{[@metadata][s3][object_key]}" } } }

With the path field now containing the file path, regardless of input, we use the next filter to chop the path down to just the log file name (like E123456789ABCD.2020-01-02-03.abcd1234.gz):

# strip directory path from logfile path, and canonicalize field name mutate { rename => { "path" => "log.file.path" } gsub => ["log.file.path", ".*/", ""] remove_field => "host" }

We also have the filter rename the path field to log.file.path (the canonical ECS name for it); and have the filter remove the host field (added by the file input along with the path field, based on the host Logstash is running on — which we don't really care to have as part of our log record in Elasticsearch).

The last filter in our pipeline renames all CloudFront fields that have equivalent ECS (Elastic Common Schema) field names:

# canonicalize field names, and drop unwanted fields mutate { rename => { "c_ip" => "client.ip" "cs_bytes" => "http.request.bytes" "sc_content_len" => "http.response.body.bytes" "sc_content_type" => "http.response.body.type" "cs_method" => "http.request.method" "cs_protocol" => "url.scheme" "cs_protocol_version" => "http.version" "cs_referer" => "http.request.referrer" "cs_uri_query" => "url.query" "cs_uri_stem" => "url.path" "sc_bytes" => "http.response.bytes" "sc_status" => "http.response.status_code" "ssl_cipher" => "tls.cipher" "ssl_protocol" => "tls.protocol_version" "x_host_header" => "url.domain" }

To match the ECS field specs, the middle part of the filter removes the HTTP/ prefix from the http.version field values (converting values like HTTP/2.0 to just 2.0); and removes the TLSv prefix from the tls.protocol_version field values (converting values like TLSv1.2 to just 1.2):

gsub => [ "http.version", "HTTP/", "", "tls.protocol_version", "TLSv", "" ]

And finally, the last part of the filter removes miscellaneous CloudFront fields that we don't care about:

remove_field => [ "c_port", "cs_cookie", "cs_host", "fle_encrypted_fields", "fle_status", "sc_range_end", "sc_range_start", "x_forwarded_for" ] } } }

The output section of the pipeline simply outputs each log record to Logstash's own log output — which is what you see when you tail Logstash's logs:

output { stdout { codec => "rubydebug" } }

Set up main Logstash pipeline

Once you have this test pipeline working to your satisfaction, it's time to change the output section of the pipeline to push the output to Elasticsearch. Replace the output block of the /etc/logstash/conf.d/my-cloudfront-pipeline.conf file with this block (substituting your own host, user, and password settings, as well as any custom SSL settings you need — see the Elasticsearch output plugin docs for details):

output { # don't try to index anything that didn't get a document_id if [@metadata][document_id] { elasticsearch { hosts => ["https://elasticsearch.example.com:9243"] user => "elastic" password => "password123" document_id => "%{[@metadata][document_id]}" ecs_compatibility => "v1" index => "ecs-logstash-%{[@metadata][type]}-%{+YYYY.MM.dd}" } } }

This following line in this block serves as one more guard to avoid indexing anything that didn't get parsed properly (you may want to send such log entries to a dedicated errors index, to keep an eye on entries that failed to parse):

if [@metadata][document_id] {

And this line uses the [@metadata][document_id] field to set the record ID for each entry (recall in the pipeline filters, we copied the value of the CloudFront x_edge_request_id, which should be unique for each request, to the [@metadata][document_id] field):

document_id => "%{[@metadata][document_id]}"

And since our output block includes setting ecs_compatibility to v1, which directs Logstash to use ECS-compatible index templates, this line directs Logstash to create a separate index for each day and type of log entry we process:

index => "ecs-logstash-%{[@metadata][type]}-%{+YYYY.MM.dd}"

For example, Logstash will create an index named ecs-logstash-cloudfront-2020.01.02 if we process a CloudFront log entry for January 2, 2020 (or use the existing index with that name, if it already exists).

Restart Logstash once you change the output block. In Logstash's own log output, you should see entries indicating succesful connections to your Elasticsearch host, as well as a ginormous entry for the index template it installs in Elasticsearch. Once you see that, check your Elasticsearch instance — you should see a new ecs-logstash-cloudfront-YYYY.MM.DD index created, with entries from your sample CloudFront log file.

You can use this same mechanism to backfill your existing CloudFront log files to Elastic search — manually download the log files to backfill to your Logstash machine (like via the sync command of the s3 CLI), and customize the file input block's path property (with wildcards) to direct Logstash to read them in.

For future CloudFront log files, however, we're going to make one more change to our pipeline, and use the S3 via SNS/SQS input (aka s3snssqs) to pull CloudFront log files from S3 as soon as CloudFront publishes them.

First, create a new IAM policy for your Logstash machine to use that will allow it to both read from your logging bucket, and to read and delete items from the SQS queue we set up above. The policy should look like this (change the Resource elements to point to your own S3 log bucket and SQS log queue, set up in the first two sections of this article):

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-log-bucket" }, { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-log-bucket/my-cloudfront-logs/*" }, { "Effect": "Allow", "Action": [ "sqs:Get*", "sqs:List*", "sqs:ReceiveMessage", "sqs:ChangeMessageVisibility", "sqs:DeleteMessage" ], "Resource": [ "arn:aws:sqs:us-east-1:123456789012:my-cloudfront-log-notifications" ] } ] }

Then install the logstash-input-s3-sns-sqs plugin on your Logstash machine:

cd /usr/share/logstash sudo -u logstash bin/logstash-plugin install logstash-input-s3-sns-sqs

Then update the input section of your pipeline to be the following (substituting your own SQS queue name and its AWS region):

input { # pull new logfiles from s3 when notified s3snssqs { region => "us-east-1" queue => "my-cloudfront-log-notifications" from_sns => false type => "cloudfront" } }

If you're running the Logstash machine in AWS, you can use the usual EC2 instance profiles or IAM roles for tasks to grant the machine access to the policy you created above. Otherwise, you'll need to add some AWS credential settings to the s3snssqs input as well; consult the S3 input plugins docs for options (the s3snssqs input allows for the same AWS credential options as the s3 input does, but the s3 input has better documentation for them).

Now restart Logstash. You should see the same output in Logstash's own log as before; but if you check Elasticsearch, you should see new records being added.

View logs in Kibana

Eventually you'll want to create fancy dashboards in Kibana for your new CloudFront data; but for now we'll just get started by setting up a listing where you can view them in the "Discover" section of Kibana.

First log into Kibana, and navigate to the "Management" > "Stack Management" section of Kibana. Within the "Stack Management" section, if you navigate to the "Data" > "Index management" subsection, you should see a bunch of new indexes named in the form of ecs-logstash-cloudfront-YYYY.MM.DD (like ecs-logstash-cloudfront-2020.01.01 and so on):

Once you've verified Kibana is seeing the indexes, navigate to the "Kibana" > "Index Patterns" subsection, and click the "Create index pattern" button. Specify ecs-logstash-cloudfront-* as the pattern, and select @timestamp as the time field:

With the new index pattern created, navigate out of the "Stack Management" section of Kibana into the main "Kibana" > "Discover" section. This will show your most recent "Discover" search. On the left side of the page, change the selected index pattern to the pattern you just created (ecs-logstash-cloudfront-*). You should now see your most recent CloudFront entries listed (if not, use the time window selector in the top right of the page to expand the time window to include a range you know should include some entries). You can use this page to create a list with custom columns and custom filter settings for your CloudFront logs:

Building a Logstash Offline Plugin Pack with Docker

If you run a Logstash node in an environment where it doesn't have access to the public Internet, and need to install some extra plugins, you have to build an "offline plugin pack" (a zip containing the plugins and their dependencies) on a machine that does have public Internet access. You can then copy the pack to your Logstash node, and install the plugins from it directly.

Here's a quick little script I whipped up to build the offline plugin pack using the official Logstash docker container:

#!/bin/sh -e logstash_version=7.10.0 logstash_plugins=$(echo ' logstash-codec-cloudfront logstash-input-s3-sns-sqs ' | xargs) echo " bin/logstash-plugin install $logstash_plugins bin/logstash-plugin prepare-offline-pack \ --output /srv/logstash/logstash-plugins.zip \ $logstash_plugins " | docker run -i -u $(id -u) -v $(pwd):/srv/logstash --rm \ docker.elastic.co/logstash/logstash:$logstash_version /bin/sh

Set the script's logstash_version variable to the version of Logstash you're using, and set the (whitespace-separated) list of plugins in the logstash_plugins variable to the plugins you need. Run the script, and it will output a logstash-plugins.zip into your working directory.

You can then copy the logstash-plugins.zip file to your Logstash node (for example, to the /usr/share/logstash directory of the machine), and install the contained plugins like this:

cd /usr/share/logstash sudo -u logstash bin/logstash-plugin install file://logstash-plugins.zip

Make sure you run the logstash-plugin command as the same user you use to run Logstash itself (typically the logstash user) — otherwise the plugins will be installed with the wrong filesystem permissions (and you'll see errors about it when you run the main Logstash process).

Antora Deploy to S3 and CloudFront

Even though there aren't any dedicated Antora components for deploying to AWS CloudFront or S3, it's still really easy to do — most of the Antora settings you'd use for a generic web hosting site work perfectly for S3 + CloudFront. Here's how:

1. Playbook Settings
2. S3 Settings
3. CloudFront Settings
4. Upload Script
5. Redirects Script

Playbook Settings

Here's an example playbook file that you'd use to build your documentation as part of your production deploy process:

# antora-playbook.yml site: robots: allow start_page: example-user-guide::index.adoc title: Example Documentation url: https://docs.example.com content: sources: - url: https://git.example.com/my-account/my-docs.git branches: master start_path: content/* output: clean: true runtime: fetch: true ui: bundle: url: https://ci.example.com/my-account/my-docs-ui/builds/latest/ui-bundle.zip snapshot: true urls: html_extension_style: indexify

If you run Antora in the same directory as this playbook, with a command like the following, Antora will generate your site to the build/site sub-directory:

antora generate antora-playbook.yml

These are the key playbook settings for S3/CloudFront (including some settings omitted from the above playbook, because the default value is perfect already):

site.robots: Set this to allow (or disallow if you want to forbid search-engines from crawling your docs), so that Antora will generate a robots.txt file for you.

site.url: Make sure you set this to an absolute URL — doing so will trigger Antora to build out a bunch of desirable files, like a 404.html and sitemap.xml. If your documentation has its own dedicated domain name, like docs.example.com, set site.url to https://docs.example.com; if instead your documentation can be found at a sub-directory of your main website, like under the docs directory of www.example.com, set site.url to https://www.example.com/docs. In either case, omit the trailing slash (eg don't set it to https://www.example.com/docs/ — do set it to https://www.example.com/docs).

output.dir: By default, Antora will generate your site to the build/site sub-directory of whatever directory you ran the antora command from. If this is good for you, you can omit the output.dir setting; otherwise you can set output.dir to some other local filesystem path.

urls.html_extension_style: Set this to indexify, which directs Antora to a) build out each documentation page to an index.html file in a sub-directory named for the path of the page, and b) to build links to each page via the path to the page with a trailing slash. For example, for a page named how-it-works.adoc in the ROOT module of the example-ui-guide component, with indexify Antora will build the page out as a file named example-ui-guide/how-it-works/index.html (within its build/site output directory), and build links to the page as /example-ui-guide/how-it-works/. This is exactly what you want when you your site is served by S3.

urls.redirect_facility: The default setting, static is what you want for S3, so you can omit this setting from your playbook (or set it explicity to static if you like).

S3 Settings

When hosting Antora-generated sites on S3, you don't need to do anything different than you would for any other statically-generated website, so you can following any of the dozens of online guides for S3 website hosting, like Amazon's own S3 static website hosting guide. The key things you need to set are:

1. Turn on static website hosting for the S3 bucket.
2. Set the "index document" to index.html (the default for S3 website hosting).
3. Set the "error document" to 404.html (Antora generates this file for you).
4. Either configure the permissions of the S3 bucket to explicitly allow public access to read all objects in the bucket; or when you upload files to the bucket, explicitly upload them with a canned ACL setting that allows public read access (as the scripts covered later in this article will).

You need to make the files in your S3 bucket publicly-accessible (point #4 above) so that CloudFront can access them. While there technically is a way to configure S3 and CloudFront so that the files are not publicly-accessible in S3 but CloudFront can still access them (via an Origin Access Identity), it's kind of a pain. Since these files are ultimately meant to be served to the public through CloudFront anyway, it's simpler just to make them publicly-accessible in S3.

CloudFront Settings

There's also nothing special you need to do for Antora-generated sites with CloudFront — any of the dozens of online guides for S3 + CloudFront hosting will work to set it up. Just make sure that when you set the origin for your CloudFront distribution, you use the "website endpoint" of your S3 bucket, and not the standard endpoint.

For example, if your S3 bucket is named "example-bucket" and it's located in the us-west-2 region, don't use example-bucket.s3.us-west-2.amazonaws.com as your CloudFront origin — instead do use example-bucket.s3-website-us-west-2.amazonaws.com. Using the website endpoint will ensure that CloudFront serves the Antora-generated 404.html page for pages that don't exist, and that it also serves a 301 redirect for pages for which you've configured S3 to redirect (as the scripts covered later in this article will).

Upload Script

Once you've set up your Antora playbook, S3 bucket, and CloudFront distribution, you're ready to deploy your site. If you've set up your antora-playbook.yml as above, you can build your documentation, upload it to S3, and clear the CloudFront caches of the old version of your docs with the following simple script:

#!/bin/sh -e build_dir=build/site cf_distro=E1234567890ABC s3_bucket=example-bucket antora generate antora-playbook.yml aws s3 sync $build_dir s3://$s3_bucket --acl public-read --delete aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/*'

The first line generates your documentation to the build/site directory. The second line replaces the existing content of example-bucket with the content of the build/site directory (granting public read-access to each individual file uploaded). The third line clears the CloudFront caches for all the content of your CloudFront distribution.

If you documentation is part of a larger site (eg hosted as https://www.example.com/docs/ instead of being hosted as its own site (eg https://docs.example.com/), add the sub-directory under which your documentation is hosted (eg /docs) to the last two lines of the above script; for example, like the following:

aws s3 sync $build_dir s3://$s3_bucket/docs --acl public-read --delete aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/docs/*'

Redirects Script

The redirect pages that Antora will generate when you set the Antora urls.redirect_facility setting to static will work fine for your website users as is. But search engines will like it better if you serve real HTTP redirect responses (with the redirect information embedded in HTTP header fields) instead of just HTML pages that indicate that the client browser should redirect to a different location once parsed. You can get S3 + CloudFront to serve 301 Moved Permanently redirects in place of all the redirect pages Antora generates by uploading them separately to S3 with a special x-amz-website-redirect-location header.

To do so, insert the following block into your upload script between the aws s3 sync and aws cloudfront create-invalidation commands:

#!/bin/sh build_dir=build/site cf_distro=E1234567890ABC s3_bucket=docs.example.com antora generate antora-playbook.yml aws s3 sync $build_dir s3://$s3_bucket --acl public-read --delete grep -lR 'http-equiv="refresh"' $build_dir | while read file; do redirect_url=$(awk -F'"' '/rel="canonical"/ { print $4 }' $file) aws s3 cp $file s3://$s3_bucket/${file##$build_dir/} \ --website-redirect $redirect_url --acl public-read done aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/*'

The above script block will search the Antora build dir for all redirect pages (with the grep command), and loop over each (with the while command, reading the local filepath to each into the file variable). It will pull out the canonical URL of the page to redirect to from the redirect page (via the awk command, into the redirect_url variable), and re-upload the file using the --website-redirect flag of the aws s3 cp command to indicate that S3 should serve a 301 redirect to the specified URL instead of the file content itself (when accessed through the S3 website endpoint).

As a concrete example of this redirect capability, say you had a page named how-it-works.adoc in the ROOT module of your example-ui-guide component. If you added metadata to that how-it-works.adoc page to add a redirect to it from the non-existant inner-workings.adoc page (eg via a page-aliases header attribute value of inner-workings.adoc), Antora would generate the following redirect page for you at build/site/example-user-guide/inner-workings/index.html:

<!DOCTYPE html> <meta charset="utf-8"> <link rel="canonical" href="https://docs.example.com/example-user-guide/how-it-works/"> <script>location="../how-it-works/"</script> <meta http-equiv="refresh" content="0; url=../how-it-works/"> <meta name="robots" content="noindex"> <title>Redirect Notice</title> <h1>Redirect Notice</h1> <p>The page you requested has been relocated to <a href="../how-it-works/">https://docs.example.com/example-user-guide/how-it-works/</a>.</p>

The above script would re-upload this file to S3 like so (with all variables expanded, and some additional line-wrapping for legibility):

aws s3 cp build/site/example-user-guide/inner-workings/index.html \ s3://example-bucket/example-user-guide/inner-workings/index.html \ --website-redirect https://docs.example.com/example-user-guide/how-it-works/ --acl public-read

If a user (or search engine) then navigates to https://docs.example.com/example-user-guide/inner-workings/, S3 + CloudFront will send this response back:

HTTP/2 301 location: https://docs.example.com/example-user-guide/how-it-works/