Antora Deploy to S3 and CloudFront

Even though there aren't any dedicated Antora components for deploying to AWS CloudFront or S3, it's still really easy to do — most of the Antora settings you'd use for a generic web hosting site work perfectly for S3 + CloudFront. Here's how:

1. Playbook Settings
2. S3 Settings
3. CloudFront Settings
4. Upload Script
5. Redirects Script

Playbook Settings

Here's an example playbook file that you'd use to build your documentation as part of your production deploy process:

# antora-playbook.yml site: robots: allow start_page: example-user-guide::index.adoc title: Example Documentation url: https://docs.example.com content: sources: - url: https://git.example.com/my-account/my-docs.git branches: master start_path: content/* output: clean: true runtime: fetch: true ui: bundle: url: https://ci.example.com/my-account/my-docs-ui/builds/latest/ui-bundle.zip snapshot: true urls: html_extension_style: indexify

If you run Antora in the same directory as this playbook, with a command like the following, Antora will generate your site to the build/site sub-directory:

antora generate antora-playbook.yml

These are the key playbook settings for S3/CloudFront (including some settings omitted from the above playbook, because the default value is perfect already):

site.robots: Set this to allow (or disallow if you want to forbid search-engines from crawling your docs), so that Antora will generate a robots.txt file for you.

site.url: Make sure you set this to an absolute URL — doing so will trigger Antora to build out a bunch of desirable files, like a 404.html and sitemap.xml. If your documentation has its own dedicated domain name, like docs.example.com, set site.url to https://docs.example.com; if instead your documentation can be found at a sub-directory of your main website, like under the docs directory of www.example.com, set site.url to https://www.example.com/docs. In either case, omit the trailing slash (eg don't set it to https://www.example.com/docs/ — do set it to https://www.example.com/docs).

output.dir: By default, Antora will generate your site to the build/site sub-directory of whatever directory you ran the antora command from. If this is good for you, you can omit the output.dir setting; otherwise you can set output.dir to some other local filesystem path.

urls.html_extension_style: Set this to indexify, which directs Antora to a) build out each documentation page to an index.html file in a sub-directory named for the path of the page, and b) to build links to each page via the path to the page with a trailing slash. For example, for a page named how-it-works.adoc in the ROOT module of the example-ui-guide component, with indexify Antora will build the page out as a file named example-ui-guide/how-it-works/index.html (within its build/site output directory), and build links to the page as /example-ui-guide/how-it-works/. This is exactly what you want when you your site is served by S3.

urls.redirect_facility: The default setting, static is what you want for S3, so you can omit this setting from your playbook (or set it explicity to static if you like).

S3 Settings

When hosting Antora-generated sites on S3, you don't need to do anything different than you would for any other statically-generated website, so you can following any of the dozens of online guides for S3 website hosting, like Amazon's own S3 static website hosting guide. The key things you need to set are:

1. Turn on static website hosting for the S3 bucket.
2. Set the "index document" to index.html (the default for S3 website hosting).
3. Set the "error document" to 404.html (Antora generates this file for you).
4. Either configure the permissions of the S3 bucket to explicitly allow public access to read all objects in the bucket; or when you upload files to the bucket, explicitly upload them with a canned ACL setting that allows public read access (as the scripts covered later in this article will).

You need to make the files in your S3 bucket publicly-accessible (point #4 above) so that CloudFront can access them. While there technically is a way to configure S3 and CloudFront so that the files are not publicly-accessible in S3 but CloudFront can still access them (via an Origin Access Identity), it's kind of a pain. Since these files are ultimately meant to be served to the public through CloudFront anyway, it's simpler just to make them publicly-accessible in S3.

CloudFront Settings

There's also nothing special you need to do for Antora-generated sites with CloudFront — any of the dozens of online guides for S3 + CloudFront hosting will work to set it up. Just make sure that when you set the origin for your CloudFront distribution, you use the "website endpoint" of your S3 bucket, and not the standard endpoint.

For example, if your S3 bucket is named "example-bucket" and it's located in the us-west-2 region, don't use example-bucket.s3.us-west-2.amazonaws.com as your CloudFront origin — instead do use example-bucket.s3-website-us-west-2.amazonaws.com. Using the website endpoint will ensure that CloudFront serves the Antora-generated 404.html page for pages that don't exist, and that it also serves a 301 redirect for pages for which you've configured S3 to redirect (as the scripts covered later in this article will).

Upload Script

Once you've set up your Antora playbook, S3 bucket, and CloudFront distribution, you're ready to deploy your site. If you've set up your antora-playbook.yml as above, you can build your documentation, upload it to S3, and clear the CloudFront caches of the old version of your docs with the following simple script:

#!/bin/sh -e build_dir=build/site cf_distro=E1234567890ABC s3_bucket=example-bucket antora generate antora-playbook.yml aws s3 sync $build_dir s3://$s3_bucket --acl public-read --delete aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/*'

The first line generates your documentation to the build/site directory. The second line replaces the existing content of example-bucket with the content of the build/site directory (granting public read-access to each individual file uploaded). The third line clears the CloudFront caches for all the content of your CloudFront distribution.

If you documentation is part of a larger site (eg hosted as https://www.example.com/docs/ instead of being hosted as its own site (eg https://docs.example.com/), add the sub-directory under which your documentation is hosted (eg /docs) to the last two lines of the above script; for example, like the following:

aws s3 sync $build_dir s3://$s3_bucket/docs --acl public-read --delete aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/docs/*'

Redirects Script

The redirect pages that Antora will generate when you set the Antora urls.redirect_facility setting to static will work fine for your website users as is. But search engines will like it better if you serve real HTTP redirect responses (with the redirect information embedded in HTTP header fields) instead of just HTML pages that indicate that the client browser should redirect to a different location once parsed. You can get S3 + CloudFront to serve 301 Moved Permanently redirects in place of all the redirect pages Antora generates by uploading them separately to S3 with a special x-amz-website-redirect-location header.

To do so, insert the following block into your upload script between the aws s3 sync and aws cloudfront create-invalidation commands:

#!/bin/sh build_dir=build/site cf_distro=E1234567890ABC s3_bucket=docs.example.com antora generate antora-playbook.yml aws s3 sync $build_dir s3://$s3_bucket --acl public-read --delete grep -lR 'http-equiv="refresh"' $build_dir | while read file; do redirect_url=$(awk -F'"' '/rel="canonical"/ { print $4 }' $file) aws s3 cp $file s3://$s3_bucket/${file##$build_dir/} \ --website-redirect $redirect_url --acl public-read done aws cloudfront create-invalidation --distribution-id $cf_distro --paths '/*'

The above script block will search the Antora build dir for all redirect pages (with the grep command), and loop over each (with the while command, reading the local filepath to each into the file variable). It will pull out the canonical URL of the page to redirect to from the redirect page (via the awk command, into the redirect_url variable), and re-upload the file using the --website-redirect flag of the aws s3 cp command to indicate that S3 should serve a 301 redirect to the specified URL instead of the file content itself (when accessed through the S3 website endpoint).

As a concrete example of this redirect capability, say you had a page named how-it-works.adoc in the ROOT module of your example-ui-guide component. If you added metadata to that how-it-works.adoc page to add a redirect to it from the non-existant inner-workings.adoc page (eg via a page-aliases header attribute value of inner-workings.adoc), Antora would generate the following redirect page for you at build/site/example-user-guide/inner-workings/index.html:

<!DOCTYPE html> <meta charset="utf-8"> <link rel="canonical" href="https://docs.example.com/example-user-guide/how-it-works/"> <script>location="../how-it-works/"</script> <meta http-equiv="refresh" content="0; url=../how-it-works/"> <meta name="robots" content="noindex"> <title>Redirect Notice</title> <h1>Redirect Notice</h1> <p>The page you requested has been relocated to <a href="../how-it-works/">https://docs.example.com/example-user-guide/how-it-works/</a>.</p>

The above script would re-upload this file to S3 like so (with all variables expanded, and some additional line-wrapping for legibility):

aws s3 cp build/site/example-user-guide/inner-workings/index.html \ s3://example-bucket/example-user-guide/inner-workings/index.html \ --website-redirect https://docs.example.com/example-user-guide/how-it-works/ --acl public-read

If a user (or search engine) then navigates to https://docs.example.com/example-user-guide/inner-workings/, S3 + CloudFront will send this response back:

HTTP/2 301 location: https://docs.example.com/example-user-guide/how-it-works/

Antora Quick Start Tutorial

Antora is a sophisticated documentation "static site generator" (SSG) for AsciiDoc/Asciidoctor. It's built for big projects with multiple sets of documentation, so just getting started can be a little intimidating. Here's a quick guide for how to get up and running with Antora, using Docker and some Makefiles on your local dev machine.

At minimum, you really should use at least three separate git repos with Antora:

1. Your documentation content (AsciiDoc source files).
2. Your customized UI theme/style (JavaScript, CSS, and Handlebars templates).
3. Your Antora build and deployment configuration (antora-playbook.yml, and whatever additional scripts/config files you use to deploy to production).

In practice, you may in fact have several different repos for #1, like the docs directory from several different software products, or simply different repos for different pieces of documentation (like an Install Guide, Product Manual, API Reference, etc). And #3 might be part of other repos you use for devops configuration/deployment/infrastructure/etc.

But you're really going to want to have a new dedicated repo for #2 — this is where you customize your page header and footer content with links to your own websites, with your own logos, colors, and general look-and-feel.

So this guide is just going to focus on #2, with development-focused Antora config (#3) and initial documentation content (#1) snuck into the same repo. Before you begin, make sure you have Git, Make, and Docker installed on your local dev machine.

1. UI Set Up
2. UI Customization
3. Documentation Set Up
4. Documentation Customization
5. Development Workflow

UI Set Up

Clone Antora Default UI

To start off, clone the Antora Default UI repo. We're going to save all our UI customizations, our initial doc content, and our dev Antora config in this new repo, which we'll call docs-ui. Run these commands in a terminal:

$ git clone https://gitlab.com/antora/antora-ui-default.git docs-ui $ cd docs-ui $ git remote rename origin antora-ui-default $ git branch --move antora

This will keep antora-ui-default as a remote source for your git repository, so you can easily diff and pull in core fixes to the UI from the Antora Default UI repo. The local copy of the Antora Default UI we'll keep in a branch called antora.

But you'll want to set the main remote source of your customized repo (called origin by convention) to a new remote source. For example, if you have a github account named my-account, create a new repo named docs-ui in it, and push the local content of this new repo to it. We'll call the local branch of the main source main:

$ git checkout -b main $ git remote add origin https://github.com/my-account/docs-ui.git $ git push -u origin main

Dockerized UI Build

Now we're ready to start making changes to the repo. To make it easy to preview and build those changes, we'll add three files to the root of the repo: ui.dockerfile, docker-compose.yml, and Makefile. Create these files:

# ui.dockerfile FROM node:12-buster WORKDIR /srv/docs-ui

# docker-compose.yml version: '3' services: ui: build: context: . dockerfile: ui.dockerfile ports: - 8052:5252 volumes: - .:/srv/docs-ui

# Makefile # help: @ Lists available make tasks help: @egrep -oh '[0-9a-zA-Z_\.\-]+:.*?@ .*' $(MAKEFILE_LIST) | \ awk 'BEGIN {FS = ":.*?@ "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' | sort # node_modules: @ Runs initial ui npm install node_modules: docker-compose run ui npm install # ui.build: @ Builds ui production output (to build/ui-bundle.zip) ui.build: node_modules docker-compose run -u $$(id -u) ui node_modules/.bin/gulp bundle # ui.lint: @ Runs ui linting ui.lint: node_modules docker-compose run -u $$(id -u) ui node_modules/.bin/gulp lint # ui.run: @ Runs ui server in preview mode (on port 8052) ui.run: node_modules docker-compose run -u $$(id -u) --service-ports ui node_modules/.bin/gulp preview # ui.shell: @ Opens bash shell in ui container ui.shell: CMD ?= /bin/bash ui.shell: docker-compose run -u $$(id -u) ui $(CMD)

Note that Makefiles require you to use tabs for indentation, instead of spaces, so make sure your editor has preserved the indentation in the above Makefile with tabs.

Run UI Preview

With those three files in place, run this command from the repo root:

make ui.run

This will build the Docker image defined by the ui.dockerfile file, launch it as a container, and run the Antora UI preview server in it, exposed on port 8052. Open up a browser, and navigate to:

http://localhost:8052/

You should see a test page for the Antora UI (titled "Hardware and Software Requirements") with a bunch of sample content.

UI Customization

Customize Header Content

Now open up the src/partials/header-content.hbs file from the repo. This is the template for the page header content. Change the line with the "Home" link to this:

<a class="navbar-item" href="https://example.com/">Example Co Home</a>

Save the file, go back to your browser, and refresh the page. The "Home" link in the page header should now read "Example Co Home" (and link to https://example.com/).

Customize Header Color

Next open up the src/css/vars.css file. This file defines (via CSS variables, which begin with the -- prefix) the basic colors and sizes of the UI. Change the --navbar-background variable to this:

--navbar-background: #39f;

Save the file, go back to your browser, and refresh the page. The background of the page header should now be a medium blue.

Build UI Bundle

Kill the make ui.run task (by pressing control-C in the terminal it's running in), and run this command in it's place:

make ui.build

This will build an Antora UI bundle as a zip file called ui-bundle.zip in the repo's build directory. We'll use this bundle in the next step, when we set up our basic Antora documentation build.

Documentation Set Up

Now we're ready to actually build some documentation. The Antora Default UI repo includes an Antora module in its docs directory, so we'll start by building it.

Create Antora Playbook

The first thing to do is create an antora-playbook.yml file. This file defines the configuration for the Antora build process. Add the following antora-playbook.yml to the root of the repo:

# antora-playbook.yml site: robots: allow start_page: antora-ui-default::index.adoc title: Example Documentation url: https://docs.example.com content: sources: - url: ./ branches: HEAD start_path: docs runtime: cache_dir: ./build/cache ui: bundle: url: ./build/ui-bundle.zip urls: html_extension_style: indexify

This will configure Antora to use "author mode", where it pulls its documentation content from local files instead of remote git repos. Your production build scripts will probably configure Antora to pull documentation content from specific release branches of various product or documentation repos, but while you're developing the documentation, you'll just want to build the latest docs from your working copy.

The Antora documentation covers all playbook settings thoroughly in its Antora Playbook section, but let's briefly touch on each setting of this antora-playbook.yml:

site.robots: Setting this to allow directs Antora to generate a robots.txt file which allows bots to spider your site; setting it to disallow generates a robots.txt that disallows bots. While you can omit this setting in your dev build (and not generate a robots.txt file), it's nice to see what Antora will generate.

site.start_page: This is the page ID of the page to redirect to when a visitor navigates to the root of your site (eg https://docs.example.com/). A value of antora-ui-default::index.adoc sets it to redirect to the index.adoc page of the ROOT module in the antora-ui-default component (https://docs.example.com/antora-ui-default/ under this configuration). This source file for this page is located at docs/modules/ROOT/pages/index.adoc in our repo.

site.title: This is the site-wide title that will be displayed in the header of every page, as well as the browser titlebar.

site.url: This is the base URL of the site, when deployed, minus the trailing slash. For deployments where the documentation site has its own domain name (eg docs.example.com), the value of this setting should just be the URL scheme plus domain name (eg https://docs.example.com); for deployments where the documentation lives as a sub-directory of a larger site, the value should also include the sub-directory under which the documentation will live (eg https://www.example.com/docs). While you can safely omit this setting for dev builds, it's nice to include just so you can see how this URL will be used in production.

content.sources: This is a list of sources (each separate item in a YAML list is denoted by a - sign); initially, we'll just have one source: the docs directory that came with the Antora Default UI repo.

content.sources[0].url: With your production configuration, you'd usually specify a full URL to a git repo with this setting; but with our dev config, we'll just use this local repo, indicated by ./. Note that this setting has to be the path to the root of a git repo — it can't specify a sub-directory of the repo itself (like say ./docs).

content.sources[0].branches: This specifies which branch of the repo to use. With your production config, you'd probably want to specify specific branches that correspond to releases of your product (eg [1.0, 1.1, 1.2]); but for development, we just want whatever your current working branch is (HEAD in git parlance).

content.sources[0].start_path: This specifies the path to the content source root of each Antora component in the repo you want to include. We just have one to start with, located in the docs directory. Each component must have an antora.yml file in it's source root; this file defines its component name, display title, version string, and navigation structure (and it can also include AsciiDoc setting customizations that apply to just the particular component).

runtime.cache_dir: This specifies Antora's internal cache of files. The default location (~/.cache/antora) would put it inside our Docker container, which is fine except that it means that Antora has to rebuild the cache again every time we run one of our Makefile commands. So for a moderately improved experience, we'll move it to the build/cache directory inside the repo itself (the build directory is already conveniently listed in the .gitignore file we cloned from the Antora Default UI).

ui.bundle.url: This specifies the location where Antora should fetch your customized UI files from. We want to use the UI built by this very repo, so we'll specify the path to to the UI bundle built by our make ui.build command, ./build/ui-bundle.zip. We prefix the path with ./ to indicate to Antora that this is a local file path — usually this would be the full URL to the location where your build system has stored the latest build of your customized UI; the build system for the Antora project saves the latest stable build of the Antora Default UI to https://gitlab.com/antora/antora-ui-default/-/jobs/artifacts/master/raw/build/ui-bundle.zip?job=bundle-stable.

urls.html_extension_style: Setting this to indexify directs Antora to generate each page as an index.html file within a directory named for the page — eg https://docs.example.com/antora-ui-default/index.html for our start page — and to drop index.html from the page URL when linking to it — eg https://docs.example.com/antora-ui-default/. This is the style you'd usually use when hosting static content with NGINX, Apache, and many static hosting services. Antora allows for several different options, however, and while you don't need to set this in your dev config, it's nice to just so you can see what your URLs will look like in production.

Dockerized Docs Build

To make it easy to run the build, we'll add another Dockerfile, and add to our existing docker-compose configuration and Makefile. Add this antora.dockerfile (substituting 2.3.4 in the file for whatever Antora's latest stable version number is):

# antora.dockerfile FROM antora/antora:2.3.4 RUN yarn global add http-server onchange WORKDIR /srv/docs

Then add an antora service to our existing docker-compose.yml file:

# docker-compose.yml version: '3' services: antora: build: context: . dockerfile: antora.dockerfile environment: CI: 'true' ports: - 8051:8080 volumes: - .:/srv/docs ui: build: context: . dockerfile: ui.dockerfile ports: - 8052:5252 volumes: - .:/srv/docs-ui

Note that the CI: 'true' environment variable will suppress the "Edit this Page" link that otherwise would be displayed in the top right of each page (making the output generated by Antora in our Docker containers more similar to what we'd see in production).

Finally, add some antora.* tasks to our Makefile:

# Makefile # help: @ Lists available make tasks help: @egrep -oh '[0-9a-zA-Z_\.\-]+:.*?@ .*' $(MAKEFILE_LIST) | \ awk 'BEGIN {FS = ":.*?@ "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' | sort # antora.build: @ Builds documentation production output (to build/site) antora.build: docker-compose run -u $$(id -u) antora antora generate --clean antora-playbook.yml # antora.run: @ Serves documentation output (on port 8051) antora.run: docker-compose run --service-ports antora http-server build/site -c-1 # antora.watch: @ Watches for documentation changes and rebuilds (to build/site) antora.watch: docker-compose run -u $$(id -u) -T antora onchange \ -i antora-playbook.yml 'docs/**' \ -- antora generate antora-playbook.yml # antora.shell: @ Opens bash shell in antora container antora.shell: CMD ?= /bin/sh antora.shell: docker-compose run -u $$(id -u) antora $(CMD) # node_modules: @ Runs initial ui npm install node_modules: docker-compose run ui npm install # ui.build: @ Builds ui production output (to build/ui-bundle.zip) ui.build: node_modules docker-compose run -u $$(id -u) ui node_modules/.bin/gulp bundle # ui.lint: @ Runs ui linting ui.lint: node_modules docker-compose run -u $$(id -u) ui node_modules/.bin/gulp lint # ui.run: @ Runs ui server in preview mode (port 8052) ui.run: node_modules docker-compose run -u $$(id -u) --service-ports ui node_modules/.bin/gulp preview # ui.shell: @ Opens bash shell in ui container ui.shell: CMD ?= /bin/bash ui.shell: docker-compose run -u $$(id -u) ui $(CMD)

This antora.dockerfile takes the base Antora image produced by the Antora project, and adds the http-server and onchange Node.js modules to it. Our Makefile's antora.run task uses the http-server module to serve the content Antora generates, and our antora.watch task uses the onchange module to automatically rebuild that content whenever we change a documentation source file.

Build Documentation

With those file changes in place, run this command from the repo root:

make antora.build

This will build the Docker image defined by the antora.dockerfile file, launch it as a container, and run Antora using the antora-playbook.yml playbook we just wrote. Antora will generate the built documentation files to the build/site directory in our repo.

Run Documentation Preview

Now let's take a look at that documentation — run this command from the repo root:

make antora.run

This will serve the built files via the http-server Node.js module on port 8051. Open up a browser, and navigate to:

http://localhost:8051/

You should be redirected to http://localhost:8051/antora-ui-default/, and see the start page of the Antora Default UI documentation.

Rebuild on Documentation Changes

Finally, in another terminal (while the make antora.run command is still running in the first terminal), run this command:

make antora.watch

This will use the onchange Node.js module to watch for changes to the documentation source, and automatically trigger Antora to rebuild whenever you make a change. We'll rely on this functionality with the next step.

Documentation Customization

Create New Component

Now, finally, we're ready to start writing our own documentation! We'll start with a "User Guide" component, initially consisting of a "root" module with one page. Create a content directory, and a user-guide directory within it — this is where our "User Guide" component will live.

Within the user-guide directory, create a new antora.yml file. This file will contain the basic metadata about our "User Guide" component. Create it with this content:

# content/user-guide/antora.yml name: example-user-guide title: Example User Guide version: master

Note that master is a special keyword in Antora that means a component with no version. Usually Antora includes the version number of a component in all the URLs it generates for that component, but it will omit the version number if it is master.

Create First Page

Also within the user-guide directory, create a modules directory; within the modules directory, create an ROOT directory; and within the ROOT directory, create a pages directory. Within the ROOT directory, create our first page, called index.adoc:

# content/user-guide/modules/ROOT/pages/index.adoc = User Guide == Welcome Welcome to our product! This is the user guide.

Add Watch Path

Update our make antora.watch command in our Makefile to the following, adding components/** as a path to watch for changes:

# Makefile antora.watch: docker-compose run -u $$(id -u) -T antora onchange \ -i antora-playbook.yml 'components/**' 'docs/**' \ -- antora generate antora-playbook.yml

If you still have the make antora.watch command running in a terminal, kill it (by pressing control-C in the terminal it's running in), and re-run make antora.watch again.

Register Component

Now we'll register the component in the antora-playbook.yml at the root of our repo, so that we can have Antora build it. We could add it as a second item to our sources list, but since we already have the current repo listed as a source, we can just change the existing single start_path setting to be a multiple start_paths setting (note the "s" on the end of the setting), and direct Antora to include any sub-directory of the content directory in our repo as a component:

# antora-playbook.yml site: robots: allow start_page: antora-ui-default::index.adoc title: Example Documentation url: https://docs.example.com content: sources: - url: ./ branches: HEAD start_path: docs start_paths: - docs - content/* runtime: cache_dir: ./build/cache ui: bundle: url: ./build/ui-bundle.zip urls: html_extension_style: indexify

With our make antora.watch command running, as soon as you save this change to antora-playbook.yml, Antora will rebuild all its content. So make the change, and refresh the browser window you have opened at http://localhost:8051.

The page will look exactly the same as it did before — but click the "Antora Default UI" label in the bottom-left corner of the page. The bottom of the left navigation should expand to show two items: "Antora Default UI" and "Example User Guide". Click the "master" label directly below "Example User Guide", and you will navigate to your new "Example User Guide" index page at http://localhost:8051/example-user-guide/.

The component is listed as "Example User Guide", because that is the title we gave it in the content/user-guide/antora.yml file. And its URL path is /example-user-guide/ because we set its name to example-user-guide in that same file.

The title of the page is "User Guide", as displayed in the browser titlebar and the main title of the page's body, because that's the title we gave the page in its AsciiDoc source at content/user-guide/modules/ROOT/pages/index.adoc. And because we added a "Welcome" section to that page, we see "Welcome" as a section title in the page's body, as well as its right navigation.

You'll also want to change your antora-playbook.yml to make the new component the start page for the site (so that whenever someone navigates to https://docs.example.com/, they'll be redirected to https://docs.example.com/example-user-guide/ instead of https://docs.example.com/antora-ui-default/). Just replace antora-ui-default in the start_page setting of your antora-playbook.yml with example-user-guide:

# antora-playbook.yml start_page: antora-ui-default::index.adoc start_page: example-user-guide::index.adoc

Create Navigation

We don't have any left navigation for our new component, however, so let's fix that. Create a navigation file at content/user-guide/modules/ROOT/nav.adoc with this content:

# content/user-guide/modules/ROOT/nav.adoc * xref:index.adoc#_welcome[Welcome!]

Then configure Antora to use this navigation file by updating the content/user-guide/modules/antora.yml file like so:

# content/user-guide/antora.yml name: example-user-guide title: Example User Guide version: master nav: - modules/ROOT/nav.adoc

Go back to your browser and refresh the page, and you will see that Antora added a navigation list for the user-guide component to the left navigation, with one item beneath the root. This item links to the "Welcome" section of the "root" module's index page, displaying the link text as "Welcome!".

Create Another Page

Now create a second page for our component at content/user-guide/modules/ROOT/pages/how-it-works.adoc with this content:

# content/user-guide/modules/ROOT/pages/how-it-works.adoc = This Is How It Works :navtitle: But How Does It Work? Well, to be honest, we're not exactly sure how any of this works.

Update the navigation file at content/user-guide/modules/ROOT/nav.adoc to add an item for our new page:

# content/user-guide/modules/ROOT/nav.adoc * xref:index.adoc#_welcome[Welcome!] * xref:how-it-works.adoc[]

Go back to your browser and refresh the page again, and you will see that Antora added another item to the left navigation. The item label is "But How Does It Work?", matching the navtitle attribute set under the title of the how-it-works.adoc page. The URL it links to is http://localhost:8051/example-user-guide/how-it-works/ — the first path segment comes from the component name defined in the antora.yml file of the component, and the last path segment comes from the file name of the page itself. If the file was part of any module other than the "root" module for the component, it would have another path segment between the component name and page name for the module name (taken from the directory name of the module).

When you click the link, you'll see that the page title is "This Is How It Works", as displayed in the browser titlebar and the main title of the page's body, matching the page title from the how-it-works.adoc file.

Development Workflow

Work on the UI

With your initial UI and documentation content now set up, whenever you want to make changes to the look-and-feel of your documentation, you would follow these steps:

1. Fire up the Antora UI preview server with make ui.run
2. View the Antora UI preview in your web browser at http://localhost:8052/
3. Iteratively make changes to the CSS, Handlebars, and other UI files in the src directory of your project

Build the UI

Once your changes look good with the Antora UI preview content, follow these steps to incorporate them into the local version of your documentation:

1. Build the UI bundle with make ui.build
2. Pull the new UI bundle into your local doc build with make antora.build
3. Fire up a web server for your local docs with make antora.run
4. View your UI changes with your local copy of the docs at http://localhost:8051/

Write Documentation

And whenever you get the hankering to write some documentation, follow these steps:

1. Fire up a web server for your local docs with make antora.run
2. View your local copy of the docs at http://localhost:8051/
3. Fire up the onchange watcher for the docs in a separate terminal with make antora.watch (to automatically re-build the docs whenever you make a change)
4. Iteratively make changes to AsciiDoc files in the content directory of your project

How To Test OpenRC Services with Docker-Compose

Similar to how I abused Docker conceptually to test systemd services with docker-compose, I spent some time recently trying to do the same thing with OpenRC for Alpine Linux.

It basically requires the same steps as systemd. With the base 3.12 Alpine image, it's a matter of:

1. Install OpenRC
2. Optionally map /sys/fs/cgroup
3. Start up with /sbin/init
4. Run tests via docker exec

1. Install OpenRC

The base Alpine images don't include OpenRC, so you have to install it with apk. I do this in my Dockerfile:

FROM alpine:3.12 RUN apk add openrc CMD ["/sbin/init"]

2. Optionally map /sys/fs/cgroup

Unlike with systemd, I didn't have to set up any tmpfs mounts to get OpenRC services running. I also didn't have to map the /sys/fs/cgroup directory -- but if I didn't, I would get a bunch of cgroup-related error messages when starting and stopping services (although the services themselves still seemed to work fine). So I just went ahead and mapped the dir in my docker-compose.yml to avoid those error messages:

version: '3' services: my_test_container: build: . image: my_test_image volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro

3. Start up with /sbin/init

With the Alpine openrc package, the traditional /sbin/init startup command works to start OpenRC. I added CMD ["/sbin/init"] to my Dockerfile to start up with it, but you could instead add command: /sbin/init to the service in your docker-compose.yml file.

4. Run tests via docker exec

The above docker-compose.yml and Dockerfile will allow you to start up OpenRC in my_test_container with one command:

docker-compose up -d my_test_container

With OpenRC up and running, you can use a second command to execute a shell on the very same container to test it out:

docker-compose exec my_test_container /bin/sh

Or use exec to run other commands to test the services managed by OpenRC:

docker-compose exec my_test_container rc-status --servicelist

Cleaning up

The clean up steps with OpenRC are also basically the same as with systemd:

1. Stop the running container: docker-compose stop my_test_container
2. Remove the saved container state: docker-compose rm my_test_container
3. Remove the built image: docker image rm my_test_image

Testing Systemd Services on Arch, Fedora, and Friends

Following up on a previous post about how to test systemd services with docker-compose on Ubuntu, I spent some time recently trying to do the same thing with a few other Linux distributions. I was able to get the same tricks to work on these other distributions:

• Amazon Linux
• Arch
• CentOS
• Debian
• Fedora
• openSUSE
• RHEL

A few of those distros required an additional tweak, however.

One more tmpfs directory for Arch and Fedora

For Arch and Fedora, I had to do one more thing: add /tmp as a tmpfs mount.

So the docker-compose.yml file for those distros should look like this:

version: '3' services: my_test_container: build: . image: my_test_image tmpfs: - /run - /run/lock - /tmp volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro

Or when running as a regular docker command, start the container like this:

docker run \ --tmpfs /run --tmpfs /run/lock --tmpfs /tmp \ --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \ --detach --rm \ --name my_test_container my_test_image

Different init script location for openSUSE

For openSUSE, the systemd init script is located at /usr/lib/systemd/systemd instead of just /lib/systemd/systemd. So the Dockerfile I used for it looks like this:

FROM opensuse/leap:15 RUN zypper install -y systemd CMD ["/usr/lib/systemd/systemd"]

Elixir Ed25519 Signatures With Enacl

The most-actively supported library for using ed25519 with Elixir currently looks to be enacl. It provides straightforward, idiomatic Erlang bindings for libsodium.

Installing

Installing enacl for a Mix project requires first installing your operating system's libsodium-dev package on your dev & build machines (as well as the regular libsodium package anywhere else you run your project binaries). Then in the mix.exs file of your project, add {:enacl, "~> 1.0.0"} to the deps section of that file; and then run mix deps.get to download the enacl package from Hex.

Keys

In the parlance of libsodium, the "secret key" is the full keypair, the "public key" is the public part of the keypair (the public curve point), and the "seed" is the private part of the keypair (the 256-bit secret). The seed is represented in enacl as a 32-byte binary string, as is the public key; and the secret key is the 64-byte binary concatenation of the seed plus the public key.

You can generate a brand new ed25519 keypair with enacl via the sign_keypair/0 function. After generating, usually you'd want to save the keypair somewhere as a base64- or hex-encoded string:

iex> keypair = :enacl.sign_keypair() %{ public: <<215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, 14, 225, 114, 243, 218, 166, 35, 37, 175, 2, 26, 104, 247, 7, 81, 26>>, secret: <<157, 97, 177, 157, 239, 253, 90, 96, 186, 132, 74, 244, 146, 236, 44, 196, 68, 73, 197, 105, 123, 50, 105, 25, 112, 59, 172, 3, 28, 174, 127, 96, 215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, ...>> } iex> <<seed::binary-size(32), public_key::binary>> = keypair.secret <<157, 97, 177, 157, 239, 253, 90, 96, 186, 132, 74, 244, 146, 236, 44, 196, 68, 73, 197, 105, 123, 50, 105, 25, 112, 59, 172, 3, 28, 174, 127, 96, 215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, 14, 225, ...>> iex> public_key == keypair.public true iex> seed <> public_key == keypair.secret true iex> public_key_base64 = public_key |> Base.encode64() "11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo=" iex> public_key_hex = public_key |> Base.encode16(case: :lower) "d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a" iex> private_key_base64 = seed |> Base.encode64() "nWGxne/9WmC6hEr0kuwsxERJxWl7MmkZcDusAxyuf2A=" iex> private_key_hex = seed |> Base.encode16(case: :lower) "9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60"

You can also reconstitute a keypair from just the private part (the "seed") with the enacle sign_seed_keypair/1 function:

iex> reloaded_keypair = ( ...> "9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60" ...> |> Base.decode16!(case: :lower) ...> |> :enacl.sign_seed_keypair() ...>) %{ public: <<215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, 14, 225, 114, 243, 218, 166, 35, 37, 175, 2, 26, 104, 247, 7, 81, 26>>, secret: <<157, 97, 177, 157, 239, 253, 90, 96, 186, 132, 74, 244, 146, 236, 44, 196, 68, 73, 197, 105, 123, 50, 105, 25, 112, 59, 172, 3, 28, 174, 127, 96, 215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, ...>> } iex> reloaded_keypair == keypair true

Signing

Libsodium has a series of functions for signing large documents that won't fit into memory or otherwise have to be split into chunks — but for most cases, the simpler enacl sign/2 or sign_detached/2 functions are what you want to use.

The enacl sign/2 function produces a binary string that combines the original message with the message signature, which the sign_open/2 function can later unpack and verify. This is ideal for preventing misuse, since it makes it harder to just use the message without verifying the signature first.

The enacl sign_detached/2 function produces the message signature as a stand-alone 64-byte binary string — if you need to store or send the signature separately from the message itself, this is the function you'd use. And often when using detached signatures, you will also base64- or hex-encode the resulting signature:

iex> message = "test" "test" iex> signed_message = :enacl.sign(message, keypair.secret) <<143, 152, 176, 38, 66, 39, 246, 31, 9, 107, 120, 221, 227, 176, 240, 13, 25, 1, 236, 254, 16, 80, 94, 65, 71, 57, 6, 144, 122, 82, 53, 107, 233, 83, 26, 215, 109, 77, 1, 219, 7, 67, 77, 72, 147, 94, 245, 81, 222, 80, ...>> iex> signature = :enacl.sign_detached(message, keypair.secret) <<143, 152, 176, 38, 66, 39, 246, 31, 9, 107, 120, 221, 227, 176, 240, 13, 25, 1, 236, 254, 16, 80, 94, 65, 71, 57, 6, 144, 122, 82, 53, 107, 233, 83, 26, 215, 109, 77, 1, 219, 7, 67, 77, 72, 147, 94, 245, 81, 222, 80, ...>> iex> signature <> message == signed_message true iex> signature |> Base.encode64() "j5iwJkIn9h8Ja3jd47DwDRkB7P4QUF5BRzkGkHpSNWvpUxrXbU0B2wdDTUiTXvVR3lBULDNm0/t1DY8GBoxfCA==" iex> signature |> Base.encode16(case: :lower) "8f98b0264227f61f096b78dde3b0f00d1901ecfe10505e41473906907a52356be9531ad76d4d01db07434d48935ef551de50542c3366d3fb750d8f06068c5f08"

Verifying

To verify a signed message (the message combined with the signature), and then access the message itself, you'd use the enacl sign_open/2 function:

iex> unpacked_message = :enacl.sign_open(signed_message, public_key) {:ok, "test"}

If you try to verify the signed message with a different public key (or if the message is otherwise improperly signed or not signed at all), you'll get an error result from the sign_open/2 function:

iex> wrong_public_key = ( ...> "3d4017c3e843895a92b70aa74d1b7ebc9c982ccf2ec4968cc0cd55f12af4660c" ...> |> Base.decode16!(case: :lower) ...> ) <<61, 64, 23, 195, 232, 67, 137, 90, 146, 183, 10, 167, 77, 27, 126, 188, 156, 152, 44, 207, 46, 196, 150, 140, 192, 205, 85, 241, 42, 244, 102, 12>> iex> error_result = :enacl.sign_open(signed_message, wrong_public_key) {:error, :failed_verification}

To verify a message with a detached signature, you need the original message itself (in the same binary form with which it was signed), and the signature (in binary form as well). You pass them both, plus the public key, to the sign_verify_detached/3 function; sign_verify_detached/3 returns true if the signature is legit, and false otherwise:

iex> :enacl.sign_verify_detached(signature, message, public_key) true iex> :enacl.sign_verify_detached(signature, "wrong message", public_key) false iex> :enacl.sign_verify_detached(signature, message, wrong_public_key) false

Full Example

To put it all together, if you have an ed25519 private key, like "nWGxne/9WmC6hEr0kuwsxERJxWl7MmkZcDusAxyuf2A=", and you want to sign a message ("test") that someone else already has in their possession, you'd do the following to produce a stand-alone signature that you can send them:

iex> secret_key = ( ...> "nWGxne/9WmC6hEr0kuwsxERJxWl7MmkZcDusAxyuf2A=" ...> |> Base.decode64!() ...> |> :enacl.sign_seed_keypair() ...> |> Map.get(:secret) ...> ) <<157, 97, 177, 157, 239, 253, 90, 96, 186, 132, 74, 244, 146, 236, 44, 196, 68, 73, 197, 105, 123, 50, 105, 25, 112, 59, 172, 3, 28, 174, 127, 96, 215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, 14, 225, ...>> iex> signature_base64 = ( ...> "test" ...> |> :enacl.sign_detached(secret_key) ...> |> Base.encode64() ...> ) "j5iwJkIn9h8Ja3jd47DwDRkB7P4QUF5BRzkGkHpSNWvpUxrXbU0B2wdDTUiTXvVR3lBULDNm0/t1DY8GBoxfCA=="

And if you're the one given an ed25519 public key ("11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo=") and signature ("j5iwJkIn9h8Ja3jd47DwDRkB7P4QUF5BRzkGkHpSNWvpUxrXbU0B2wdDTUiTXvVR3lBULDNm0/t1DY8GBoxfCA=="), with the original message ("test") in hand you can verify the signature like the following:

iex> public_key = ( ...> "11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo=" ...> |> Base.decode64!() ...> ) <<215, 90, 152, 1, 130, 177, 10, 183, 213, 75, 254, 211, 201, 100, 7, 58, 14, 225, 114, 243, 218, 166, 35, 37, 175, 2, 26, 104, 247, 7, 81, 26>> iex> signature_legitimate? = ( ...> "j5iwJkIn9h8Ja3jd47DwDRkB7P4QUF5BRzkGkHpSNWvpUxrXbU0B2wdDTUiTXvVR3lBULDNm0/t1DY8GBoxfCA==" ...> |> Base.decode64!() ...> |> :enacl.sign_verify_detached("test", public_key) ...> ) true

Postgrex Ecto Types

One thing I found confusing about Postgrex, the excellent PostgreSQL adapter for Elixir, was how to use PostgresSQL-specific data types (cidr, inet, interval, lexeme, range, etc) with Ecto. As far as I can tell, while Postgrex includes structs that can be converted to each type (like Postgrex.INET etc), you still have to write your own Ecto.Type implementation for each type you use in an Ecto schema.

For example, to implement an Ecto schema for a table like the following:

CREATE TABLE hits ( id BIGSERIAL NOT NULL PRIMARY KEY, url TEXT NOT NULL, ip INET NOT NULL, inserted_at TIMESTAMP NOT NULL );

You'd need minimally to create an Ecto type implementation like the following:

# lib/my_app/inet_type.ex defmodule MyApp.InetType do @moduledoc """ `Ecto.Type` implementation for postgres `INET` type. """ use Ecto.Type def type, do: :inet def cast(term), do: {:ok, term} def dump(term), do: {:ok, term} def load(term), do: {:ok, term} end

So that you could then define the Ecto schema with your custom Ecto type:

# lib/my_app/hits/hit.ex defmodule MyApp.Hits.Hit do @moduledoc """ The Hit schema. """ use Ecto.Schema schema "hits" do field :url, :string field :ip, MyApp.InetType timestamps updated_at: false end def changeset(hit, attrs) do hit |> cast(attrs, [:url, :ip]) |> validate_required([:url, :ip]) end end

Note that in your migration code, you use the native database type name (eg inet), not your custom type name:

# priv/repo/migrations/202001010000_create_hits.exs defmodule MyApp.Repo.Migrations.CreateHits do use Ecto.Migration def change do create table(:hits) do add :url, :text, null: false add :ip, :inet, null: false timestamps updated_at: false end end end

A Fancier Type

However, the above basic InetType implementation limits this example Hit schema to working only with Postgrex.INET structs for its ip field — so, while creating a Hit record with the IP address specified via a Postgrex.INET struct works nicely:

iex> ( ...> %MyApp.Hits.Hit{} ...> |> MyApp.Hits.Hit.changeset(%{url: "/", ip: %Postgrex.INET{address: {127, 0, 0, 1}}}) ...> |> MyApp.Repo.insert!() ...> |> Map.get(:ip) ...> ) %Postgrex.INET{address: {127, 0, 0, 1}}

Creating one with the IP address specified as a string (or even a plain tuple like {127, 0, 0, 1}) won't work:

iex> ( iex> %MyApp.Hits.Hit{} ...> |> MyApp.Hits.Hit.changeset(%{url: "/", ip: "127.0.0.1"}}) ...> |> MyApp.Repo.insert!() ...> |> Map.get(:ip) ...> ) ** (Ecto.InvalidChangesetError) could not perform insert because changeset is invalid.

This can be solved by implementing a fancier version of the cast function in the MyApp.InetType module, enabling Ecto to cast strings (and tuples) to the Postgrex.INET type. Here's a version of MyApp.InetType that does that, as well as allows the Postgrex.INET struct to be serialized as a string (including when serialized to JSON with the Jason library, or when rendered as part of a Phoenix HTML template):

# lib/my_app/inet_type.ex defmodule MyApp.InetType do @moduledoc """ `Ecto.Type` implementation for postgres `INET` type. """ use Bitwise use Ecto.Type alias Postgrex.INET def type, do: :inet def cast(nil), do: {:ok, nil} def cast(""), do: {:ok, nil} def cast(%INET{address: nil}), do: {:ok, nil} def cast(%INET{} = term), do: {:ok, term} def cast(term) when is_tuple(term), do: {:ok, %INET{address: term}} def cast(term) when is_binary(term) do [addr | mask] = String.split(term, "/", parts: 2) with {:ok, address} <- parse_address(addr), {:ok, number} <- parse_netmask(mask), {:ok, netmask} <- validate_netmask(number, address) do {:ok, %INET{address: address, netmask: netmask}} else message -> {:error, [message: message]} end end def cast(_), do: :error def dump(term), do: {:ok, term} def load(term), do: {:ok, term} defp parse_address(addr) do case :inet.parse_strict_address(String.to_charlist(addr)) do {:ok, address} -> {:ok, address} _ -> "not a valid IP address" end end defp parse_netmask([]), do: {:ok, nil} defp parse_netmask([mask]) do case Integer.parse(mask) do {number, ""} -> {:ok, number} _ -> "not a CIDR netmask" end end defp validate_netmask(nil, _addr), do: {:ok, nil} defp validate_netmask(mask, _addr) when mask < 0 do "CIDR netmask cannot be negative" end defp validate_netmask(mask, addr) when mask > 32 and tuple_size(addr) == 4 do "CIDR netmask cannot be greater than 32" end defp validate_netmask(mask, _addr) when mask > 128 do "CIDR netmask cannot be greater than 128" end defp validate_netmask(mask, addr) do ipv4 = tuple_size(addr) == 4 max = if ipv4, do: 32, else: 128 subnet = if ipv4, do: 8, else: 16 bits = addr |> Tuple.to_list() |> Enum.reverse() |> Enum.with_index() |> Enum.reduce(0, fn {value, index}, acc -> acc + (value <<< (index * subnet)) end) bitmask = ((1 <<< max) - 1) ^^^ ((1 <<< (max - mask)) - 1) if (bits &&& bitmask) == bits do {:ok, mask} else "masked bits of IP address all must be 0s" end end end defimpl String.Chars, for: Postgrex.INET do def to_string(%{address: address, netmask: netmask}) do "#{address_to_string(address)}#{netmask_to_string(netmask)}" end defp address_to_string(nil), do: "" defp address_to_string(address), do: address |> :inet.ntoa() defp netmask_to_string(nil), do: "" defp netmask_to_string(netmask), do: "/#{netmask}" end defimpl Jason.Encoder, for: Postgrex.INET do def encode(term, opts), do: term |> to_string() |> Jason.Encode.string(opts) end defimpl Phoenix.HTML.Safe, for: Postgrex.INET do def to_iodata(term), do: term |> to_string() end

Alternative Canonical Representation

Note that an alternative way of implementing your Ecto type would be to make the dump and load functions round-trip the Postgrex.INET struct to and from some more convenient canonical representation (like a plain string). For example, a MyApp.InetType like the following would allow you to use plain strings to represent IP address values in your schemas (instead of Postgrex.INET structs). It would dump each such string to a Postgrex.INET struct when Ecto attempts to save the value to the database, and load the value from a Postgrex.INET struct into a string when Ecto attempts to load the value from the database:

# lib/my_app/inet_type.ex defmodule MyApp.InetType do @moduledoc """ `Ecto.Type` implementation for postgres `INET` type. """ use Ecto.Type alias Postgrex.INET def type, do: :inet def cast(nil), do: {:ok, ""} def cast(term) when is_tuple(term), do: {:ok, address_to_string(term)} def cast(term) when is_binary(term), do: {:ok, term} def cast(_), do: :error def dump(nil), do: {:ok, nil} def dump(""), do: {:ok, nil} def dump(term) when is_binary(term) do [addr | mask] = String.split(term, "/", parts: 2) with {:ok, address} <- parse_address(addr), {:ok, number} <- parse_netmask(mask), {:ok, netmask} <- validate_netmask(number, address) do {:ok, %INET{address: address, netmask: netmask}} else message -> {:error, [message: message]} end end def dump(_), do: :error def load(nil), do: {:ok, ""} def load(%INET{address: address, netmask: netmask}) do "#{address_to_string(address)}#{netmask_to_string(netmask)}" end def load(_), do: :error defp parse_address(addr) do case :inet.parse_strict_address(String.to_charlist(addr)) do {:ok, address} -> {:ok, address} _ -> "not a valid IP address" end end defp parse_netmask([]), do: {:ok, nil} defp parse_netmask([mask]) do case Integer.parse(mask) do {number, ""} -> {:ok, number} _ -> "not a CIDR netmask" end end defp validate_netmask(nil, _addr), do: {:ok, nil} defp validate_netmask(mask, _addr) when mask < 0 do "CIDR netmask cannot be negative" end defp validate_netmask(mask, addr) when mask > 32 and tuple_size(addr) == 4 do "CIDR netmask cannot be greater than 32" end defp validate_netmask(mask, _addr) when mask > 128 do "CIDR netmask cannot be greater than 128" end defp validate_netmask(mask, addr) do ipv4 = tuple_size(addr) == 4 max = if ipv4, do: 32, else: 128 subnet = if ipv4, do: 8, else: 16 bits = addr |> Tuple.to_list() |> Enum.reverse() |> Enum.with_index() |> Enum.reduce(0, fn {value, index}, acc -> acc + (value <<< (index * subnet)) end) bitmask = ((1 <<< max) - 1) ^^^ ((1 <<< (max - mask)) - 1) if (bits &&& bitmask) == bits do {:ok, mask} else "masked bits of IP address all must be 0s" end end defp address_to_string(nil), do: "" defp address_to_string(address), do: address |> :inet.ntoa() defp netmask_to_string(nil), do: "" defp netmask_to_string(netmask), do: "/#{netmask}" end

Elixir Event Queue

As I'm learning Elixir, I was trying to search for the idiomatic way for building a event queue in Elixir. After a few twists and turns, I found that it's easy, elegant, and pretty well documented — you just need to know what to look for.

There are a number of nifty "job queue" libraries for Elixir (like Honeydew or Oban), but they're directed more toward queueing jobs themselves, rather than enabling a single job to work on a queue of items. What I was looking for was this:

1. A singleton queue that would have events enqueued from multiple processes (in Elixir-world, this would take the form of an Agent).
2. A client app running multiple processes that would enqueue events (in my case, a Phoenix app).
3. A worker process that dequeues a batch of events and processes them (in Elixir-world, this would be a GenServer).

An Example

Following is an example of what I found to be the idiomatic Elixir way of implementing this, with 1) a generic agent that holds a queue as its state (QueueAgent), 2) the bits of a Phoenix app that listens for Phoenix telemetry events and enqueues some data from them onto this queue (RequestListener), and 3) a gen-server worker that dequeues those events and saves them to the DB (RequestSaver). These three components each are started by the Phoenix application's supervisor (4).

1. The Queue Agent

The QueueAgent module holds the state of the queue, as a Qex struct. Qex is a wrapper around the native Erlang/OTP :queue module, adding some Elixir syntactic sugar and implementing the Inpsect, Collectable, and Enumerable protocols.

The QueueAgent module can pretty much just proxy basic Qex calls through the core agent get, update, and get_and_update functions. Each of these functions accepts a function itself, to which current state of the agent (the Qex queue) is passed. The functions accepted by update and get_and_update also return the new state of the agent (the updated Qex queue).

# lib/my_app/queue_agent.ex defmodule MyApp.QueueAgent do @moduledoc """ Agent that holds a queue as its state. """ use Agent @doc """ Starts the agent with the specified options. """ @spec start_link(GenServer.options()) :: Agent.on_start() def start_link(opts \\ []) do Agent.start_link(&Qex.new/0, opts) end @doc """ Returns the length of the queue. """ @spec count(Agent.agent()) :: integer def count(agent) do Agent.get(agent, & &1) |> Enum.count() end @doc """ Enqueues the specified item to the end of the queue. """ @spec push(Agent.agent(), any) :: :ok def push(agent, value) do Agent.update(agent, &Qex.push(&1, value)) end @doc """ Dequeues the first item from the front of the queue, and returns it. If the queue is empty, returns the specified default value. """ @spec pop(Agent.agent(), any) :: any def pop(agent, default \\ nil) do case Agent.get_and_update(agent, &Qex.pop/1) do {:value, value} -> value _ -> default end end @doc """ Takes the specified number of items off the front of the queue, and returns them. If the queue has less than the specified number of items, empties the queue and returns all items. """ @spec split(Agent.agent(), integer) :: Qex.t() def split(agent, max) do Agent.get_and_update(agent, fn queue -> Qex.split(queue, Enum.min([Enum.count(queue), max])) end) end end

2. The Request Listener

The RequestListener module attaches a Telemetry listener (with the arbitrary name "my_app_web_request_listener") to handle one specific event (the "response sent" event from the Phoenix Logger, identified by [:phoenix, :endpoint, :stop]). The listener's handle_event function will be called whenever a response is sent (including error responses), and the response's Plug.Conn struct will be included under the :conn key of the event metadata.

In handling the event, the RequestListener simply enqueues a new map containing the details about the request that I want to save to a named QueueAgent queue. The name can be arbitrary — in this example it's MyApp.Request (a module name that doesn't happen to exist) — what's important is that a QueueAgent with that name has been started (it will be started by the application, later on in step 4), and that the RequestSaver (later on in step 3) will use the same name to dequeue events.

# lib/my_app_web/request_listener.ex defmodule MyAppWeb.RequestListener do @moduledoc """ Listens for request telemetry events, and queues them to be saved. """ require Logger @response_sent [:phoenix, :endpoint, :stop] @events [@response_sent] @doc """ Sets up event listener. """ def setup do :telemetry.attach_many("my_app_web_request_listener", @events, &handle_event/4, nil) end @doc """ Telemetry callback to handle specified event. """ def handle_event(@response_sent, measurement, metadata, _config) do handle_response_sent(measurement, metadata, MyApp.RequestQueue) end @doc """ Handles Phoenix response sent event. """ def handle_response_sent(measurement, metadata, queue_name) do conn = metadata.conn reason = conn.assigns[:reason] MyApp.QueueAgent.push(queue_name, %{ inserted_at: DateTime.utc_now(), ip: conn.remote_ip, request_id: Logger.metadata()[:request_id], controller: conn.private[:phoenix_controller], action: conn.private[:phoenix_action], status: conn.status, method: conn.method, path: conn.request_path, query: conn.query_string, error: if(reason, do: Exception.message(reason)), # nanoseconds duration: measurement.duration }) end end

3. The Request Saver

The RequestSaver module is run as a dedicated process, dequeueing batches of up to 100 events, and saving each batch. When done saving, it will "sleep" for a minute, then try to dequeue some more events. Everything but the do_work, save_next_batch, save_batch, and batch_changeset functions are boilerplate gen-server functionality for running a process periodically.

The do_work function uses the same MyApp.RequestQueue name as the RequestListener to identify the queue, ensuring that both modules use the same QueueAgent instance. The save_next_batch function dequeues up to 100 events and saves them via the save_batch function (and continues working until it has emptied the queue). The save_batch and batch_changeset functions create and commit an Ecto.Multi changeset using the app's MyApp.RequestEvent schema (not included in this example, but as you can imagine, it would include fields for the various properties that the RequestListener extracted from the event metadata).

The handle_info callback is the entry point for the gen-server's processing. It ignores the gen-server's state (it doesn't need to maintain any state itself) — it simply does some work, and then calls schedule_work to schedule itself to be called again in another minute.

# lib/my_app/request_saver.ex defmodule MyApp.RequestSaver do @moduledoc """ Saves queued events to the DB. """ use GenServer @doc """ Starts the server with the specified options. """ def start_link(_opts) do GenServer.start_link(__MODULE__, %{}) end @doc """ GenServer callback to start process. """ @impl true def init(state) do schedule_work() {:ok, state} end @doc """ GenServer callback to handle process messages. """ @impl true def handle_info(:work, state) do do_work() schedule_work() {:noreply, state} end @doc """ Does the next unit of work. """ def do_work do save_next_batch(MyApp.RequestQueue) end @doc """ Pops the next 100 events from the specified queue and saves them. """ def save_next_batch(queue_name) do batch = MyApp.QueueAgent.split(queue_name, 100) if Enum.count(batch) > 0 do save_batch(batch) save_next_batch(queue_name) end end @doc """ Saves the specified list of events in one big transaction. """ def save_batch(batch) do batch_changeset(batch) |> MyApp.Repo.transaction() end @doc """ Creates an Ecto.Multi from the specified list of events. """ def batch_changeset(batch) do batch |> Enum.reduce(Ecto.Multi.new(), fn event, multi -> changeset = MyApp.RequestEvent.changeset(event) Ecto.Multi.insert(multi, {:event, event.request_id}, changeset) end) end defp schedule_work do # in 1 minute Process.send_after(self(), :work, 60 * 1000) end end

4. The Application Supervisor

The above three components are all started in my Phoenix app via the standard Phoenix Application module. On start, it calls the RequestListener setup function, registering the RequestListener to receive Phoenix Telemetry events. Then the RequestSaver gen-server is started as a child process of the app (with no arguments, identified by its own module name); and the QueueAgent agent is also started as a child process — but with a name option, so that it can be identified via the MyApp.RequestQueue name. (Lines added to the boilerplate Phoneix Application module are highlighted in green.)

# lib/my_app/application.ex defmodule MyApp.Application do @moduledoc false use Application def start(_type, _args) do MyAppWeb.RequestListener.setup() children = [ MyApp.Repo, MyAppWeb.Endpoint, MyApp.RequestSaver, {MyApp.QueueAgent, name: MyApp.RequestQueue} ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end

However, you usually don't want periodic jobs popping up randomly while you run your unit tests; so I added a little extra logic to avoid starting up the RequestSaver in test mode:

# lib/my_app/application.ex defmodule MyApp.Application do @moduledoc false use Application def start(_type, _args) do MyAppWeb.RequestListener.setup() periodic_jobs = if Mix.env != :test do [MyApp.RequestSaver] else [] end children = [ MyApp.Repo, MyAppWeb.Endpoint, {MyApp.QueueAgent, name: MyApp.RequestQueue} ] ++ periodic_jobs opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end

The overall processing flow of this queueing system, then, works like this:

1. A request is handled by Phoenix, which raises a "response sent" telemetry event.
2. The RequestListener handle_event function is called by the Phoenix process.
3. The RequestListener calls the QueueAgent push function to queue the event (which the QueueAgent does within its own internal process).
4. Once a minute, the RequestSaver process runs the handle_info function, which tries to dequeue the next batch of events via the QueueAgent split function (again with the QueueAgent managing the state update in its own internal process).
5. The RequestSaver, continuing on in its process, saves any dequeued events to the DB.