Thursday, January 14, 2021

Using an Ecto Readonly Replica Repo

Elixir Ecto has excellent documentation for how to use read-only replica databases, but because I'm so dense it took me a bit of trial and error to figure out where all the changes suggested by the documentation should go in my own app. Here's a concrete example of what I had to change for my conventional Mix + Phoenix application.

(The docs describe how to add N different replicas dynamically — MyApp.Repo.Replica1, MyApp.Repo.Replica2, etc — but since I only have to worry about a single endpoint for my read replicas, I simplified things and just used a single, static MyApp.Repo.Replica instance in my Elixir configuration and code.)

Mix Environment Helper

To allow my app to determine whether it was compiled and is running with a test, dev, or prod configuration, I added a config_env setting to my app, and set it to the value of the Mix.env/0 function at compile time:

# config/config.exs config :my_app, config_env: Mix.env(), ecto_repos: [MyApp.Repo] end

Note that with Elixir 1.11 and newer, you can instead use Config.config_env/0 in place of Mix.env/0:

# config/config.exs config :my_app, config_env: Config.config_env(), ecto_repos: [MyApp.Repo] end

And in my root MyApp module, I added a helper function to access this config_env setting:

# lib/my_app.ex defmodule MyApp do def config_env, do: Application.get_env(:my_app, :config_env) end

This means that I can call MyApp.config_env/0 at runtime in various places in my app's code, and get the Mix.env/0 value with which the app was compiled (like :test, :dev, or :prod).

Replica Module

To my existing lib/my_app/repo.ex file (which already contained the MyApp.Repo module), I added the definition for my new MyApp.Repo.Replica module, like so:

# lib/my_app/repo.ex defmodule MyApp.Repo do use Ecto.Repo, otp_app: :my_app, adapter: Ecto.Adapters.Postgres def replica, do: MyApp.Repo.Replica end defmodule MyApp.Repo.Replica do use Ecto.Repo, otp_app: :my_app, adapter: Ecto.Adapters.Postgres, default_dynamic_repo: if(MyApp.config_env() != :test, do: MyApp.Repo.Replica, else: MyApp.Repo), read_only: true end

The default_dynamic_repo option in the MyApp.Repo.Replica module uses the config_env helper I added above to set up the module to use the primary MyApp.Repo's own connection pool for the read replica in the test environment, as recommended by the Ecto docs. This way the replica instance will just delegate to the primary repo instance for all of its read operations in the test environment, but will still enforce its own read-only setting. Also, this way I don't have to configure any test-env-specific settings for the read replica in my config/test.exs file (nor do I need to start up another child process for the replica, as we'll see in the next section).

Application Module

In non-test environments, the new read replica module does need to be started as a child process, alongside the primary repo. So I modified the start/2 function in my application module to start it:

# lib/my_app/application.ex defmodule MyApp.Application do use Application def start(_type, _args) do # don't start separate readonly repo in test mode repos = if MyApp.config_env() != :test do [MyApp.Repo, MyApp.Repo.Replica] else [MyApp.Repo] end children = repos ++ [ MyApp.Repo, MyAppWeb.Endpoint ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end

Dev Config

For my dev environment configuration, I updated my config/dev.exs file to simply duplicate the configuration of the primary MyApp.Reop for the MyApp.Repo.Replica (creating a separate connection pool to the same database as the primary for the replica):

# config/dev.exs config :my_app, MyApp.Repo, for repo <- [MyApp.Repo, MyApp.Repo.Replica] do config :my_app, repo, username: "myusername", password: "mypassword", database: "mydatabase", hostname: "localhost", show_sensitive_data_on_connection_error: true, pool_size: 10 end

Prod Config

For the prod environment configuration, I updated my config/releases.exs file to use a similar configuration as the primary for the replica, but have it instead pull the replica hostname from a different environment variable (DB_READONLY in this case):

# config/releases.exs config :my_app, MyApp.Repo, ssl: true, username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), database: System.get_env("DB_DATABASE"), hostname: System.get_env("DB_HOSTNAME"), pool_size: String.to_integer(System.get_env("DB_POOLSIZE") || "10") config :my_app, MyApp.Repo.Replica, ssl: true, username: System.get_env("DB_USERNAME"), password: System.get_env("DB_PASSWORD"), database: System.get_env("DB_DATABASE"), hostname: System.get_env("DB_READONLY"), pool_size: String.to_integer(System.get_env("DB_POOLSIZE") || "10")

Using the Replica

With all the above in place, everywhere in my Elixir code that I want to query a read replica instead the primary database, I can just replace MyApp.Repo with MyApp.Repo.replica():

# lib/my_app/users.ex import Ecto.Query alias MyApp.Repo alias MyApp.Users.User def list_usernames do from(u in User, select: u.username) |> Repo.replica().all() end