Intro - Phoenix/Elixir hosting on fly.io

Fly.io is a popular hosting for personal Phoenix projects because small apps can be hosted for free for up to 3 shared vCPU cores and up to 3 GB of storage. It is a breeze to deploy a non umbrella Phoenix project to fly.io however an umbrella project - not so much. Hopefully the following step by step instructions will be help for someone as I was struggling with the deployment myself.

All plans require a credit card. Your first organization starts on a free Hobby plan (subject to change), and any subsequent new organizations you create are on the $5/month Hobby Plan. Hobby plans are a straightforward pay-as-you-go option.

We don’t offer a “free tier.” Instead, we offer some free resource allowances that apply to all plans, including the Hobby plan, and includes enough usage per month to run a small full-stack app, full-time, for free.

source: https://fly.io/docs/about/pricing/#plans

If I am reading it correctly the free allowance might change in future, however as of time of the writing it is a good free(ish) hosting for Phoenix.

Step by step instructions

The following steps are based on my personal experiense that worked for me at the time of writing this post. Depending on changes in the flyctl and your Phoenix version the steps might need to be changed.

1. Generate a Phoenix umbrella project

We will create a new project for the demonstration purposes. In a terminal run:

mix phx.new hello_app --database sqlite3 --umbrella --adapter bandit

Output:

* creating hello_app_umbrella/.gitignore
* creating hello_app_umbrella/config/config.exs
* creating hello_app_umbrella/config/dev.exs
* creating hello_app_umbrella/config/test.exs
* creating hello_app_umbrella/config/prod.exs
* creating hello_app_umbrella/config/runtime.exs
* creating hello_app_umbrella/mix.exs
* creating hello_app_umbrella/README.md
* creating hello_app_umbrella/.formatter.exs
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web.ex
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web/application.ex
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web/endpoint.ex
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web/router.ex
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web/telemetry.ex
* creating hello_app_umbrella/apps/hello_app_web/lib/hello_app_web/controllers/error_json.ex
* creating hello_app_umbrella/apps/hello_app_web/mix.exs
* creating hello_app_umbrella/apps/hello_app_web/README.md
* creating hello_app_umbrella/apps/hello_app_web/.gitignore
...

Fetch and install dependencies? [Yn] y
* running mix deps.get
* running mix assets.setup
* running mix deps.compile

We are almost there! The following steps are missing:
    $ cd hello_app_umbrella

Then configure your database in config/dev.exs and run:
    $ mix ecto.create

Start your Phoenix app with:
    $ mix phx.server

You can also run your app inside IEx (Interactive Elixir) as:
    $ iex -S mix phx.server

2. Open the umbrella project in the terminal

cd hello_app_umbrella

3. Launch Phoenix locally to make sure it works

mix phx.server

Output:

...
[info] Migrations already up
[info] Running HelloAppWeb.Endpoint with Bandit 1.1.0 at 127.0.0.1:4000 (http)
[info] Access HelloAppWeb.Endpoint at http://localhost:4000
[watch] build finished, watching for changes...

(optional) What happens if we run fly launch in app’s root folder?

Now we are moving to the tricky part. Let’s see what happens if we try to deploy our Phoenix umbrella app straight away. In the hello_app_umbrella folder run fly launch. Reply ‘No’ to Postgres and Redis setup.

fly launch

You should see an error:

Running Docker release generator
Error: failed running /opt/homebrew/bin/mix phx.gen.release --docker: exit status 1

Full output:

/hello_app_umbrella (main)> fly launch

Creating app in /Users/mikatuo/dev/tutorials/1_deploy_phoenix_umbrella_to_fly_io/hello_app_umbrella
Scanning source code
Resolving Hex dependencies...
Resolution completed in 0.097s
Unchanged:
  ...
[info] Migrations already up
Detected a Phoenix app
? Choose an app name (leave blank to generate one): 
automatically selected personal organization: ...
Some regions require a paid plan (bom, fra, maa).
See https://fly.io/plans to set up a plan.

? Choose a region for deployment: Los Angeles, California (US) (lax)
App will use 'lax' region as primary

Created app 'icy-glitter-8336' in organization 'personal'
Admin URL: https://fly.io/apps/icy-glitter-8336
Hostname: icy-glitter-8336.fly.dev
Set secrets on icy-glitter-8336: SECRET_KEY_BASE
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
Generating rel/env.sh.eex for distributed Elixir support
Preparing system for Elixir builds
Installing application dependencies
Running Docker release generator
Error: failed running /opt/homebrew/bin/mix phx.gen.release --docker: exit status 1

Fly uses docker images for deployments. To generate a Dockerfile fly launch calls mix phx.gen.release command.

4. Run fly launch in the *_web folder

The correct way to generate the Docker file will be to do it from the _web folder of your Phoenix project. In our case it will be apps/hello_app_web. That said let’s switch to that folder:

cd apps/hello_app_web

Now that we are in the correct folder let’s try one more time:

fly launch

/hello_app_web (main)> fly launch
Creating app in /hello_app_umbrella/apps/hello_app_web
Scanning source code
Resolving Hex dependencies...
Resolution completed in 0.108s
Unchanged:
  ...
[info] Migrations already up
Detected a Phoenix app
? Choose an app name (leave blank to generate one): 
automatically selected personal organization: ...
Some regions require a paid plan (bom, fra, maa).
See https://fly.io/plans to set up a plan.

? Choose a region for deployment: Los Angeles, California (US) (lax)
App will use 'lax' region as primary

Created app 'throbbing-dust-8712' in organization 'personal'
Admin URL: https://fly.io/apps/throbbing-dust-8712
Hostname: throbbing-dust-8712.fly.dev
Set secrets on throbbing-dust-8712: SECRET_KEY_BASE
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
Generating rel/env.sh.eex for distributed Elixir support
Preparing system for Elixir builds
Installing application dependencies
Running Docker release generator
Wrote config file fly.toml
? Would you like to deploy now? No

If we would try to deploy the app then we would get a few errors. For me 3 of the Docker build steps were failing:

==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
...
 => ERROR [builder  5/17] COPY mix.exs mix.lock ./
...
 => ERROR [builder  8/17] COPY config/config.exs config/prod.exs config/
...
 => ERROR [builder 15/17] COPY config/runtime.exs config/
...
Error: failed to fetch an image or build from source: error building: failed to solve: failed to compute cache key: "/config/runtime.exs" not found: not found

We are missing a few files in the hello_ap_web folder of our Phoenix project and the reason for that is because they are in our umbrella’s project root folder. This gives us a hint to try running fly deploy from the root folder and so we are going to rearrange files that fly launch generated for us.

5. Move files generated by fly launch into the root folder

The following files need to be moved from the /apps/hello_app_web into the Phoenix umbrella project’s root folder:

/apps/hello_app_web/.dockerignore
/apps/hello_app_web/Dockerfile
/apps/hello_app_web/fly.toml
/apps/hello_app_web/rel/*          <- folder with the files

Like so:

We are about half way through. We will try to run fly deploy now from the umbrella’s root folder.

We can now go back into the project’s root folder and try deploying the app:

cd ../..
fly deploy

 ...
 => CACHED [builder  2/17] RUN apt-get update -y && apt-get install -y build-essential git && apt-get clean && rm -f /var/lib/apt/lists/*_*
 => CACHED [builder  3/17] WORKDIR /
 => CACHED [builder  4/17] RUN mix local.hex --force && mix local.rebar --force
 => CACHED [builder  5/17] COPY mix.exs mix.lock ./
 => CACHED [builder  6/17] RUN mix deps.get --only prod
 => CACHED [builder  7/17] RUN mkdir config
 => CACHED [builder  8/17] COPY config/config.exs config/prod.exs config/
 => CACHED [builder  9/17] RUN mix deps.compile
 => ERROR [builder 10/17] COPY priv priv
 => ERROR [builder 11/17] COPY lib lib
------
 > [builder 10/17] COPY priv priv:
------
------
 > [builder 11/17] COPY lib lib:
------
Error: failed to fetch an image or build from source: error building: failed to solve: failed to compute cache key: "/lib" not found: not found

Despite the errors we are making progress!

6. Fix Dockerfile

With a few simple changes we can fix the Dockerfile to work for our umbrella Phoenix project. Let’s do that now.

To fix the fly.io error ERROR [builder 10/17] COPY priv priv we have to specify correct path to the priv folder. Please change the COPY priv priv line in the Dockerfile:

COPY apps/hello_app/priv apps/hello_app/priv
COPY apps/hello_app_web/priv apps/hello_app_web/priv

Similarly let’s fix the fly.io error ERROR [builder 11/17] COPY lib lib by specifying the correct path. Replace the COPY lib lib line with:

COPY apps/hello_app/lib apps/hello_app/lib
COPY apps/hello_app_web/lib apps/hello_app_web/lib

The assets folder is also in the umbrella’s /apps/hello_app_web while the Dockerfile is in the root folder. So edit the COPY assets assets line:

COPY apps/hello_app_web/assets apps/hello_app_web/assets

Replace the RUN mix assets.deploy line with:

RUN cd apps/hello_app_web && mix assets.deploy

In the Dockerfile there is a section to install mix dependencies. We have to modify it to resolve dependencies for our nested projects. In order to do that we will copy nested mix.exs files when building the Docker image. After the COPY mix.exs mix.lock ./ line in the Dockerfile add two more lines:

COPY apps/hello_app/mix.exs ./apps/hello_app/mix.exs
COPY apps/hello_app_web/mix.exs ./apps/hello_app_web/mix.exs

Git diff for the changes in Dockerfile we’ve made so far:

At the very bottom of the Dockerfile replace hello_app_web with hello_app_umbrella:

COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/hello_app_umbrella ./

7. Modify mix.exs

Next we will add the following lines into the project function after the aliases: aliases() line:

releases: [
  hello_app_umbrella: [
    applications: [
      hello_app: :permanent,
      hello_app_web: :permanent
    ]
  ]
]

Git diff for the mix.exs should look like:

8. Modify config/runtime.exs

Uncomment a line in the runtime.exs file for OTP releases:

config :hello_app_web, HelloAppWeb.Endpoint, server: true

9. Fix /app/bin/migrate and /app/bin/server commands

If we run fly deploy the docker image will build successfully now and we would get an error:

Running throbbing-dust-8712 release_command: /app/bin/migrate

...
-------
 ✖ release_command failed
-------
Error release_command failed running on machine 4d89dee0c29758 with exit code 127.
Check its logs: here's the last 100 lines below, or run 'fly logs -i 4d89dee0c29758':
  Pulling container image registry.fly.io/throbbing-dust-8712:deployment-01HEYFF608Y0H7VAECQHGJ8QS0
  Successfully prepared image registry.fly.io/throbbing-dust-8712:deployment-01HEYFF608Y0H7VAECQHGJ8QS0 (3.523646509s)
  Configuring firecracker
  [    0.046193] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
  [    0.051259] PCI: Fatal: No config space access function found
   INFO Starting init (commit: 15238e9)...
   INFO Preparing to run: `/app/bin/migrate` as nobody
   INFO [fly api proxy] listening at /.fly/api
  2023/11/11 06:01:02 listening on [fdaa:3:5c1e:a7b:1e3:643a:4cc4:2]:22 (DNS: [fdaa::3]:53)
  /app/bin/migrate: 3: exec: ./hello_app_web: not found
   INFO Main child exited normally with code: 127
   INFO Starting clean up.
   WARN hallpass exited, pid: 315, status: signal: 15 (SIGTERM)
  2023/11/11 06:01:03 listening on [fdaa:3:5c1e:a7b:1e3:643a:4cc4:2]:22 (DNS: [fdaa::3]:53)
  [    2.285248] reboot: Restarting system
  machine restart policy set to 'no', not restarting
-------
Error: release command failed - aborting deployment. error release_command machine 4d89dee0c29758 exited with non-zero status of 127

There is a lot in the output, but if we read through the logs that fly deploy is showing there is actually a hint of what is wrong. There is a line in the above output:

/app/bin/migrate: 3: exec: ./hello_app_web: not found

There is a section in fly.toml file to execute the /app/bin/migrate during the deployment which executes /rel/overlays/bin/migrate or /rel/overlays/bin/migrate.bat file from our umbrella project:

[deploy]
  release_command = "/app/bin/migrate"

In step 5. Move files generated by fly launch into the root folder we have moved the /rel folder from /apps/hello_app_web into the root of the project but we haven’t modified them yet. The fix will be simple.

In the /rel/overlays/bin/migrate replace “hello_app_web” with “hello_app_umbrella”:

#!/bin/sh
cd -P -- "$(dirname -- "$0")"
exec ./hello_app_umbrella eval HelloAppWeb.Release.migrate

In the /rel/overlays/bin/migrate.bat replace “hello_app_web” with “hello_app_umbrella”:

call "%~dp0\hello_app_umbrella" eval HelloAppWeb.Release.migrate

9. Set DATABASE_PATH, SECRET_KEY_BASE environment variables

After running fly deploy now we get an error

-------
 ✖ release_command failed
-------
...
ERROR! Config provider Config.Reader failed with:
  ** (RuntimeError) environment variable DATABASE_PATH is missing.
  For example: /etc/hello_app/hello_app.db
      /app/releases/0.1.0/runtime.exs:12: (file)

For now we will keep it as simple as possible and simply use local storage of that machine which means that our SqLite3 DB will be wiped out after each deployment. We will deploy our Phoenix umbrella project first and after that will make the DB use persistent storage.

I will run the following command in my terminal:

fly secrets set DATABASE_PATH=../hello_app.db

And another one:

fly secrets set SECRET_KEY_BASE=$(mix phx.gen.secret)

10. Final fly deploy

It was hard but worth it! Sweet feeling of success!

Conclusion

It might be hard and frustrating to deal with so many errors and problems during the deployment. I keep reminding myself to always pay attention to errors as they are our best friends in figuring out what and how to fix. Hopefully this post with step by step instructions was helpful. Feel free to rich out if you are stuck.

Thank you for reading!