Metrics on Fly.io

Illustration by Annie Ruygt of some Machines strutting along

The Fly.io platform includes a fully-managed metrics solution to help you easily monitor your Fly apps. It includes the following components:

Prometheus on Fly.io: Managed Prometheus-compatible time series storage
Dashboards: Managed Grafana with detailed visualizations of all built-in metrics
Built-in Metrics: Metrics automatically sent from every Fly app you deploy
Custom Metrics: Expose additional metrics from Fly apps for further customization

Prometheus on Fly.io

Prometheus is a popular open source monitoring system used to store and query metrics efficiently, with a stable HTTP querying API compatible with a range of systems. Prometheus on Fly.io is a fully-managed service based on VictoriaMetrics. It supports most common Prometheus querying API endpoints:

Note that remote read (/api/v1/read) remote storage integration is not supported.

MetricsQL

Prometheus queries are typically based on the PromQL query language. Prometheus on Fly.io queries use VictoriaMetrics MetricsQL, a backwards-compatible query language that fixes user experience issues and adds useful features and functions on top of PromQL. Key features:

Better rate() and increase() functions that just work. No need for irate workarounds or appending Grafana’s magical $__rate_interval selector to every query. In fact, you can even omit the square brackets entirely and MetricsQL will do the right thing.
Many more label manipulation functions such as drop_common_labels, label_set, etc.
topk_avg, which returns the top k time series averaged across the entire series range (not just individual points), plus the sum of all remaining series in an “other” label. Useful for giving a small, filtered view across a potentially large number of series.

Querying

Queries can be sent to the following endpoint:

https://api.fly.io/prometheus/<org-slug>/

You’ll need to authenticate with a Fly Access Token. Depending on the token type, the header uses either the Bearer format (Authorization: Bearer <token>) or the FlyV1 format (Authorization: FlyV1 <token>). You may only query series scoped to your organizations.

Manually

Find your Organization slug List your organizations, find the org slug and set it as a local variable.

flyctl orgs list
ORG_SLUG=[org-slug]

Get an access token

TOKEN=$(flyctl auth token)

Test it out!

curl https://api.fly.io/prometheus/$ORG_SLUG/api/v1/query \
  --data-urlencode 'query=sum(increase(fly_edge_http_responses_count)) by (app, status)' \
  -H "Authorization: Bearer $TOKEN"

Retention, cost, and limits

Prometheus on Fly.io retains metric data for approximately 15 days and is intended for operational monitoring. If you need longer retention, run your own monitoring stack. Fly.io’s Observability for User Apps guide deploys a ready-made stack, or you can federate from the Fly.io endpoint into your own Prometheus installation. There’s currently no additional charge for the managed Prometheus and Grafana. Pricing could change in the future, and Fly.io will give advance notice. A couple of practical limits to be aware of:

We cap your app’s metrics endpoint response at 16 MB and drop anything larger.
We may drop very high-cardinality custom metrics.

Dashboards

For more advanced metrics monitoring, you can use dashboards to organize and visualize complex Prometheus queries. The Metrics tab on the Fly.io Dashboard provides an overview of your Fly apps using the built-in metrics stored in Prometheus.

Managed Grafana

Grafana is a popular open source data visualization web application, that allows you to compose queries against data sources into dynamic, reusable dashboards. We provide a managed Grafana instance at fly-metrics.net, preconfigured with your Prometheus data source and detailed dashboards covering the full set of built-in metrics. You can also use the Explore panel to run ad-hoc queries against the preconfigured Prometheus datasource, or create/import additional dashboards for further customization or to visualize custom metrics. Switch between your Fly.io Organizations by clicking the “Switch organization” link beneath the user icon in the lower-left of the screen.

External or self-hosted Grafana

You can also configure your Prometheus endpoint with an existing Grafana installation, or host one on Fly.io. Either way, you can set it up like this:

Add a Prometheus data source (Settings -> Data Sources -> Add data source -> Prometheus)
Fill the form with the following:

HTTP -> URL: https://api.fly.io/prometheus/<org-slug>/
Custom HTTP Headers -> + Add Header:
- Header: Authorization
- Value: FlyV1 <token> (for tokens created with fly tokens create)
- Value: Bearer <token> (only for tokens from flyctl auth token)

You’re all set. We publish our Fly.io Dashboards to Grafana.com for use with external Grafana instances. To install, just import the dashboard using the listed IDs. If you’d like to contribute changes to the dashboards, we have created a repository for them.

Alerting

Fly.io doesn’t include built-in alerting on metrics, so you’ll need to set up alerting yourself against the Prometheus endpoint. Two common approaches:

Grafana alerting: connect the Fly.io Prometheus data source to a Grafana instance (see the External or self-hosted Grafana section), then create Grafana alert rules on queries against the data source and send the alerts to a contact point such as email, Slack, or PagerDuty.
Prometheus and Alertmanager: run your own Prometheus that federates from the Fly.io endpoint and use Alertmanager to send notifications.

If you don’t already run Grafana or Prometheus, Fly.io’s Observability for User Apps guide deploys a ready-made stack (VictoriaMetrics and Grafana) you can build alerts on.

Built-in metrics

Fly apps automatically publish a number of built-in metrics. Metric types are all Gauges unless otherwise marked. Metrics with names ending in _count are all Counters. Histogram metrics with a base name of <name> expose multiple series:

<name>_bucket{le}
<name>_sum
<name>_count

Standard Labels

All published series include the following labels:

app: App name
region: Fly.io Region
host: 4-character host ID (lowercase hexadecimal)
instance: App instance ID (for all series except fly_edge_ and fly_volume_)

If your app exposes custom metrics with the same labels, they will be overwritten.

Proxy series

Any app using a TCP-based handler (HTTP, TLS or straight TCP) publishes edge and app proxy metrics: Labels:

proxy_id: “blue” or “green” (flips when the proxy is restarted/updated)

Edge - `fly_edge_`

fly_edge_http_responses_count{status}
fly_edge_http_response_time_seconds{status} (Histogram)
fly_edge_tcp_connects_count
fly_edge_tcp_disconnects_count
fly_edge_data_out (Counter, bytes)
fly_edge_data_in (Counter, bytes)
fly_edge_tls_handshake_errors{servername} (Counter)
fly_edge_tls_handshake_time_seconds{version} (Histogram)

App - `fly_app_`

fly_app_concurrency
fly_app_http_responses_count{status}
fly_app_http_response_time_seconds{status} (Histogram)
fly_app_connect_time_seconds (Histogram)
fly_app_tcp_connects_count
fly_app_tcp_disconnects_count

Instance series - `fly_instance_`

Derived from the /proc file system of your app VMs. fly_instance_up = 1 shows the VM is reporting correctly.

Instance exit - `fly_instance_exit_`

Information about instance exits. These metrics help you understand why your instances are terminating and can be used for alerting and debugging.

fly_instance_exit_code: The exit code of the main process when the instance terminates. A value of 0 typically indicates normal termination, while non-zero values indicate errors or abnormal termination.
fly_instance_exit_oom: A boolean flag (0 or 1) indicating whether the instance was killed due to out-of-memory (OOM). A value of 1 means the instance was terminated because it exceeded its memory limits.
fly_instance_exit_vm_code: The VM-level exit code, which may differ from the application exit code. This can help distinguish between application-level failures and VM-level issues.

Instance memory - `fly_instance_memory_`

Derived from /proc/meminfo. All units are in bytes.

fly_instance_memory_mem_total
fly_instance_memory_mem_free
fly_instance_memory_mem_available
fly_instance_memory_buffers
fly_instance_memory_cached
fly_instance_memory_swap_cached
fly_instance_memory_active
fly_instance_memory_inactive
fly_instance_memory_swap_total
fly_instance_memory_swap_free
fly_instance_memory_dirty
fly_instance_memory_writeback
fly_instance_memory_slab
fly_instance_memory_shmem
fly_instance_memory_vmalloc_total
fly_instance_memory_vmalloc_used
fly_instance_memory_vmalloc_chunk

Instance Load and CPU

load_average is derived from /proc/loadavg (getloadavg). It’s a “system load average” measuring the number of processes in the system run queue, with samples representing averages over 1, 5, and 15 minutes.
cpu is derived from /proc/stat, and counts the amount of time each CPU (cpu_id) has spent performing different kinds of work (mode, which may be one of user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice). The time unit is ‘clock ticks’ of centiseconds (0.01 seconds).

The following CPU metrics are related to CPU Performance:

cpu_baseline is the baseline quota in number of CPUs, calculated from the CPU type and number of vCPUs.
cpu_balance is the accrued CPU burst balance in clock ticks (centiseconds).
cpu_throttle is derived from the throttled_time field of the cgroup cpu.stat, and counts the amount of time the CPU was throttled after exhausting its quota, in ‘clock ticks’ (centiseconds).

fly_instance_load_average{minutes}
fly_instance_cpu{cpu_id, mode} (Counter, centiseconds)
fly_instance_cpu_baseline (CPUs)
fly_instance_cpu_balance (centiseconds)
fly_instance_cpu_throttle (Counter, centiseconds)

Instance Disks - `fly_instance_disk_`

Counters derived from fields 1-11 of /proc/diskstats. The unit for time_ series is milliseconds, and the unit for sectors_ is 512-byte sectors. Labels:

device: Published for the ephemeral VM root disk (vdb) and any mounted Volume (vdc).

fly_instance_disk_reads_completed
fly_instance_disk_reads_merged
fly_instance_disk_sectors_read
fly_instance_disk_time_reading
fly_instance_disk_writes_completed
fly_instance_disk_writes_merged
fly_instance_disk_sectors_written
fly_instance_disk_time_writing
fly_instance_disk_io_in_progress
fly_instance_disk_time_io
fly_instance_disk_time_io_weighted

Instance Networking - `fly_instance_net_`

Counters derived from /proc/net/dev. Labels:

device: interface name, either eth0 or dummy0 (ignore).

fly_instance_net_recv_bytes
fly_instance_net_recv_packets
fly_instance_net_recv_errs
fly_instance_net_recv_drop
fly_instance_net_recv_fifo
fly_instance_net_recv_frame
fly_instance_net_recv_compressed
fly_instance_net_recv_multicast
fly_instance_net_sent_bytes
fly_instance_net_sent_packets
fly_instance_net_sent_errs
fly_instance_net_sent_drop
fly_instance_net_sent_fifo
fly_instance_net_sent_colls
fly_instance_net_sent_carrier
fly_instance_net_sent_compressed

Instance File Descriptors - `fly_instance_filefd_`

Information about allocated, and maximum allowed allocated file descriptors derived from /proc/sys/fs/file-nr.

fly_instance_filefd_allocated
fly_instance_filefd_maximum

Instance Filesystem - `fly_instance_filesystem_`

Filesystem metrics derived from VFS File System Information. Labels:

mount: mount point name(s), / and if using Volumes, the destination name in fly.toml.

fly_instance_filesystem_blocks
fly_instance_filesystem_block_size
fly_instance_filesystem_blocks_free
fly_instance_filesystem_blocks_avail

Volumes - `fly_volume_`

Labels:

id: Volume ID

If you’re using Volumes for any of your organization’s apps, you’ll be able to query these series, derived from the LSize and Data% of the volume’s thin LV.

fly_volume_size_bytes
fly_volume_used_pct (0-100)

Postgres - `pg_`

If you have a Postgres database hosted on Fly.io, you’ll automatically get the following series, published via postgres_exporter:

pg_stat_activity_count
pg_stat_activity_max_tx_duration
pg_stat_archiver_archived_count
pg_stat_archiver_failed_count
pg_stat_bgwriter_buffers_alloc
pg_stat_bgwriter_buffers_backend_fsync
pg_stat_bgwriter_buffers_backend
pg_stat_bgwriter_buffers_checkpoint
pg_stat_bgwriter_buffers_clean
pg_stat_bgwriter_checkpoint_sync_time
pg_stat_bgwriter_checkpoint_write_time
pg_stat_bgwriter_checkpoints_req
pg_stat_bgwriter_checkpoints_timed
pg_stat_bgwriter_maxwritten_clean
pg_stat_bgwriter_stats_reset
pg_stat_database_blk_read_time
pg_stat_database_blk_write_time
pg_stat_database_blks_hit
pg_stat_database_blks_read
pg_stat_database_conflicts_confl_bufferpin
pg_stat_database_conflicts_confl_deadlock
pg_stat_database_conflicts_confl_lock
pg_stat_database_conflicts_confl_snapshot
pg_stat_database_conflicts_confl_tablespace
pg_stat_database_conflicts
pg_stat_database_deadlocks
pg_stat_database_numbackends
pg_stat_database_stats_reset
pg_stat_database_tup_deleted
pg_stat_database_tup_fetched
pg_stat_database_tup_inserted
pg_stat_database_tup_returned
pg_stat_database_tup_updated
pg_stat_database_xact_commit
pg_stat_database_xact_rollback
pg_stat_replication_pg_current_wal_lsn_bytes
pg_stat_replication_pg_wal_lsn_diff
pg_stat_replication_reply_time
pg_replication_lag
pg_database_size_bytes

Custom Metrics

For further customization beyond built-in metrics, Fly apps can expose a metrics endpoint we’ll automatically scrape every 15 seconds and send the results to Prometheus.

Configuration

Add a [metrics] section to your application’s fly.toml:

[metrics]
port = 9091
path = "/metrics" # default for most prometheus exporters

If your app uses multiple processes, you can add multiple [[metrics]] sections, each with its own set of processes:

[[metrics]]
port = 9394
path = "/metrics"
processes = ["web"]

[[metrics]]
port = 9113
path = "/metrics"
processes = ["proxy"]

Instrumentation

Instrument your app and expose your metrics on 0.0.0.0. There are many supported client libraries as well as off-the-shelf exporters able to return Prometheus-formatted metrics.

Authentication

Authenticating to the Prometheus API can be achieved a few different ways, depending on the level of access you want your token to have.

Fly Access Token

As in the earlier example, a full access token can be generated with flyctl auth token and then passed as a bearer token in the Authorization header. The header looks like:

Authorization: Bearer THE_TOKEN

Fly org-restricted or read-only token

This kind of token or “macaroon” can be scoped to a single organization or configured to only allow read operations, which can be safer than using a full-blown read-write token that grants access to all organizations under your account.

Generating tokens

Create an org-restricted token:

fly tokens create org -o THE_ORGANIZATION

Create a read-only org-restricted token:

fly tokens create readonly

These tokens look like this once generated: FlyV1 fm2_lJPECAAAAAAAAC7txBAzYI6PRWhHLT...(a lot of base64-encoded text).

Using tokens

Use the correct scheme based on token type:

Bearer <token> → for flyctl auth token.
FlyV1 <token> → for all tokens created with fly tokens create.

Example with a token created via fly tokens create:

Authorization: FlyV1 fm2_lJPECAAAAAAAAC7txBAzYI6PRWhHLT...(a lot of base64-encoded text)

​Prometheus on Fly.io

​MetricsQL

​Querying

​Manually

​Retention, cost, and limits

​Dashboards

​Managed Grafana

​External or self-hosted Grafana

​Alerting

​Built-in metrics

​Standard Labels

​Proxy series

​Edge - fly_edge_

​App - fly_app_

​Instance series - fly_instance_

​Instance exit - fly_instance_exit_

​Instance memory - fly_instance_memory_

​Instance Load and CPU

​Instance Disks - fly_instance_disk_

​Instance Networking - fly_instance_net_

​Instance File Descriptors - fly_instance_filefd_

​Instance Filesystem - fly_instance_filesystem_

​Volumes - fly_volume_

​Postgres - pg_

​Custom Metrics

​Configuration

​Instrumentation

​Authentication

​Fly Access Token

​Fly org-restricted or read-only token

​Generating tokens

​Using tokens