# Fly.io Billing
Source: https://docs.fly.io/about/billing
Understand how Fly.io bills each organization monthly: Machine, volume and snapshot charges, invoices, unified billing, payment options, and account deletion.
## Overview
Billing for Fly.io is monthly per organization. Organizations are administrative entities that enable you to add members, share app development environments, and manage billing.
New customers and all new organizations (including those created by previously existing customers) are billed monthly for resource usage. Refer to our resource [Pricing page](/about/pricing) for details.
***
## View invoices
You can view and download invoices from the **Billing** section of your [dashboard](https://fly.io/dashboard/personal/billing) for each organization.
To download a past invoice, click **View** for the relevant cycle. You'll be sent to the Stripe billing portal where you'll have the option to download the invoice.
***
### Understand your invoice charges
The prices of provisioned services are mostly fixed. If you provision Machines and leave them running all month, we’ll charge you a predictable amount for that Machine.
A common reason for additional charges is extra RAM usage.
For a breakdown of charges, check out your invoice in the Billing section of the [dashboard](https://fly.io/dashboard/personal/billing).
We’re happy to discuss a refund if you do create compute resources by mistake, or if your app receives unexpected traffic due to an attack and generates a surprisingly large bill. Just send an email to [billing@fly.io](mailto:billing@fly.io).
***
## Machine billing
Started Machines are billed per second that they're running (the time they spend in the `started` state), based on the price of a named CPU/RAM combination, plus the price of any additional RAM you specify.
For example, a Machine described in your dashboard as "shared-1x-cpu\@1024MB" is the “shared-cpu-1x” Machine size preset, which comes with 256MB RAM, plus additional RAM (1024MB - 256MB = 768MB). For pricing and available CPU/RAM combinations, read about [Compute pricing](/about/pricing#compute).
Stopped and suspended Machines are billed based on their root file system (rootfs) usage per second (the time they spend in the `stopped` or `suspended` state) by \$0.15 per GB per month.
For example, a Machine described in your dashboard as having 1GB of rootfs stopped for 10 days in the entire month will cost \$0.05. If you have 3 stopped Machines of 1GB rootfs each stopped for 30 days it will cost \$0.45 (\$0.15 x 3).
***
## Volume billing
Volume billing is pro-rated to the hour. For details, see [Volume pricing](/about/pricing#persistent-storage-volumes).
If you create a volume, you will be charged for it. You're billed for volumes that aren't attached to Machines, and for volumes that are attached to Machines in any state, including stopped Machines. Volumes in `pending_destroy` state do not accrue charges.
***
## Volume snapshot billing
Volume snapshot billing is pro-rated to the hour and we subtract the free allowance first. For details, see [Volume Snapshots pricing](/about/pricing#volume-snapshots).
If you create a snapshot of a volume (either manually or through automatic daily snapshots), you will be charged for the storage space it occupies.
Snapshots are stored incrementally, so you're only charged for data that has changed since the previous snapshot for the volume.
For example, if you have a 10GB volume with 2GB of actual data and 5 daily snapshots with minimal changes between them, you might be charged for around 2.5GB of total snapshot storage rather than 50GB (5 × 10GB).
***
## Unified Billing
Unified billing allows you to consolidate billing for multiple organizations under a single organization. This means you'll receive a single invoice for all organizations under unified billing. This feature is useful for managing billing for multiple organizations, such as when you have multiple teams or projects.
**Billing Organization**: The organization that is responsible for paying for the resources used by all organizations linked to it. Any organization in good standing with an attached payment method can be a Billing Organization.
**Linked Organization**: An organization that is linked to another organization for billing purposes. It otherwise functions as a normal organization.
### How Unified Billing Works
* You'll receive a single invoice for all organizations under unified billing.
* Billing Organization must have a payment method on file (credit card or Stripe Link).
* Linked Organizations share credits with their Billing Organization.
* Billing Organizations can have a max of 100 Linked Organizations.
### How to use Unified Billing
You can either create a new Linked Organization or link an existing one:
#### New Organization
A **Linked Organization** can be created from the [new organization page](https://fly.io/organizations/new). When creating a new organization, you can choose an existing Organization to link it to.
#### Link an existing Organization
An existing organization can be converted to a **Linked Organization** from the organization's billing page by clicking the "Link billing to another Organization" button.
* Only admins can link an organization to another organization.
* We recommend saving any invoices you may need for accounting purposes before linking an organization.
***
## Some things to consider
* Each organization has its own [private network](/networking/private-networking). Organizations are isolated from each other, which might
be advantageous from a security standpoint.
* Cross organization networking is still possible with [Flycast](/networking/flycast)
* Detaching a Linked Organization from its Billing Organization is not possible at this time. Instead we suggest creating a new organization and moving resources there.
***
## Payment options
We process payments through Stripe. Monthly invoicing requires a credit card or credits.
### How we use credit cards
We require an active, valid credit card on file for most Fly.io accounts to do things like deploying multiple apps and deploying public images. This is primarily a means to prevent abuse and ensure that we can collect payment at the end of the month.
### Pre-authorization Requests
We may send a pre-authorization request to the issuing bank to verify a bank will authorize charges. Some credit card companies present these as real charges, which may surprise you.
A pre-auth is really just a hold on a credit card; it is not a real charge. We’ll typically pre-authorize a small amount (usually less than US \$10) after signup, then cancel the authorization immediately. Banks may show the pre-auth for up to 10 business days, even though we have not collected any money.
### If you don't have a credit card
While a credit card is the preferred payment method, if you don't have one, then you can add credits to your account. You can purchase credits from the Billing section of the [dashboard](https://fly.io/dashboard/personal/billing) in various amounts (minimum purchase \$25). Your usage cost is subtracted from the credit balance each month.
### Prepaid cards
You can't use a prepaid card as a default (or saved) payment method. You can, however, use a prepaid card to add credits to your account.
***
## Delete your account
To delete your Fly.io account, go to Account > Settings > [Delete Account](https://fly.io/user/deactivate) in your dashboard. You'll see a list of clean-up tasks that need to be done before you can delete the account, including:
* Delete or remove yourself from organizations to avoid leaving behind inaccessible orgs. This does not include your personal organization.
* Delete apps (or remove yourself from the associated organizations).
* Remove certificates so that any custom domains are released.
* Remove extensions to ensure that you don't get billed by partners when you can no longer access your account.
When you've completed the clean-up tasks, click **Delete** to permanently delete your Fly.io account.
Deleting your account removes your name, username, and email address from the account record and revokes your sign-in identities and API tokens. If the account was billed, invoices are retained as required by tax and accounting law.
### Deletion requests by email
Fly.io can't process account or data deletion requests sent by email, including requests made under GDPR Article 17 or similar privacy laws. An email doesn't let us verify that the sender owns the account, so the self-service steps above are the only way to delete an account and the personal data attached to it. If you've lost access to your account, [reset your password](https://fly.io/app/password_resets/new) using the email address on the account.
Newsletter email is managed separately from your account. To stop it, use the unsubscribe link at the bottom of any newsletter. It takes effect immediately and doesn't affect your account or service notifications.
***
## Legacy plans
On a legacy plan and wondering how it's billed? Read more about [Discontinued Plans](/about/discontinued-plans).
# Using Our Brand
Source: https://docs.fly.io/about/brand
Download Fly.io logos in color and monochrome landscape, portrait, and brandmark formats, and review the trademark usage agreement for using them.
**The following guidelines will help you correctly make use of our brand.** By using our brand assets, you accept the terms of our [Trademark Usage Agreement](#trademark-usage-agreement) and understand that, if violated, your permission to use these materials will be revoked.
## *Color Landscape Logo*
Our color landscape logo combines our brandmark and serif wordmark in a landscape format. This is our preferred logo, and we encourage you to use it whenever possible.
[Download Color Landscape Logo](/images/brand/logo-landscape.zip)
[Download Inverted Color Landscape Logo](/images/brand/logo-landscape-inverted.zip)
## *Monochrome Landscape Logo*
The monochrome landscape logo is used to maximize clarity and contrast. Use it when the color version isn’t a good option, such as on a busy or low-contrast background.
[Download Dark Landscape Logo](/images/brand/logo-landscape-dark.zip)
[Download Light Landscape Logo](/images/brand/logo-landscape-light.zip)
## *Color Portrait Logo*
Our color portrait logo combines our brandmark and serif wordmark in a portrait format. Use it when our landscape logo isn’t a good option, such as a tight layout where horizontal space is limited.
[Download Color Portrait Logo](/images/brand/logo-portrait.zip)
[Download Inverted Portrait Logo](/images/brand/logo-portrait-inverted.zip)
## *Monochrome Portrait Logo*
The monochrome portrait logo is used to maximize clarity and contrast. Use it when the color version isn’t a good option, such as on a busy or low-contrast background.
[Download Dark Portrait Logo](/images/brand/logo-portrait-dark.zip)
[Download Light Portrait Logo](/images/brand/logo-portrait-light.zip)
## *Color Brandmark*
Some circumstances call for our brandmark to stand on its own. This is typically reserved for extremely small dimensions, when space is limited, or where the brand is already established.
[Download Color Brandmark](/images/brand/brandmark.zip)
[Download Inverted Color Brandmark](/images/brand/brandmark-inverted.zip)
## *Monochrome Brandmark*
The monochrome brandmark is used to maximize clarity and contrast. Use it when the color version isn’t a good option, such as on a busy or low-contrast background.
[Download Dark Brandmark](/images/brand/brandmark-dark.zip)
[Download Light Brandmark](/images/brand/brandmark-light.zip)
## *Trademark Usage Agreement*
This agreement explains how you must use and present our brand assets at all times. Your use of our brand assets indicates your acceptance of this agreement, as well as your understanding that, if violated, your permission to use these materials will be revoked.
* The Fly.io name and brand assets are trademarks of Fly.io, Inc.
* Use of our brand assets must be expressly authorized in writing.
* Our brand assets are provided for use in informational articles, video tutorials, and other related media only. You may not use our brand assets for t-shirts, stickers, or other merchandise at any time.
* Your use must not mislead consumers as to our sponsorship of, affiliation with, or endorsement of your company or your products or services.
* Our brand assets are our exclusive property. All goodwill that results from your use of our brand assets will be solely to our benefit. You will not take any action that is at odds with our rights or ownership of our brand assets. Where used on a web page, our brand assets, as well as the mention of our name, should include embedded hyperlinks to our homepage: [https://fly.io](https://fly.io).
* Our brand assets must be used in a respectful manner. Our brand assets may not be used in a way that harms us, our products or services, or in a manner which, in our opinion, lessens or otherwise damages our reputation or the goodwill in our brand assets. In other words, please do not associate our brand assets with any illicit or illegal activities or use them in a way that is deceptive or harmful.
# Cost Management on Fly.io
Source: https://docs.fly.io/about/cost-management
Predict your Fly.io bill: budget for always-on Machines, watch bandwidth, volumes, IPv4 and leftover managed services, and know free allowances don't cap it.
## Predicting your Fly.io bill and avoiding surprises
We’ve done our best to make billing on Fly.io something you can reason about. Machines don’t appear out of nowhere. If something is running, it’s because you launched it, or you configured something that did. In general, when we talk about **autoscaling**, we mean starting or stopping machines you’ve already defined. We don’t quietly spin up extras in the background that turn into mystery charges on your bill. The idea is that your bill should always be traceable to something you can see, name, and plausibly explain when someone asks. If it’s not, we’re happy to help you sort it out.
> [!NOTE]
>
> **Need more help understanding your bill?** Reach out to us at [billing@fly.io](mailto:billing@fly.io).
## A few examples
So when you’re budgeting, you can look at what you’ve provisioned and do some quick math. Let’s say you’ve deployed three “shared-1x 1GB” machines in `sjc`. That’s \$23.95/month, tops. If you let them stop when idle, you might only pay for a few hours of compute—but the worst-case scenario is still capped at that monthly rate.
Here are a few more grounded examples:
* A **staging app** running in the `sjc` region with one `shared-1x 256MB` machine that auto-stops when idle might cost less than **\$1/month**. If you leave it running full-time, it’s still only **\$2.62/month**.
* A **queue worker** (also in `sjc`) that uses metrics-based autoscaling on `shared-1x 512MB` machines might average 3–5 machines running during busy hours. If those stay up for 50% of the month, you’re looking at **\$6.61–11.01/month**, worst-case around **\$22 month** if they all run 24/7.
We recommend budgeting for the “always-on” cost. You can get under that number with auto-stop or bursty usage, but don’t depend on it to hit your budget.
If your estimate seems high, the most predictable way to save money isn’t fiddling with auto-stop/start settings (since any random request might spin a machine up again), but by just…running fewer machines. Or smaller ones.
> [!NOTE]
>
> **Want to play with numbers?** Try the [Fly.io pricing calculator](https://fly.io/calculator) to get a rough sense of what your setup might cost. For the full breakdown, here’s our [pricing page](/about/pricing).
## Metrics-based autoscaling can affect costs
There is one important exception to the rule that Fly.io doesn’t create machines on your behalf. If you use the **metrics-based autoscaler**, for example to scale a background worker based on queue depth, it *can* create or destroy machines automatically.
**When can the metrics-based autoscaler create machines?** When you deploy and configure the [**Fly Autoscaler**](/launch/autoscale-by-metric), and use `FAS_CREATED_MACHINE_COUNT` instead of `FAS_STARTED_MACHINE_COUNT`, you're giving it permission to create and destroy machines on your behalf. It still uses the machine sizes and regions you specify, and anything it spins up counts toward your bill. [Read more in the docs](/launch/autoscale-by-metric).
## Other Stuff to Watch For
A few more things that can quietly run up your bill if you're not paying attention:
* **Bandwidth costs can add up.** Outbound data transfer is billed at \$0.02/GB in North America and Europe, with higher rates in some other regions. Ingress is free. If you're serving media, syncing large datasets, or replicating across regions, bandwidth can quietly become a big line item. [Learn about data transfer pricing.](/about/pricing#data-transfer-pricing)
* **Volumes don’t stop billing when your machines do.** Persistent volumes are billed hourly, whether or not your machine is running. That 3GB volume you forgot about in `fra`? It's still costing you. [Read about storage pricing here](/about/pricing#volumes).
* **Volume snapshots are billed.** Snapshot storage is billed on the data stored, after a free allowance each month, so automatic snapshots you keep for backups add to your bill. [See snapshot pricing](/about/pricing#volume-snapshots).
* **Managed services live outside your apps.** Fly's [Managed Postgres](/postgres/index) (MPG), [Upstash Redis](/upstash/redis), and [Tigris object storage](/tigris) don’t get deleted when you delete your apps. You’ll find them in your Dashboard. If you're cleaning up, double-check there too. We've seen people get surprised by bills from leftover services and databases they thought were gone.
* **Dedicated public IPv4 addresses aren’t free.** Each http app gets billed \$2/month if you assign it a dedicated public IPv4. If you’ve got a bunch of apps or deploy previews per branch, this can sneak up on you.
* **Internal-only apps still count.** Just because a Fly app doesn’t serve web traffic doesn’t mean it’s free. Background workers, queue processors, cron jobs: they all run machines, and those still cost money.
## Understanding Free Usage and Overages
* **Free allowances don’t cap your bill.** We may give you free credits and usage allowances, especially when you’re starting out or have a legacy billing plan. But there’s no soft ceiling. If you go over, we’ll bill you. We don't support billing alerts (yet), so budget accordingly.
* **There is no "free account/free tier" on Fly.io.** We do have a Free Trial program, which you can read about [here](/about/free-trial).
* **Check your dashboard often.** To spot ballooning costs and overages before they become an issue, check the "current month to date bill" item in your dashboard.
## Related Reading
Want to go deeper on scaling strategies, autoscaling configuration, or controlling concurrency? These docs walk through the mechanics:
* [Autoscale Fly Machines](/blueprints/autoscale-machines) A practical guide to getting basic autoscaling working with Fly Machines. Covers HTTP-based scaling triggers.
* [Metrics-based Autoscaling](/launch/autoscale-by-metric) This is the one that *can* create machines automatically, based on queue depth or another metric you specify. Use it wisely.
* [Setting Concurrency Limits](/blueprints/setting-concurrency-limits) Shows how to keep your app from getting overwhelmed by too many concurrent requests. Especially useful if you’re scaling horizontally.
# Discontinued Plans
Source: https://docs.fly.io/about/discontinued-plans
Fly.io stopped selling Hobby, Launch, and Scale plans on October 7, 2024. See the legacy allowances still honored, pro-rated billing, and Pay As You Go.
## Overview
Fly.io no longer offers plans to new customers. If you purchased a Launch or Scale plan before October 7, 2024, you can remain on those plans unless you convert to Pay As You Go, delete your payment method, or otherwise stop using Fly.io.
This page covers what those discontinued plans include and how they're billed. If you're not on a legacy plan, see [Pricing](/about/pricing) and [Billing](/about/billing) instead.
***
## Legacy free allowances
Fly.io has deprecated plans as of October 7, 2024, but is still honoring them for users who purchased them prior to that date. The following resources were included for free on the Hobby (deprecated), Launch, and Scale plans, and are still honored for any organizations that were on these plans before we sunset them:
* Up to 3 shared-cpu-1x 256mb VMs
* 3GB persistent volume storage (total)
Outbound data transfer:
* 100 GB North America & Europe
* 30 GB Asia Pacific, Oceania & South America
* 30 GB Africa & India
If your plan includes free allowances, we subtract them off the top of your usage total. Once your usage exceeds the free allowance for any given resource, we start metering usage of that resource for billing purposes.
***
## Legacy plan billing
> [!NOTE]
>
> Customers who received a \$5 free trial usage credit when signing up for the \$5/mo Hobby plan will be billed monthly after that credit is used up.
Legacy plan billing, and any included usage, is pro-rated over the month. For example, if you signed up for a Scale plan on September 15th, then you'd be charged \$14.50 (half of the \$29/mo plan cost) right away and you'll have \$14.50 of usage included up to September 30th. Then you'll be charged \$29 at the start of October and have \$29/mo of usage included for that billing cycle.
***
## Paid Hobby plan with Free Trial credit
If you signed up for the now deprecated \$5/month paid Hobby plan prior the release of the Pay As You Go plan, you got a one-time \$5 free trial credit to let you test-drive Fly.io at no cost. When the free trial credit was used up, we automatically placed your organization on the \$5/month Hobby plan, which included \$5/month of usage and free allowances. There are no free allowances during the free trial. We'll send you an email when your free trial credit is used up and you won't be charged before that. The free trial credit doesn't expire and applies only to the default (“personal”) organization that we created for you on sign-up.
To change your plan to the Pay As You Go plan, go to the [**Organizations** page](https://fly.io/organizations) in the dashboard, click the organization name to change, then click **Choose Pay As You Go**. If you change your plan, you won't be able to return to the paid Hobby Plan.
***
## Legacy (Free) Hobby plan
If you were on the free Hobby plan at the time that the paid Hobby plan became the default for new organizations, your plan is now called the Legacy Hobby plan. Your costs stay the same as they were, with no monthly subscription fee, and no included usage beyond the free resource allowances.
If you change your plan, you won't be able to return to the Legacy Hobby Plan.
***
## **Related reading**
* [Fly.io Resource Pricing](/about/pricing) What Fly.io charges for compute, storage, and network today.
* [Billing for Fly.io](/about/billing) How invoicing, payment methods, and usage tracking work.
* [Cost Management](/about/cost-management) Best practices for estimating, monitoring, and controlling your spend.
# Fly.io Extensions Program
Source: https://docs.fly.io/about/extensions
Partner with Fly.io as an extension provider: requirements for account and resource provisioning, SSO via flyctl ext, usage billing APIs, and hosting.
Fly.io is a global cloud service for running full stack apps close to users. We run on our own hardware in [datacenters all over the world](/reference/regions).
We have two main goals: to deliver a slick developer experience, and to build the best possible platform for running full stack apps. We're looking for partners that share this goal, extending Fly.io with services our customers need.
Services such as managed databases, exception handlers, CI runners or log aggregators are great examples. Check out our [managed Upstash for Redis](/upstash/redis) to get an idea of what's possible, and how we integrate services directly into our CLI.
[Check out our documentation](https://fly.io/docs) to see what our platform has to offer. Contact us at [extensions@fly.io](mailto:extensions@fly.io) to discuss your case!
## Provider Requirements
As an extension provider, you should give us a way to:
* Provision resources, user accounts and organizations
* Allow single sign-on login into your UI
* Fetch billing details on a daily basis so customers can stay up-to-date with metered costs
Our [Extensions API documentation](/reference/extensions_api) outlines our API and SSO requirements in detail.
## Account provisioning
Where possible, our organizations should map to your organizations, and our user accounts to yours. In both cases, we provide obfuscated addresses that route to individual customers or to organization administrators.
## Resource provisioning
You should give us an API endpoint to provision extensions synchronously. This request should respond with unique resource ID, connection strings, tokens and other relevant information.
## Single sign on
Customers should be able to login to your UI using a single command like \`flyctl ext logjam dashboard\`\`. This may be done by integrating with our OAuth authentication flow.
Once logged in, customers should have access to your standard platform features. There may be details to iron out here, such as which privileges to assign to Fly.io-provisioned users.
## Billing
We want you to be free to model your pricing as you see fit. But, for the best customer experience, Fly.io should ultimately be the one that charges a customer. Also, Pay-as-you-go pricing fits our model best, since it's how we bill.
So you should run your own billing logic, and offer an API to provide up-to-date, per-account billing details.
Contact us at [extensions@fly.io](mailto:extensions@fly.io) to discuss billing options.
## Deploying latency sensitive services on the Fly.io platform
To fulfil our promise of a performant application in any Fly.io region, latency sensitive services — like Redis or MySQL — should be deployed on Fly.io. You'll also benefit from volume discounts on cloud resources and a growing number of [global points of presence](/reference/regions).
Your Fly.io network is essentially a global, encrypted LAN, with DNS service discovery and load balancing built-in. This greatly simplifies configuration of services that cluster and gossip. Customers access your service via a single private IP address that automatically routes traffic to the nearest provider VM.
We also offer cost-cutting features such as [automatic VM stop/start based on incoming request volume](/launch/autostop-autostart). Contact us at [extensions@fly.io](mailto:extensions@fly.io) to learn more - we're happy to help you get started!
# Fly.io Free Trial
Source: https://docs.fly.io/about/free-trial
Check what the Fly.io free trial includes, its VM hour, Machine, volume and memory limits, how to track usage, and what happens when the trial ends.
**Fly.io runs your apps close to your users. This page explains how our free trial works and what resources you can use before you need to add a payment method.**
## Free Trial overview
A free trial on Fly.io includes 2 hours of machine runtime or 7 days of access, whichever comes first. That’s just enough to get a real app up and running before you decide if it’s a fit for you.
> [!NOTE]
>
> If you hit any of the limits below before your 7 days are up, **your trial is considered exhausted**, and your apps will stop until you add a payment method.
Here’s what’s **included** during the trial:
* **2 total VM hours** (Shared across any machines you launch) \
**Note: Trial Machines are set to automatically stop after running for 5 minutes**
* **10 machines max**
* **20GB of volume storage**
* Up to **2 vCPUs per machine**
* **4GB memory per machine**
These are **not included** in the free trial:
* Dedicated IPv4 addresses
* Access to performance-optimized vCPUs
> [!NOTE]
>
> You can add a credit card from the dashboard at any time during the trial. This lifts the resource limits and keeps your apps running without interruption. **Note: adding a card ends the free trial** and your usage starts counting toward your bill from that point on.
## Checking usage
You can see what you’ve used and what’s left under **“Trial Status”** in the Fly.io dashboard. Machines usage (including memory) is tracked per second, while Volumes and Machine count are tracked per hour.
The free trial is meant for kicking the tires. Spin something up, poke around, deploy a real app or two. If Fly.io fits your use case, great! Add a credit card to keep going. If not, no hard feelings.
## What happens when the free trial ends?
If you don’t add a payment method by the end of your 7-day trial, or if you use up the included resources, your apps will stop running. You won’t be able to launch new machines, attach volumes, or deploy changes until billing is set up.
Whether you're testing the waters or gearing up to launch something real, the free trial gives you room to get started. If Fly.io feels like a fit, adding a payment method from the Dashboard is all it takes to keep going. No surprises, no downtime, no pressure.
***
## Related Reading
* [Pricing Overview](/about/pricing)
* [How Billing Works](/about/billing)
* [Fly.io Pricing Calculator](https://fly.io/calculator)
# Healthcare apps on Fly
Source: https://docs.fly.io/about/healthcare
See how Fly.io supports healthcare apps: SOC2 Type 2 audits, BAA signing, WireGuard encryption in transit, encrypted volumes at rest, 6PN private networking.
Fly.io is a great place to develop and host healthcare applications! You can get started for free and be up and running with a fully secure solution in minutes.
We recognize that healthcare apps and data are different beasts and we're here to protect your patients' data (and keep your auditors happy). We're SOC2 (Type 2) audited, we'll sign BAAs, and we're available to answer questions you might have about how our platform meets your compliance needs [(just ask!)](mailto:sales@fly.io).
Nailing the security for a HIPAA-compliant application can be a big task, and no hosting provider can do it all for you. But Fly.io has a security-first design and a number of features that make HIPAA simpler:
## Access Control
### Database Endpoint Security
Whether you’re running Fly.io Postgres or your own database, there’s no network ACLs required to ensure that only your application can talk to the database server; databases on Fly.io talk to app servers over 6PN and WireGuard, and never to the public Internet.
### Anti-Spoofing Controls
Attackers that boot up evil Fly.io apps can’t spoof packets to other Fly.io instances; we use both Linux kernel controls, IP routing, and eBPF programs to make sure of that. It’s kind of a 1998 sort of attack to worry about, but in case your auditor cares, we took care of it.
## Audit
### Centralizing Logging and Metrics
Fly.io collects logs and metrics from your apps, and can send them wherever you need them to go; you can get a single feed of logs from all your instances into an ELK cluster or a Splunk instance, or aim Grafana at our metrics, so you have visibility and an audit trail of what’s going on with your app.
## Integrity
### Hardened Hosting
Apps running on Fly.io run inside Firecracker, a Rust-based, memory-safe KVM hypervisor designed at Amazon as the engine for Fargate. At Fly.io, we take container images from our users and transmogrify them into VMs, for full, no-shared-kernel isolation between applications. Firecracker VMs run as userland processes on our hosts, and are further locked down with modern Linux security tools, including cgroups, file and network namespaces, resource limits, and privilege separation.
### Kernel Vulnerability Response
Fly.io is responsible for the security both of our host kernels (of course) and of the guest kernels our apps run in; one less thing for you worry about.
## Authentication
### Multi-Factor Authentication
Fly.io supports standard multifactor authentication apps, because of course we do. We feel a little miffed you had to ask.
### Secrets Management
It’s easy to expose secrets to your running apps without leaking them in configurations: we provide a “write-only” secrets management scheme that keeps app secrets encrypted, exposing them only to running instances of your app, using a token-based system that ensures your plaintext secrets never hit machines not running your apps.
## Confidentiality
### WireGuard Everywhere
The Fly.io platform is knitted together out of hosts connected via a WireGuard mesh. Everything talks to everything else over WireGuard.
WireGuard is a next-generation in-kernel (and userland) VPN designed by vulnerability researchers for simplicity, auditability, and modern cryptography. The Linux kernel implementation of WireGuard runs in steady state without requiring dynamic memory allocation! WireGuard is great, and is the gold standard for secure network transports.
We run a full mesh of WireGuard. That means that once a request arrives at our edge from the Internet, when we proxy it to a host running your app, that communication occurs over a WireGuard virtual network.
We also use WireGuard in our API. Need to SSH to an instance of your app? That’ll happen over WireGuard. Need to open a Postgres shell? WireGuard. Deploy a new instance using a remote builder running in Fly.io? You guessed it: the Docker protocol stuff is running over WireGuard, from your machine to our hosts.
### Encryption In Transit
See above! WireGuard runs 256-bit ChaCha20-Poly1305 with an authenticated Curve25519 key exchange.
### TLS Everywhere
Fly.io terminates TLS for our users at our edge. We run a fleet of memory-safe, Rust-based proxies that use the Hyper and Rustls libraries to implement HTTPS, in a tight configuration that scores an A grade from Qualys SSL Labs, without you needing to lift a finger.
You can terminate TLS in your application instead of at our edge, if you really want to. But you won’t want to.
### Automatic Certificate Management with LetsEncrypt
Fly.io manages the ACME protocol to securely provision LetsEncrypt TLS certificates, so that’s another thing you’re just not going to have to think much about.
### HTTP Strict Transport Security and HTTPS-Only
It’s easy (a single configuration line) to lock your apps to HTTPS-only, using the HSTS protocol to direct browsers exclusively to the secure endpoint of your application and defeating SSL-stripping attacks.
### Default-Deny Public Networking
Apps running on Fly.io get routable IPv6 addresses and shared IPv4 addresses. But when your users hit those addresses, they’re not bouncing directly off your app instances; instead, they’re routed to our edge, where we use our memory-safe Rust proxy to direct traffic. What that means for you is that nothing on your app is exposed unless you ask us to expose it. No security group rules or network ACLs required! You’re locked down by default.
### 6PN Private Networking
Modern applications are often composed of ensembles of services. Some of those services are fit for talking to random users on the Internet. Others are best kept under wraps. Fly.io makes it easy to deploy complicated applications built out of multiple services: all Fly.io apps live in a private IPv6 network exclusive to your organization. We use eBPF in the Linux kernel to ensure that private networks can’t talk to each other; they’re completely private, without any extra configuration. We call this feature 6PN (for “IPv6 Private Network”), and it means there’s zero security lockdown work required to deploy a database, Redis cache, or background job scheduler. Your app server will be able to talk to them right away. Nobody outside your organization will be able to talk to them at all.
### Network Segregation
“Network Segregation” is the term an enterprise IT administrator at a big bank would describe 6PN. Yup, your networks are segregated.
### Secure Network Architecture
Sure, you could call 6PN that, too. Did we mention there are no security group rules to review? There are no security group rules to review.
### Encryption At Rest
Databases like Fly.io Postgres are built on Fly.io Volumes, our persistent storage feature. It works like a drive plugged in and mounted in your app instance. And those drives are block-level encrypted with AES-XTS. We manage the keys for the drives for you, using a token-based orchestration system that ensures the keys are only accessible from privileged processes on hosts actually running your app. Check off another HITRUST CSF requirement from your list!
## Availability
### High Availability
Scaling apps across geographic regions is, like, the whole point of the service? I think it’s the whole point? At any rate: it’s definitely a thing we do.
### DDoS Protection
Fly.io isn’t a DDoS Protection provider. But our upstreams have sophisticated DDoS tools, including blackhole routing and traffic scrubbing, which get regular workouts.
### Rolling Deployments
Confidently deploy your Fly.io apps without worrying that you’re going to break everything: we’ll do rolling deploys, with canaries and health-checks, so a known-good version of your app is always running.
### Advanced App Instance Recovery Space Modulator
If your app crashes or exits, we’ll relaunch the VM. We have the technology.
### Off-Site Backups
Anything stored persistently in a Fly.io volume is backed up on a regular schedule.
## More info
Our blueprint for [Going to production with healthcare apps](/blueprints/going-to-production-with-healthcare-apps) runs a developer or operations engineer through the process of evaluating Fly.io’s security for HIPAA healthcare apps, launching a pilot application, signing a BAA, and deploying to production.
You can also check our [community](https://community.fly.io) for answers to your questions.
If you can't find what you're looking for or want to know more, [get in touch](mailto:sales@fly.io).
# About Fly.io
Source: https://docs.fly.io/about/index
The About section of the Fly.io docs: pricing, billing, cost management, the free trial, support, healthcare controls, open source, brand, and legal terms.
More details about using and working with the Fly.io platform.
* **[Pricing](/about/pricing):** Pricing for compute, storage, and all the Fly.io platform services.
* **[Billing](/about/billing):** Learn how billing works on Fly.io.
* **[Cost Management](/about/cost-management):** Read this guide to learn how Fly.io billing really works, how to estimate your costs, and what to watch out for, complete with real examples and gotchas.
* **[Free Trial](/about/free-trial):** Find out about Fly.io's Free Trial.
* **[Support](/about/support):** Who to contact and where to go when you need Fly.io support.
* **[Healthcare](/about/healthcare):** Controls relevant to running healthcare apps on Fly.io.
* **[Engineering Jobs and Hiring](/about/extensions):** Learn more about our hiring process.
* **[Open Source](/about/open-source):** How we help those OSS projects that help us.
* **[Using Our Brand](/about/brand):** Assets and guidelines to help represent our brand.
* **[Privacy Policy](https://fly.io/legal/privacy-policy/):** Our policy on privacy, detailed here.
* **[Terms of Service](https://fly.io/legal/terms-of-service/):** The legal terms of service for Fly.io users.
# Fly.io Merchandise
Source: https://docs.fly.io/about/merch
Where to buy Fly.io merch: T-shirts, hoodies, stickers, and mugs with Frankie the balloon, sold through the Redbubble shop, and who to email for help.
## Fly.io Merchandise
Looking for some superfly swag? We have some Fly merch available in our Redbubble shop. Enjoy!
Visit our [Redbubble shop](https://www.redbubble.com/people/annieruygt/shop) to check out our latest merchandise.
## Available Items
Our Redbubble shop features a variety of Fly.io branded items including:
* T-shirts
* Hoodies
* Stickers
* Mugs
* And more!
## Contact
For any questions about our merchandise, please contact us at [annie@fly.io](mailto:annie@fly.io).
# Fly.io and Open Source
Source: https://docs.fly.io/about/open-source
How Fly.io supports the open source it depends on: equity grants to authors of projects like isolated-vm and acme-client, plus recurring sponsorships.
Fly.io builds on the work of a tremendous open source infrastructure community. We want open source authors to benefit from their work because we think a healthy, thriving open source ecosystem will help us build better products.
## Equity Grants
There's a tradition of giving equity grants to startup advisors who help growing companies. We too give equity grants—with us, though, they go to open source authors whose projects we rely on such as [isolated-vm](https://github.com/laverdet/isolated-vm) and [acme-client](https://github.com/unixcharles/acme-client). Those projects are important to Fly.io and helped us build the developer experience we wanted.
## Recurring donations
We sponsor the developers behind our favorite open source projects. Some we use heavily, like the [Hyper HTTP client for Rust](https://github.com/hyperium/hyper/). Some are just cool, like the [Gleam language](https://gleam.run).
# Fly.io Resource Pricing
Source: https://docs.fly.io/about/pricing
See what Fly.io bills for, from Machines and volumes to Managed Postgres, data transfer, IPs, SSL, support plans and extensions, with sample app costs.
**Fly.io bills for the resources you use, with no monthly platform fee. The smallest always-on app costs \$2.19 per 30 days. A small web app with a Managed Postgres database costs about \$44 a month. The table below lists what we charge for, followed by a few examples of what real-ish apps cost.**## Everything we bill forPrices are in USD. Machine prices in this table are for Ashburn, Virginia (`iad`) and Secaucus, NJ (`ewr`), our lowest-priced regions. Machine prices vary by region; see [Started Fly Machines](#started-fly-machines) for the full list.Monthly Machine prices assume 30 days of continuous runtime. Machines are billed by the second while they're running, so an always-on Machine costs slightly more in a 31-day month.| What you pay for | Price | Notes |
| - | - | - |
| Started Machine, `shared-cpu-1x` with 256MB RAM | \$2.19/month | Smallest Machine size. Billed by the second while running |
| Started Machine, `shared-cpu-1x` with 512MB RAM | \$3.69/month | |
| Started Machine, `shared-cpu-1x` with 1GB RAM | \$6.70/month | |
| Started Machine, `shared-cpu-2x` with 2GB RAM | \$13.39/month | |
| Started Machine, `performance-1x` with 2GB RAM | \$33.00/month | |
| Started Machine, `performance-2x` with 4GB RAM | \$66.00/month | |
| Additional Machine RAM | \$6.00/GB/month | RAM added beyond the amount included with the CPU preset |
| Stopped or suspended Machine | \$0.15/GB/month of rootfs | Only the Machine's root file system is billed while it's stopped |
| Volume | \$0.15/GB/month | Based on provisioned capacity, including while detached or attached to a stopped Machine |
| Volume snapshots | \$0.08/GB/month | First 10GB free each month. Based on data stored, not provisioned volume size |
| Dedicated IPv4 address | \$2/month | Optional. Apps get a shared IPv4 address and IPv6 addresses at no charge |
| Static egress IP | \$0.005/hour (\~\$3.60/month) | Optional |
| SSL certificates | \$0.10/month per single hostname; \$1/month per wildcard | First 10 single-hostname certificates are free |
| Outbound data to the internet | \$0.02/GB in North America and Europe \$0.04/GB in Asia Pacific, Oceania, and South America \$0.12/GB in Africa and India | Inbound data is free |
| Private network data between regions | \$0.006/GB in North America and Europe \$0.015/GB in Asia Pacific, Oceania, and South America \$0.050/GB in Africa and India | Data transfer within a region is free. Different rates apply to some older organizations; see [data transfer pricing](#data-transfer-pricing) |
| Managed Postgres | \$38/month (Basic) to \$1,922/month (Performance) | Storage is \$0.28/GB/month based on usage. See [Managed Postgres pricing](/postgres#pricing) |
| Support | \$29/month Standard; \$199/month Premium; Enterprise starts at \$2,500/month | Optional. Community support is free |
| Fly Kubernetes | \$75/month per cluster | Compute and volumes are billed separately |
| Extensions such as Tigris and Upstash | Provider's list price | Pricing is set by the extension provider and charges appear on your Fly.io invoice |New organizations use Pay As You Go pricing. There's no monthly platform fee; you pay for the resources and services you use.## What does a typical app cost?Here are three example apps and what they'd cost per month. We use Ashburn (`iad`) for the examples unless otherwise noted, and assume the Machines run continuously. Traffic and storage marked "assumed" are example workloads, not included allowances. You can use the [pricing calculator](https://fly.io/calculator/) to estimate costs for your own app.If your Machines [auto-stop](/launch/autostop-autostart) when they're idle, your compute costs can be lower. You pay for the time a Machine is running and for its rootfs while it's stopped. If the Machine runs all month, you'll pay the full always-on cost shown here. See [Cost Management](/about/cost-management) for more ways to estimate and control your bill.### A small web app with Managed Postgres| Item | Monthly cost |
| - | -: |
| One `shared-cpu-1x` 512MB Machine, running continuously | \$3.69 |
| 10GB volume | \$1.50 |
| Managed Postgres, Basic plan | \$38.00 |
| 1GB of Managed Postgres storage (assumed) | \$0.28 |
| Volume snapshots, within the 10GB free allowance (assumed) | \$0.00 |
| 10GB outbound data to the internet from North America (assumed) | \$0.20 |
| **Total** | **\$43.67** |The app and database run in the same region, so data transfer between them is free. The app also uses a shared IPv4 address, which is free.In this example, Managed Postgres accounts for most of the bill. The app Machine and its 10GB volume cost \$5.19 per month.### A Discord bot or background worker running 24/7| Item | Monthly cost |
| - | -: |
| One `shared-cpu-1x` 256MB Machine, running continuously | \$2.19 |
| 1GB outbound data to the internet from North America (assumed) | \$0.02 |
| **Total** | **\$2.21** |This example doesn't use a volume, database, or dedicated IPv4 address, so there's nothing else to pay for. If the worker needs more memory, a `shared-cpu-1x` Machine with 512MB costs \$3.69 per month when running continuously.### An app running in three regionsSuppose an app runs one 1GB Machine each in Ashburn (`iad`), Frankfurt (`fra`), and Singapore (`sin`). Each Machine serves 50GB of traffic to users and sends 10GB of private network traffic to Machines in other regions during the month.| Item | Monthly cost |
| - | -: |
| One `shared-cpu-1x` 1GB Machine in Ashburn (`iad`) | \$6.70 |
| One `shared-cpu-1x` 1GB Machine in Frankfurt (`fra`) | \$7.73 |
| One `shared-cpu-1x` 1GB Machine in Singapore (`sin`) | \$8.50 |
| 100GB outbound data from `iad` and `fra` at \$0.02/GB (assumed) | \$2.00 |
| 50GB outbound data from `sin` at \$0.04/GB (assumed) | \$2.00 |
| 20GB cross-region private network data from `iad` and `fra` at \$0.006/GB (assumed) | \$0.12 |
| 10GB cross-region private network data from `sin` at \$0.015/GB (assumed) | \$0.15 |
| **Total** | **\$27.20** |Machine prices vary by region, which accounts for the different compute costs above.Cross-region private network traffic is billed in both directions. The request is billed at the rate for the region it leaves, and the response at the rate for the region it leaves. So if a small request returns a large response, most of the cost is at the responding region's rate.This example uses granular data transfer pricing. Organizations created before July 18, 2024 that haven't switched to granular pricing pay the public internet rate for cross-region traffic instead. See [data transfer pricing](#data-transfer-pricing).## Is there a free tier?No. New organizations don't have a free tier or a monthly free usage allowance.New accounts do get a [free trial](/about/free-trial), which lasts for up to 2 hours of Machine runtime or 7 days, whichever comes first. You don't need a credit card to start the trial. After the trial ends, you'll need to add a credit card to keep using billable resources.The cheapest always-on app is a `shared-cpu-1x` Machine with 256MB of RAM in Ashburn (`iad`) or Secaucus (`ewr`). It costs \$2.19 per 30 days of continuous runtime, plus any outbound data transfer (\$0.02/GB in North America and Europe).Some resources and allowances are free even after the trial ends. These include the first 10GB of volume snapshot storage each month, the first 10 single-hostname SSL certificates, inbound data transfer, shared IPv4 addresses, and IPv6 addresses.Organizations on plans discontinued in October 2024 keep the free allowances included with those plans. See [Discontinued Plans](/about/discontinued-plans).## How it worksFly.io services are billed per organization, with [Linked Organizations](/about/billing#unified-billing) reporting resource usage to their parent Billing Organization. Plans get complicated, so we just charge based on usage. Pick and choose which pieces you need for your application; that's all you'll see on your invoice.Organizations are administrative entities on Fly.io that let you add members, share app development environments, and manage billing. [Billing](/about/billing) is based on the resources provisioned for your apps, pro-rated for the time they are provisioned.Organizations may be subject to automated scaling limits to prevent abuse or to help with capacity planning. Email the address in the error message if you run into such a limit and it's getting in your way.All organizations (except for Linked Organizations) require a [credit card](/about/billing#payment-options) on file.## ComputeWe charge for started and stopped Machines differently. For more details about how costs are calculated, see [Machine billing](/about/billing#machine-billing). To understand the difference between `performance` and `shared` CPU types in Machines, see [CPU performance](/machines/cpu-performance).### Started Fly Machines
Region:
The price of a running [Fly Machine](/machines) VM is the price of a named CPU/RAM preset, plus about
per 30 days per GB of additional RAM.
Here’s the pricing for named presets and a few standard additional RAM configurations:
\Preset
\CPU(s)
\RAM
\Price/second
\Price/hour
\Price/month
### Stopped Fly MachinesFor stopped Machines we charge only for the root file system (rootfs) needed for each Machine. Each 1GB of rootfs for a Machine stopped for 30 days is \$0.15. The amount of rootfs needed is defined by your OCI image generated on your app plus a few [containerd](https://containerd.io/) tweaks on the underlying file system.### Machine reservation blocksYou'll get a 40% discount when you reserve a block of compute time, either for `performance` Machines or `shared` Machines, in a specific region.
Reservations apply to any number of Machines of the specified CPU class, in the specified region, in any of your organization's apps.The available reservation sizes are:**Performance Machines**- \$144/year for \$20/month of usage
- \$1,440/year for \$200/month of usage
- \$14,400/year for \$2,000/month of usage**Shared Machines**- \$36/year for \$5/month of usage
- \$360/year for \$50/month of usage
- \$3,600/year for \$500/month of usageYou pay the "per year" amount upfront, and each month receive a credit worth the "per month" amount. The credit does not rollover; it's only valid for the month in which it's granted. The credit applies only to CPU and additional RAM charges.There's no limit on the number or combinations of blocks that can be purchased. Reservations are backdated to the first day of the month in which they're purchased.For example, if you purchase a \$36/year `shared` Machines block in `cdg`, you'll pay \$36 upfront and receive \$5/month of credits applicable to `shared` Machines in `cdg` for 12 months, starting with the month of the purchase. Amortised over 12 months, the \$36 upfront cost is \$3/month, which is a 40% discount on the \$5/month of credits you receive.You can set up reservations via self-service in the billing section of your Fly.io [dashboard](https://fly.io/dashboard). They apply to usage starting on the 1st, so setting up reservations any time in the month will give you the credits the entire month.## Managed PostgresThe price of running Fly.io Managed Postgres depends on your selected Managed Postgres Plan and the amount of storage your databases use.Current pricing for Managed Postgres plans and storage is available [here](/postgres#pricing).> [!WARNING]
>
> **Important:** Managed Postgres lives outside your apps. Deleting an app won’t delete its database. Have a look in your Dashboard when you're cleaning up. A quick check can save you a surprise charge later.## Persistent Storage Volumes### Volumes[Fly Volumes](/volumes) are local persistent storage for Machines.- \$0.15/GB per month of provisioned capacity[Volume billing](/about/billing#volume-billing) is pro-rated to the hour.You'll be charged for volumes that you create, whether they are attached to a Machine or not, including when an attached Machine is stopped.### Volume Snapshots- \$0.08/GB per month
- First 10GB free each month[Volume Snapshot billing](/about/billing#volume-snapshot-billing) is pro-rated to the hour.Automatic daily snapshots with 5 days retention are enabled by default on new volumes. This can be [adjusted](/volumes/snapshots#set-or-change-the-snapshot-retention-period) or [disabled](/volumes/snapshots#disable-automatic-daily-snapshots).Usage is calculated based on the total stored size of the snapshots, not the provisioned volume size. You're only charged for the actual data stored - if you've written 1GB to a 10GB volume, you'll be charged for around 1GB of snapshot storage.Snapshots for each volume are stored incrementally, so you'll only be charged for data that has changed since the previously stored snapshot.You can check your snapshot usage in the **Billing** section of the [dashboard](https://fly.io/dashboard/personal/billing), on your Upcoming Invoice and in the Cost Explorer.## Network prices### Anycast IP addressesEach application receives a [shared IPv4 address](/networking/services#shared-ipv4) and unlimited [Anycast IPv6](/networking/services#ipv6) addresses for global load balancing.Dedicated IPv4 addresses are \$2/mo.### Managed SSL certificatesWe use Let's Encrypt to issue certificates, and donate half of our SSL fees to them at the end of each calendar year.- Single hostname certificates: \$0.10/mo
- Wildcard certificates: \$1/moEvery organization's first 10 single hostname certificates are free.### Data transfer pricingWe bill for data leaving your app destined for the public internet or for apps or Machines in other regions, including:- Data egress to the Internet, from Machine to edge server to Internet
- Data transfer over private network between regions, from Machine to edge server and edge server to Machine
- Data transfer to some extensions like Upstash RedisThe following types of traffic are free:- All inbound data transfer
- Data transfer between apps or Machines in the same region (for organizations using granular data transfer rates)
- Data transfer from apps without an assigned IP address (for organizations not using granular data transfer rates)Fly.io pricing is per region group for outbound data transfer. You'll see a more detailed breakdown of cost per region and per traffic type on your monthly invoice.> [!NOTE]
>
> **Important:** Organizations created after July 18 2024 are automatically opted-in to use the granular data transfer rates and are billed at a different rate for private network data transfer between regions, per the following table. Organizations not using granular data transfer rates are billed for all data transfer (excluding that listed as free above) at the "Egress to public internet" rate.| Region groups | Egress to public internet cost | Private network cross-region transfer cost |
| - | - | - |
| - North America - Europe | \$0.02 per GB | \$0.006 per GB |
| - Asia Pacific - Oceania - South America | \$0.04 per GB | \$0.015 per GB |
| - Africa - India | \$0.12 per GB | \$0.050 per GB |To opt-in to granular bandwidth pricing, go to the [**Organizations** page](https://fly.io/organizations) in the dashboard, click the organization name to change, then click **Switch to granular bandwidth pricing**. You won't be able to return to using the non-granular data transfer rates once you opt in.### Static Egress IPs for MachinesStatic egress IPs for Machines provide dedicated outbound IP addresses for your Machines. When you allocate a static egress IP, you'll get both an IPv4 and IPv6 address for this single price.- \$0.005 per hour (\~\$3.60/month)
- Machines do not have a static IP by default## Support[Community support](https://community.fly.io/) is included for all customers, regardless of usage level.
You can get access to a support plan by purchasing a Standard (\$29/month), Premium (\$199/month), or Enterprise (starting at \$2500/month) package in the **Support** section of your dashboard. For more about Support, see [Support at Fly.io](/about/support).## Fly Kubernetes[Fly Kubernetes](/kubernetes) (FKS) is a managed Kubernetes service that runs on Fly.io.- \$75/month per cluster
- Plus the cost of [compute](#compute) and [Fly volumes](#persistent-storage-volumes) that you create## ExtensionsFly.io offers managed services operated by third parties, such as [Tigris Object Storage](/tigris) and [Upstash Redis](/upstash/redis).When you provision their services, you become their customer, and you pay their list prices via your monthly Fly.io bill. Charges are updated daily in your Fly.io dashboard.You will not be billed separately for:- Machines running the services, which are hosted in the provider's account
- IP addresses associated with the serviceYou **will** be billed separately for data transfer to these external third-party services, including Tigris Object Storage. See our [data transfer pricing](#data-transfer-pricing) for details.## Unsupported Products### Unmanaged Fly Postgres (Unsupported)[Fly Postgres](/unmanaged-postgres) is a PostgreSQL database that you create and then manage yourself. Fly Postgres clusters are Fly Apps that consist of Machines, volumes, and any configured extra memory.The [Machine price](#compute) and [volume price](#persistent-storage-volumes) for Fly Postgres are the same as any other Machine and volume you'd run on Fly.io. Assuming the Machines are running all the time, the cost for the preset configurations is about \$2/month for a single node cluster for dev projects and from about \$82 to \$164/month for a three-node production cluster. You don't need to keep the preset configurations, you can [scale your Fly Postgres Machines](/unmanaged-postgres/managing/scaling) to suit your workload at any time.## Legacy plansOn a legacy plan and wondering what that includes? Read more about [Discontinued Plans](/about/discontinued-plans).## **Related reading**- [Billing for Fly.io](/about/billing) How invoicing, payment methods, and usage tracking work.
- [Cost Management](/about/cost-management) Best practices for estimating, monitoring, and controlling your spend.
- [Free Trial](/about/free-trial) What the Fly.io Free trial gives you and when billing begins.
- [Organization Roles & Permissions](/security/org-roles-permissions) How org structure, permissions and billing interplay.
- [Optimize Compute Costs: Fine‑tune your app](/apps/fine-tune-apps) Tuning machine size, memory/CPU, and stop‑/start behavior to reduce waste.
# Support
Source: https://docs.fly.io/about/support
Get help with Fly.io through the community forum, billing email, paid email support and Support Portal, and see what to include in a ticket and what's in scope.
Need support? We can help! Here's where to look.
## Fly.io status
For platform-wide issues and updates, please view our [status page](https://status.flyio.net/).
For host-level issues that might affect your apps, see your [dashboard](https://fly.io/dashboard).
## Community forum
**Who can use this:** Everyone
**Use this for:** General questions, troubleshooting advice, best practices, and sharing tips with other Fly.io users.
All customers have access to our active [community forum](https://community.fly.io). We're in the forums regularly to help out with customer issues and post new feature announcements.
Customers also frequently help each other out and share insights on running their apps on Fly.io. Especially helpful contributors can earn a special "Aeronaut" badge to signify their troubleshooting prowess, great advice, and exceptional kindness.
## Billing and account support
**Who can use this:** Everyone
**Use this for:** Billing and account management issues.
For questions about a specific invoice or account management issues, you can email us at [billing@fly.io](mailto:billing@fly.io).
## Paid support plans
**Find out more:** Want to learn more about our different support plans or ready to get started? Just head over to our [Support page](https://fly.io/support). We've got all the details there to help you choose what works best for you!
### Email support
**Who can use this:** Organizations who have purchased a Standard, Premium, or Enterprise Support package or organizations with legacy Launch or Scale plans.
**Use this for:** Technical support for issues or questions specific to you.
Our Standard, Premium, and Enterprise packages have access to email support. These plans come with an organization-specific address for emailing support questions, typically `@support.fly.io`.
You can find your support address is in the [Fly.io dashboard](https://fly.io/dashboard). Select your organization, and look for your unique email address in the **Support** section. You'll also find a form to submit new support tickets, as well as an overview of recent support interactions.
### Support Portal
**Who can use this:** Organizations who have purchased a Standard, Premium, or Enterprise Support package or organizations with legacy Launch or Scale plans. Organizations with a Managed Postgres (MPG) database cluster also have access to our Support Portal for issues related to MPG.
**Use this for:** Technical support for issues or questions specific to you.
The Support Portal is a self-service portal for customers to submit support tickets, view recent support interactions, and track the status of their tickets. You can access it from your [Fly.io dashboard](https://fly.io/dashboard) by clicking the **Support** tab.
## Managed Postgres Support
We now offer [Managed Postgres (MPG)](/postgres/index), a fully managed Postgres service with automated provisioning, daily snapshots, built-in high availability, global networking, and Prometheus-compatible metrics. MPG is currently available in a limited set of regions and supports up to 1 TB of storage, with an initial allocation cap of 500 GB. Managed Postgres customers have Support portal access for issues related to MPG.
Note: If you need to run in unsupported regions, require more storage, or need custom configuration, you may still need to use an [unmanaged Postgres](/unmanaged-postgres) app for now.
## Support metrics
The Fly.io Support team publishes our email support metrics to better set expectations regarding response times and help to you understand how we're doing. You can view these metrics in the public [Support Metrics Dashboard](https://fly.io/support).
## Working with support
We understand that describing technical issues can sometimes be challenging. Providing a clear and detailed description of the problem, including any noticeable symptoms and relevant artifacts, can help us identify similar cases and potentially speed up the resolution process.
Here are some things to include in your ticket:
* Your app name and/or Machine ID.
* Specific errors in your app logs or browser. What you observed (e.g. 'it threw a 500 error').
* What you were trying to achieve (e.g. deploy the application).
* What you did (e.g. I ran `fly deploy`).
* UTC timestamps for when the issue occurred.
* Prepending `LOG_LEVEL=debug` to any flyctl command will provide insight into the lower-level API calls being made.
* If you are sending us error messages or log output, it's best to send that as plain text in the body of the email (or upload a `.txt` or `.log` file for longer outputs), rather than attached screenshots of your terminal window.
## Scope of Support
This section highlights the products we actively work on and maintain, and therefore, the ones we are able to provide support for. While we can't offer direct support for your own code or the specific language or framework you use, here are some [guides to the most common frameworks](/getting-started/get-started-by-framework) used on Fly.io.
> **Supported Products**
>
> * **Networking**
> * **Machines**
> * **Managed Postgres** (MPG)
> * **Apps**
> * **Launch/Deploy** (UI & CLI)
> * **Volumes**
> * **FKS**
> * **Security**
> * **Accounts & Billing**
> * **Extensions** (Tigris, Upstash, Depot)
> * **Monitoring** (metrics and logs)
> **Not Supported**
>
> * **LiteFS**
> * **Unmanaged Postgres**
# Assign new IP address to app
Source: https://docs.fly.io/api/machines/apps/assign-new-ip-address-to-app
/api/machines/openapi.json post /v1/apps/{app_name}/ip_assignments
# Create App
Source: https://docs.fly.io/api/machines/apps/create-app
/api/machines/openapi.json post /v1/apps
Create an app with the specified details in the request body.
# Create App deploy token
Source: https://docs.fly.io/api/machines/apps/create-app-deploy-token
/api/machines/openapi.json post /v1/apps/{app_name}/deploy_token
# Destroy App
Source: https://docs.fly.io/api/machines/apps/destroy-app
/api/machines/openapi.json delete /v1/apps/{app_name}
Delete an app by its name.
# Get App
Source: https://docs.fly.io/api/machines/apps/get-app
/api/machines/openapi.json get /v1/apps/{app_name}
Retrieve details about a specific app by its name.
# List Apps
Source: https://docs.fly.io/api/machines/apps/list-apps
/api/machines/openapi.json get /v1/apps
List all apps with the ability to filter by organization slug.
# List IP assignments for app
Source: https://docs.fly.io/api/machines/apps/list-ip-assignments-for-app
/api/machines/openapi.json get /v1/apps/{app_name}/ip_assignments
# Remove IP assignment from app
Source: https://docs.fly.io/api/machines/apps/remove-ip-assignment-from-app
/api/machines/openapi.json delete /v1/apps/{app_name}/ip_assignments/{ip}
# Machines API overview
Source: https://docs.fly.io/api/machines/index
Base URL, Bearer token auth, and OpenAPI spec for the Fly.io Machines API, with links to Apps, Machines, Volumes, Secrets, and Certificates endpoints.
This site hosts the API reference generated from the Fly.io Machines API OpenAPI
specification. For how to get started, deeper explanations of each endpoint, and
examples, visit the complete [Machines API docs](/machines/api/).
Download the [OpenAPI spec](https://docs.fly.io/api/machines/openapi.json) to generate a client or explore the API in another tool.
## Base URL
All requests are made against:
```
https://api.machines.dev
```
## Authentication
Authenticate every request with a Fly.io access token using the
`Authorization: Bearer ` header. Generate one with
[`fly tokens create`](/flyctl/cmd/fly_tokens_create) or the
[Tokens](https://fly.io/tokens) page on the dashboard.
```bash theme={null}
curl https://api.machines.dev/v1/apps?org_slug=personal \
-H "Authorization: Bearer $FLY_API_TOKEN"
```
## Resources
> **[Apps](/api/machines/apps/list-apps)**
>
> Group your Machines into apps.
> **[Machines](/api/machines/machines/list-machines)**
>
> Launch and control fast-booting VMs.
> **[Volumes](/api/machines/volumes/list-volumes)**
>
> Persistent storage for your Machines.
> **[Postgres Clusters](/api/machines/postgres-clusters/list-postgres-clusters)**
>
> Create and manage Postgres clusters.
> **[Secrets](/api/machines/secrets/list-secret-keys-belonging-to-an-app)**
>
> App secrets and encryption keys.
> **[TLS Certificates](/api/machines/tls-certificates/list-certificates-for-app)**
>
> Certificates for custom hostnames.
> **[Tokens](/api/machines/tokens/authenticate-token-header)**
>
> Authenticate and mint access tokens.
# Cordon Machine
Source: https://docs.fly.io/api/machines/machines/cordon-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/cordon
“Cordoning” a Machine refers to disabling its services, so the Fly Proxy won’t route requests to it. In flyctl this is used by blue/green deployments; one set of Machines is started up with services disabled, and when they are all healthy, the services are enabled on the new Machines and disabled on the old ones.
# Create Lease
Source: https://docs.fly.io/api/machines/machines/create-lease
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/lease
Create a lease for a specific Machine within an app using the details provided in the request body. Machine leases can be used to obtain an exclusive lock on modifying a Machine.
# Create Machine
Source: https://docs.fly.io/api/machines/machines/create-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines
Create a Machine within a specific app using the details provided in the request body.
**Important**: This request can fail, and you’re responsible for handling that failure. If you ask for a large Machine, or a Machine in a region we happen to be at capacity for, you might need to retry the request, or to fall back to another region. If you’re working directly with the Machines API, you’re taking some responsibility for your own orchestration!
# Delete Metadata
Source: https://docs.fly.io/api/machines/machines/delete-metadata
/api/machines/openapi.json delete /v1/apps/{app_name}/machines/{machine_id}/metadata/{key}
Delete metadata for a specific Machine within an app by providing a metadata key.
# Destroy Machine
Source: https://docs.fly.io/api/machines/machines/destroy-machine
/api/machines/openapi.json delete /v1/apps/{app_name}/machines/{machine_id}
Delete a specific Machine within an app by Machine ID.
Only a Machine in the `stopped`, `suspended`, `failed`, `pending` or `archived` state can be deleted. Deleting a Machine in any other state, such as `started`, fails with `412 Precondition Failed` unless you set `force=true`, which kills the Machine first. Deleting a Machine that is already `destroying` or `destroyed` succeeds without doing anything.
If someone else holds a lease on the Machine, the request fails with `409 Conflict`. Pass the lease nonce in the `fly-machine-lease-nonce` header to delete a Machine you hold the lease on.
# Execute Command
Source: https://docs.fly.io/api/machines/machines/execute-command
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/exec
Execute a command on a specific Machine and return the raw command output bytes.
# Get Lease
Source: https://docs.fly.io/api/machines/machines/get-lease
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/lease
Retrieve the current lease of a specific Machine within an app. Machine leases can be used to obtain an exclusive lock on modifying a Machine.
# Get Machine
Source: https://docs.fly.io/api/machines/machines/get-machine
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}
Get details of a specific Machine within an app by the Machine ID.
# Get Machine Memory
Source: https://docs.fly.io/api/machines/machines/get-machine-memory
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/memory
Get current memory limit and available capacity for a machine
# Get Metadata
Source: https://docs.fly.io/api/machines/machines/get-metadata
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/metadata
Retrieve metadata for a specific Machine within an app.
# Get Metadata Value
Source: https://docs.fly.io/api/machines/machines/get-metadata-value
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/metadata/{key}
Get the value of a specific metadata key
# List Events
Source: https://docs.fly.io/api/machines/machines/list-events
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/events
List all events associated with a specific Machine within an app.
# List Machines
Source: https://docs.fly.io/api/machines/machines/list-machines
/api/machines/openapi.json get /v1/apps/{app_name}/machines
List all Machines associated with a specific app, with optional filters for including deleted Machines and filtering by region.
# List Processes
Source: https://docs.fly.io/api/machines/machines/list-processes
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/ps
List all processes running on a specific Machine within an app, with optional sorting parameters.
# List Versions
Source: https://docs.fly.io/api/machines/machines/list-versions
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/versions
List all versions of the configuration for a specific Machine within an app.
# Reclaim Machine Memory
Source: https://docs.fly.io/api/machines/machines/reclaim-machine-memory
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/memory/reclaim
Trigger the balloon device to reclaim memory from a machine
# Release Lease
Source: https://docs.fly.io/api/machines/machines/release-lease
/api/machines/openapi.json delete /v1/apps/{app_name}/machines/{machine_id}/lease
Release the lease of a specific Machine within an app. Machine leases can be used to obtain an exclusive lock on modifying a Machine.
# Restart Machine
Source: https://docs.fly.io/api/machines/machines/restart-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/restart
Restart a specific Machine within an app, with an optional timeout parameter.
# Set Machine Memory Limit
Source: https://docs.fly.io/api/machines/machines/set-machine-memory-limit
/api/machines/openapi.json put /v1/apps/{app_name}/machines/{machine_id}/memory
Set the memory limit for a machine using the balloon device
# Signal Machine
Source: https://docs.fly.io/api/machines/machines/signal-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/signal
Send a signal to a specific Machine within an app using the details provided in the request body.
# Start Machine
Source: https://docs.fly.io/api/machines/machines/start-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/start
Start a specific Machine within an app.
# Stop Machine
Source: https://docs.fly.io/api/machines/machines/stop-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/stop
Stop a specific Machine within an app, with an optional request body to specify signal and timeout.
# Suspend Machine
Source: https://docs.fly.io/api/machines/machines/suspend-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/suspend
Suspend a specific Machine within an app. The next start operation will attempt (but is not guaranteed) to resume the Machine from a snapshot taken at suspension time, rather than performing a cold boot.
# Uncordon Machine
Source: https://docs.fly.io/api/machines/machines/uncordon-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/uncordon
“Cordoning” a Machine refers to disabling its services, so the Fly Proxy won’t route requests to it. In flyctl this is used by blue/green deployments; one set of Machines is started up with services disabled, and when they are all healthy, the services are enabled on the new Machines and disabled on the old ones.
# Update Machine
Source: https://docs.fly.io/api/machines/machines/update-machine
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}
Update a Machine's configuration using the details provided in the request body.
# Update Metadata (deprecated PUT alias of PATCH)
Source: https://docs.fly.io/api/machines/machines/update-metadata-deprecated-put-alias-of-patch
/api/machines/openapi.json put /v1/apps/{app_name}/machines/{machine_id}/metadata
Deprecated: use `PATCH` on the same path, which this is identical to.
+ Despite the method, this does not replace the metadata: keys that are not in the request are left untouched. Null values and empty strings remove keys.
+ If `machine_version` is provided and no longer matches the current machine version, returns 412 Precondition Failed.
# Update Metadata (set/remove multiple keys)
Source: https://docs.fly.io/api/machines/machines/update-metadata-setremove-multiple-keys
/api/machines/openapi.json patch /v1/apps/{app_name}/machines/{machine_id}/metadata
Update multiple metadata keys at once. Keys that are not in the request are left untouched. Null values and empty strings remove keys.
+ If `machine_version` is provided and no longer matches the current machine version, returns 412 Precondition Failed.
# Upsert Metadata Key
Source: https://docs.fly.io/api/machines/machines/upsert-metadata-key
/api/machines/openapi.json post /v1/apps/{app_name}/machines/{machine_id}/metadata/{key}
Update metadata for a specific machine within an app by providing a metadata key.
# Wait for State
Source: https://docs.fly.io/api/machines/machines/wait-for-state
/api/machines/openapi.json get /v1/apps/{app_name}/machines/{machine_id}/wait
Wait for a Machine to reach a specific state. Specify the desired state with the state parameter. See the [Machine states table](https://fly.io/docs/machines/working-with-machines/#machine-states) for a list of possible states. The default for this parameter is `started`.
This request will block for up to 60 seconds. Set a shorter timeout with the timeout parameter.
# Models
Source: https://docs.fly.io/api/machines/models
Schema reference for every object in the Fly Machines API.
These are the data models used across the Machines API requests and responses, generated from the OpenAPI specification.
## AcmeChallenge
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | |
| `target` | `string` | No | |
## App
| Property | Type | Required | Description |
| - | - | - | - |
| `id` | `string` | No | |
| `internal_numeric_id` | `integer` | No | |
| `machine_count` | `integer` | No | |
| `name` | `string` | No | |
| `network` | `string` | No | |
| `network_cidr` | `string` | No | |
| `organization` | `AppOrganizationInfo` | No | |
| `status` | `string` | No | |
| `volume_count` | `integer` | No | |
## AppOrganizationInfo
| Property | Type | Required | Description |
| - | - | - | - |
| `internal_numeric_id` | `integer` | No | |
| `name` | `string` | No | |
| `slug` | `string` | No | |
## AppSecret
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `digest` | `string` | No | |
| `name` | `string` | No | |
| `updated_at` | `string` | No | |
| `value` | `string` | No | |
## AppSecrets
| Property | Type | Required | Description |
| - | - | - | - |
| `secrets` | `AppSecret[]` | No | |
## AppSecretsUpdateRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `values` | `map` | No | |
## AppSecretsUpdateResp
| Property | Type | Required | Description |
| - | - | - | - |
| `Version` | `integer` | No | DEPRECATED |
| `secrets` | `AppSecret[]` | No | |
| `version` | `integer` | No | The app's secrets version after this change. Pass it as `min_secrets_version` when creating or updating Machines so that they are guaranteed to see the change. |
## assignIPRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `network` | `string` | No | |
| `org_slug` | `string` | No | |
| `region` | `string` | No | |
| `service_name` | `string` | No | |
| `type` | `IPAssignmentType` | No | |
## AssignIPResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `egress` | `boolean` | No | |
| `ip` | `string` | No | |
| `ip_pair` | `IPPair` | No | ip\_pair is returned when "egress-pair" IP type is requested; in this case, ip is null. |
| `network` | `IPAssignmentNetwork` | No | |
| `region` | `string` | No | |
| `service_name` | `string` | No | |
| `shared` | `boolean` | No | |
## authenticateTokenRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `header` | `string` | No | |
## authorizeResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `access` | `flyio.Access` | No | |
| `verified_token` | `root.VerifiedToken` | No | |
## authorizeTokenRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `access` | `main.TokenAccess` | No | |
| `header` | `string` | No | |
## CertificateCheckResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `acme_requested` | `boolean` | No | |
| `certificates` | `CertificateEntry[]` | No | |
| `configured` | `boolean` | No | |
| `dns_provider` | `string` | No | |
| `dns_records` | `DNSRecords` | No | |
| `dns_requirements` | `DNSRequirements` | No | |
| `hostname` | `string` | No | |
| `rate_limited_until` | `string` | No | |
| `status` | `string` | No | |
| `validation` | `CertificateValidation` | No | |
| `validation_errors` | `CertificateValidationError[]` | No | |
## CertificateDetail
| Property | Type | Required | Description |
| - | - | - | - |
| `acme_requested` | `boolean` | No | |
| `certificates` | `CertificateEntry[]` | No | |
| `configured` | `boolean` | No | |
| `dns_provider` | `string` | No | |
| `dns_requirements` | `DNSRequirements` | No | |
| `hostname` | `string` | No | |
| `rate_limited_until` | `string` | No | |
| `status` | `string` | No | |
| `validation` | `CertificateValidation` | No | |
| `validation_errors` | `CertificateValidationError[]` | No | |
## CertificateEntry
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `expires_at` | `string` | No | |
| `issued` | `IssuedCertificate[]` | No | |
| `issuer` | `string` | No | |
| `source` | `string` | No | Enum: custom, fly |
| `status` | `string` | No | Enum: active, pending\_ownership, pending\_validation |
## CertificateSummary
| Property | Type | Required | Description |
| - | - | - | - |
| `acme_alpn_configured` | `boolean` | No | |
| `acme_dns_configured` | `boolean` | No | |
| `acme_http_configured` | `boolean` | No | |
| `acme_requested` | `boolean` | No | |
| `configured` | `boolean` | No | |
| `created_at` | `string` | No | |
| `dns_provider` | `string` | No | |
| `has_custom_certificate` | `boolean` | No | |
| `has_fly_certificate` | `boolean` | No | |
| `hostname` | `string` | No | |
| `ownership_txt_configured` | `boolean` | No | |
| `status` | `string` | No | |
| `updated_at` | `string` | No | |
## CertificateValidation
| Property | Type | Required | Description |
| - | - | - | - |
| `alpn_configured` | `boolean` | No | |
| `dns_configured` | `boolean` | No | |
| `http_configured` | `boolean` | No | |
| `ownership_txt_configured` | `boolean` | No | |
## CertificateValidationError
| Property | Type | Required | Description |
| - | - | - | - |
| `code` | `string` | No | |
| `message` | `string` | No | |
| `remediation` | `string` | No | |
| `timestamp` | `string` | No | |
## CheckStatus
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | |
| `output` | `string` | No | |
| `status` | `string` | No | |
| `updated_at` | `string` | No | |
## createAcmeCertificateRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `hostname` | `string` | No | |
## CreateAppDeployTokenRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `expiry` | `string` | No | |
## CreateAppDeployTokenResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `token` | `string` | No | |
## CreateAppRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `enable_subdomains` | `boolean` | No | |
| `idempotency_key` | `string` | No | When set, makes a retry of this exact request safe: a second create with the same key and the same name (or no name) returns the app the first request created, instead of erroring on a false name conflict or creating a duplicate. A second create with the same key but a *different* name is rejected outright; it won't silently hand back the first app. |
| `name` | `string` | No | |
| `network` | `string` | No | |
| `org_slug` | `string` | No | |
## CreateAppResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `integer` | No | |
| `id` | `string` | No | |
## createCustomCertificateRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `fullchain` | `string` | No | |
| `hostname` | `string` | No | |
| `private_key` | `string` | No | |
## CreateLeaseRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `description` | `string` | No | |
| `ttl` | `integer` | No | seconds lease will be valid |
## CreateMachineRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `config` | `fly.MachineConfig` | No | An object defining the Machine configuration |
| `lease_ttl` | `integer` | No | |
| `min_secrets_version` | `integer` | No | The minimum version of the app's secrets the Machine must run with. Set it to the `version` returned by the endpoints that set or delete app secrets, so that the Machine is guaranteed to see those changes. This field belongs at the top level of the request body, not in `config`. If omitted, the app's current secrets version is looked up for you. |
| `name` | `string` | No | Unique name for this Machine. If omitted, one is generated for you |
| `region` | `string` | No | The target region. Omitting this param launches in the same region as your WireGuard peer connection (somewhere near you). |
| `skip_launch` | `boolean` | No | |
| `skip_secrets` | `boolean` | No | |
| `skip_service_registration` | `boolean` | No | |
## CreateOIDCTokenRequest
Optional parameters
| Property | Type | Required | Description |
| - | - | - | - |
| `aud` | `string` | No | |
| `aws_principal_tags` | `boolean` | No | |
## createPostgresAttachmentRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `app_name` | `string` | Yes | Name of the Fly app to attach. |
## createPostgresAttachmentResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresAttachment` | No | |
## createPostgresBackupRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `type` | `string` | Yes | Backup type. Enum: full, incr, diff |
## createPostgresClusterRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `disk_size_gb` | `integer` | No | Disk size in gigabytes. |
| `name` | `string` | No | Name for the cluster. A name is generated when omitted. |
| `org_slug` | `string` | Yes | Slug of the organization that will own the cluster, or "personal" for the caller's personal organization. |
| `pg_major_version` | `string` | No | Postgres major version. Enum: 16, 17 |
| `plan` | `string` | Yes | Plan slug selecting CPU, memory, and disk sizing. Matched case-insensitively. Enum: basic, starter, launch, scale, Performance |
| `pool_mode` | `string` | No | Connection pooler mode. Enum: session, transaction |
| `postgis_enabled` | `boolean` | No | Enable PostGIS support, required to later enable PostGIS extensions. |
| `region` | `string` | Yes | Fly region code where the cluster's primary runs. |
## createPostgresDatabaseRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | Yes | Name of the database to create. Must start and end with an alphanumeric character and may contain hyphens and underscores, up to 63 characters. |
## createPostgresUserRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `role` | `string` | Yes | Role to grant the user. Enum: schema\_admin, writer, reader |
| `username` | `string` | Yes | Name for the new user. Must start and end with a lowercase alphanumeric character and may contain hyphens and underscores, up to 63 characters. |
## createPostgresUserResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresUser` | No | |
## CreateVolumeRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `auto_backup_enabled` | `boolean` | No | enable scheduled automatic snapshots. Defaults to `true` |
| `compute` | `fly.MachineGuest` | No | |
| `compute_image` | `string` | No | |
| `encrypted` | `boolean` | No | |
| `fstype` | `string` | No | |
| `name` | `string` | No | |
| `region` | `string` | No | |
| `require_unique_zone` | `boolean` | No | |
| `size_gb` | `integer` | No | |
| `snapshot_id` | `string` | No | restore from snapshot |
| `snapshot_retention` | `integer` | No | |
| `source_volume_id` | `string` | No | fork from remote volume |
| `unique_zone_app_wide` | `boolean` | No | |
## CurrentTokenResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `tokens` | `main.tokenInfo[]` | No | |
## DecryptSecretkeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `associated_data` | `integer[]` | No | |
| `ciphertext` | `integer[]` | No | |
## DecryptSecretkeyResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `plaintext` | `integer[]` | No | |
## DeleteAppSecretResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `Version` | `integer` | No | DEPRECATED |
| `version` | `integer` | No | The app's secrets version after this change. Pass it as `min_secrets_version` when creating or updating Machines so that they are guaranteed to see the change. |
## DeleteSecretkeyResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `Version` | `integer` | No | DEPRECATED |
| `version` | `integer` | No | |
## destroyCustomCertificateResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `acme_requested` | `boolean` | No | |
| `certificates` | `CertificateEntry[]` | No | |
| `configured` | `boolean` | No | |
| `dns_provider` | `string` | No | |
| `dns_requirements` | `DNSRequirements` | No | |
| `hostname` | `string` | No | |
| `rate_limited_until` | `string` | No | |
| `status` | `string` | No | |
| `validation` | `CertificateValidation` | No | |
| `validation_errors` | `CertificateValidationError[]` | No | |
| `warning` | `string` | No | |
## DNSRecords
| Property | Type | Required | Description |
| - | - | - | - |
| `a` | `string[]` | No | |
| `aaaa` | `string[]` | No | |
| `acme_challenge_cname` | `string` | No | |
| `cname` | `string[]` | No | |
| `ownership_txt` | `string` | No | |
| `resolved_addresses` | `string[]` | No | |
| `soa` | `string` | No | |
## DNSRequirements
| Property | Type | Required | Description |
| - | - | - | - |
| `a` | `string[]` | No | |
| `aaaa` | `string[]` | No | |
| `acme_challenge` | `AcmeChallenge` | No | |
| `cname` | `string` | No | |
| `ownership` | `OwnershipVerification` | No | |
## enablePostgresExtensionRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `create_schema` | `boolean` | No | Create the schema first if it does not exist. |
| `name` | `string` | Yes | Extension to enable. |
| `schema` | `string` | No | Schema to install the extension into. Defaults to the database's default schema. |
## EncryptSecretkeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `associated_data` | `integer[]` | No | |
| `plaintext` | `integer[]` | No | |
## EncryptSecretkeyResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `ciphertext` | `integer[]` | No | |
## ErrorResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `details` | `object` | No | Deprecated |
| `error` | `string` | No | |
| `status` | `main.statusCode` | No | |
## ExtendVolumeRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `size_gb` | `integer` | No | |
## ExtendVolumeResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `needs_restart` | `boolean` | No | |
| `volume` | `Volume` | No | |
## fly.ContainerConfig
| Property | Type | Required | Description |
| - | - | - | - |
| `cmd` | `string[]` | No | CmdOverride is used to override the default command of the image. |
| `depends_on` | `fly.ContainerDependency[]` | No | DependsOn can be used to define dependencies between containers. The container will only be started after all of its dependent conditions have been satisfied. |
| `entrypoint` | `string[]` | No | EntrypointOverride is used to override the default entrypoint of the image. |
| `env` | `map` | No | ExtraEnv is used to add additional environment variables to the container. |
| `env_from` | `fly.EnvFrom[]` | No | EnvFrom can be provided to set environment variables from machine fields. |
| `exec` | `string[]` | No | Image Config overrides - these fields are used to override the image configuration. If not provided, the image configuration will be used. ExecOverride is used to override the default command of the image. |
| `files` | `fly.File[]` | No | Files are files that will be written to the container file system. |
| `healthchecks` | `fly.ContainerHealthcheck[]` | No | Healthchecks determine the health of your containers. Healthchecks can use HTTP, TCP or an Exec command. |
| `image` | `string` | No | Image is the docker image to run. |
| `name` | `string` | No | Name is used to identify the container in the machine. |
| `restart` | `fly.MachineRestart` | No | Restart is used to define the restart policy for the container. |
| `secrets` | `fly.MachineSecret[]` | No | Secrets can be provided at the process level to explicitly indicate which secrets should be used for the process. If not provided, the secrets provided at the machine level will be used. |
| `stop` | `fly.StopConfig` | No | Stop is used to define the signal and timeout for stopping the container. |
| `user` | `string` | No | UserOverride is used to override the default user of the image. |
## fly.ContainerDependency
| Property | Type | Required | Description |
| - | - | - | - |
| `condition` | `fly.ContainerDependencyCondition` | No | |
| `name` | `string` | No | |
## fly.ContainerDependencyCondition
Type: `string`
Enum: exited\_successfully, healthy, started
## fly.ContainerHealthcheck
| Property | Type | Required | Description |
| - | - | - | - |
| `exec` | `fly.ExecHealthcheck` | No | |
| `failure_threshold` | `integer` | No | The number of times the check must fail before considering the container unhealthy. |
| `grace_period` | `integer` | No | The time in seconds to wait after a container starts before checking its health. |
| `http` | `fly.HTTPHealthcheck` | No | |
| `interval` | `integer` | No | The time in seconds between executing the defined check. |
| `kind` | `fly.ContainerHealthcheckKind` | No | Kind of healthcheck (readiness, liveness) |
| `name` | `string` | No | The name of the check. Must be unique within the container. |
| `success_threshold` | `integer` | No | The number of times the check must succeeed before considering the container healthy. |
| `tcp` | `fly.TCPHealthcheck` | No | |
| `timeout` | `integer` | No | The time in seconds to wait for the check to complete. |
| `unhealthy` | `fly.UnhealthyPolicy` | No | Unhealthy policy that determines what action to take if a container is deemed unhealthy |
## fly.ContainerHealthcheckKind
Type: `string`
Enum: readiness, liveness
## fly.ContainerHealthcheckScheme
Type: `string`
Enum: http, https
## fly.DNSConfig
| Property | Type | Required | Description |
| - | - | - | - |
| `dns_forward_rules` | `fly.dnsForwardRule[]` | No | |
| `hostname` | `string` | No | |
| `hostname_fqdn` | `string` | No | |
| `nameservers` | `string[]` | No | |
| `options` | `fly.dnsOption[]` | No | |
| `searches` | `string[]` | No | |
| `skip_registration` | `boolean` | No | |
## fly.dnsForwardRule
| Property | Type | Required | Description |
| - | - | - | - |
| `addr` | `string` | No | |
| `basename` | `string` | No | |
## fly.dnsOption
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | |
| `value` | `string` | No | |
## fly.EnvFrom
EnvVar defines an environment variable to be populated from a machine field, env\_var
| Property | Type | Required | Description |
| - | - | - | - |
| `env_var` | `string` | No | EnvVar is required and is the name of the environment variable that will be set from the secret. It must be a valid environment variable name. |
| `field_ref` | `string` | No | FieldRef selects a field of the Machine: supports id, version, app\_name, private\_ip, region, image. Enum: id, version, app\_name, private\_ip, region, image |
## fly.ExecHealthcheck
| Property | Type | Required | Description |
| - | - | - | - |
| `command` | `string[]` | No | The command to run to check the health of the container (e.g. \["cat", "/tmp/healthy"]) |
## fly.File
A file that will be written to the Machine. One of RawValue or SecretName must be set.
| Property | Type | Required | Description |
| - | - | - | - |
| `guest_path` | `string` | No | GuestPath is the path on the machine where the file will be written and must be an absolute path. For example: /full/path/to/file.json |
| `image_config` | `string` | No | The name of an image to use the OCI image config as the file contents. |
| `mode` | `integer` | No | Mode bits used to set permissions on this file as accepted by chmod(2). |
| `raw_value` | `string` | No | The base64 encoded string of the file contents. |
| `secret_name` | `string` | No | The name of the secret that contains the base64 encoded file contents. |
## fly.HTTPHealthcheck
| Property | Type | Required | Description |
| - | - | - | - |
| `headers` | `fly.MachineHTTPHeader[]` | No | Additional headers to send with the request |
| `method` | `string` | No | The HTTP method to use to when making the request |
| `path` | `string` | No | The path to send the request to |
| `port` | `integer` | No | The port to connect to, often the same as internal\_port |
| `scheme` | `fly.ContainerHealthcheckScheme` | No | Whether to use http or https |
| `tls_server_name` | `string` | No | If the protocol is https, the hostname to use for TLS certificate validation |
| `tls_skip_verify` | `boolean` | No | If the protocol is https, whether or not to verify the TLS certificate |
## fly.HTTPOptions
| Property | Type | Required | Description |
| - | - | - | - |
| `compress` | `boolean` | No | |
| `h2_backend` | `boolean` | No | |
| `headers_read_timeout` | `integer` | No | |
| `idle_timeout` | `integer` | No | |
| `replay_cache` | `fly.ReplayCache[]` | No | |
| `response` | `fly.HTTPResponseOptions` | No | |
## fly.HTTPResponseOptions
| Property | Type | Required | Description |
| - | - | - | - |
| `headers` | `map` | No | |
| `pristine` | `boolean` | No | |
## fly.MachineCacheDrive
| Property | Type | Required | Description |
| - | - | - | - |
| `size_mb` | `integer` | No | |
## fly.MachineCheck
| Property | Type | Required | Description |
| - | - | - | - |
| `grace_period` | `string` | No | The time to wait after a VM starts before checking its health |
| `headers` | `fly.MachineHTTPHeader[]` | No | |
| `interval` | `string` | No | The time between connectivity checks |
| `kind` | `string` | No | Kind of the check (informational, readiness) Enum: informational, readiness |
| `method` | `string` | No | For http checks, the HTTP method to use to when making the request |
| `path` | `string` | No | For http checks, the path to send the request to |
| `port` | `integer` | No | The port to connect to, often the same as internal\_port |
| `protocol` | `string` | No | For http checks, whether to use http or https |
| `timeout` | `string` | No | The maximum time a connection can take before being reported as failing its health check |
| `tls_server_name` | `string` | No | If the protocol is https, the hostname to use for TLS certificate validation |
| `tls_skip_verify` | `boolean` | No | For http checks with https protocol, whether or not to verify the TLS certificate |
| `type` | `string` | No | tcp or http |
## fly.MachineConfig
| Property | Type | Required | Description |
| - | - | - | - |
| `auto_destroy` | `boolean` | No | Optional boolean telling the Machine to destroy itself once it’s complete (default false) |
| `cache_drive` | `fly.MachineCacheDrive` | No | A cache drive attached to the Machine |
| `checks` | `map` | No | An optional object that defines one or more named top-level checks. The key for each check is the check name. |
| `containers` | `fly.ContainerConfig[]` | No | Containers are a list of containers that will run in the machine, each from its own image. |
| `disable_machine_autostart` | `boolean` | No | Deprecated: use Service.Autostart instead |
| `dns` | `fly.DNSConfig` | No | DNS settings for the Machine |
| `env` | `map` | No | An object filled with key/value pairs to be set as environment variables |
| `files` | `fly.File[]` | No | Files to write into the Machine, from either a raw value or an app secret |
| `guest` | `fly.MachineGuest` | No | The resources (CPUs, memory, etc.) allocated to the Machine |
| `image` | `string` | No | The docker image to run |
| `init` | `fly.MachineInit` | No | Arguments for init, Fly.io's footprint inside the Machine, which controls how your own code is run |
| `metadata` | `map` | No | Key/value pairs of Machine metadata. Fly.io uses metadata internally for routing, process groups, and clusters |
| `metrics` | `fly.MachineMetrics` | No | An optional metrics endpoint for Prometheus on Fly.io to scrape |
| `mounts` | `fly.MachineMount[]` | No | Persistent volumes to mount in the Machine. A Machine can mount at most one volume, which must be on the same host. The mounted volume cannot be changed by updating the Machine |
| `processes` | `fly.MachineProcess[]` | No | Multiple processes to run within this one Machine, all from the Machine's image. Unrelated to the process groups in fly.toml, which run on separate Machines. To run processes from different images, use containers instead |
| `restart` | `fly.MachineRestart` | No | Whether and how the Machine is restarted after its main process exits |
| `rootfs` | `fly.MachineRootfs` | No | The size of the Machine's root filesystem and whether it persists |
| `schedule` | `string` | No | Run the Machine at an interval: one of hourly, daily, weekly or monthly. The interval starts when the Machine is created |
| `services` | `fly.MachineService[]` | No | Network services that Fly Proxy routes traffic to |
| `size` | `string` | No | Deprecated: use Guest instead |
| `spot` | `fly.MachineSpot` | No | Spot pricing settings for the Machine |
| `standbys` | `string[]` | No | Standbys enable a machine to be a standby for another. In the event of a hardware failure, the standby machine will be started. |
| `statics` | `fly.Static[]` | No | Static files to serve under a URL prefix. Fly Proxy routes matching requests to a file server inside the Machine, bypassing your application's own process; the Machine must be running. The files are read either from a path inside the Machine or from a Tigris bucket that belongs to the app; no bucket is created for you |
| `stop_config` | `fly.StopConfig` | No | How the Machine is shut down gracefully: the signal sent to it and how long to wait before a hard shutdown |
## fly.MachineGuest
| Property | Type | Required | Description |
| - | - | - | - |
| `cpu_kind` | `string` | No | |
| `cpus` | `integer` | No | |
| `gpu_kind` | `string` | No | |
| `gpus` | `integer` | No | |
| `host_dedication_id` | `string` | No | |
| `kernel_args` | `string[]` | No | |
| `max_memory_mb` | `integer` | No | |
| `memory_mb` | `integer` | No | |
| `persist_rootfs` | `string` | No | Deprecated: use MachineConfig.Rootfs instead Enum: never, always, restart |
| `required_host_features` | `string[]` | No | |
## fly.MachineHTTPHeader
For http checks, an array of objects with string field Name and array of strings field Values. The key/value pairs specify header and header values that will get passed with the check call.
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | The header name |
| `values` | `string[]` | No | The header value |
## fly.MachineInit
| Property | Type | Required | Description |
| - | - | - | - |
| `cmd` | `string[]` | No | |
| `entrypoint` | `string[]` | No | |
| `exec` | `string[]` | No | |
| `kernel_args` | `string[]` | No | |
| `swap_size_mb` | `integer` | No | |
| `tty` | `boolean` | No | |
## fly.MachineMetrics
| Property | Type | Required | Description |
| - | - | - | - |
| `https` | `boolean` | No | |
| `path` | `string` | No | |
| `port` | `integer` | No | |
## fly.MachineMount
| Property | Type | Required | Description |
| - | - | - | - |
| `add_size_gb` | `integer` | No | |
| `add_size_percent` | `integer` | No | AddSizePercent grows by a percentage of the current size, rounded up to GiB. It is mutually exclusive with AddSizeGb. |
| `encrypted` | `boolean` | No | |
| `extend_threshold_percent` | `integer` | No | |
| `min_add_size_gb` | `integer` | No | MinAddSizeGb is the minimum proportional growth in GiB (default 1). |
| `name` | `string` | No | |
| `path` | `string` | No | |
| `size_gb` | `integer` | No | |
| `size_gb_limit` | `integer` | No | |
| `volume` | `string` | No | |
## fly.MachinePort
| Property | Type | Required | Description |
| - | - | - | - |
| `end_port` | `integer` | No | |
| `force_https` | `boolean` | No | |
| `handlers` | `string[]` | No | |
| `http_options` | `fly.HTTPOptions` | No | |
| `port` | `integer` | No | |
| `proxy_proto_options` | `fly.ProxyProtoOptions` | No | |
| `start_port` | `integer` | No | |
| `tls_options` | `fly.TLSOptions` | No | |
## fly.MachineProcess
| Property | Type | Required | Description |
| - | - | - | - |
| `cmd` | `string[]` | No | |
| `entrypoint` | `string[]` | No | |
| `env` | `map` | No | |
| `env_from` | `fly.EnvFrom[]` | No | EnvFrom can be provided to set environment variables from machine fields. |
| `exec` | `string[]` | No | |
| `ignore_app_secrets` | `boolean` | No | IgnoreAppSecrets can be set to true to ignore the secrets for the App the Machine belongs to and only use the secrets provided at the process level. The default/legacy behavior is to use the secrets provided at the App level. |
| `secrets` | `fly.MachineSecret[]` | No | Secrets can be provided at the process level to explicitly indicate which secrets should be used for the process. If not provided, the secrets provided at the machine level will be used. |
| `user` | `string` | No | |
## fly.MachineRestart
The Machine restart policy defines whether and how flyd restarts a Machine after its main process exits. See [https://fly.io/docs/machines/guides-examples/machine-restart-policy/](https://fly.io/docs/machines/guides-examples/machine-restart-policy/).
| Property | Type | Required | Description |
| - | - | - | - |
| `max_retries` | `integer` | No | When policy is on-failure, the maximum number of times to attempt to restart the Machine before letting it stop. |
| `policy` | `string` | No | \* no - Never try to restart a Machine automatically when its main process exits, whether that’s on purpose or on a crash. \* always - Always restart a Machine automatically and never let it enter a stopped state, even when the main process exits cleanly. \* on-failure - Try up to MaxRetries times to automatically restart the Machine if it exits with a non-zero exit code. Default when no explicit policy is set, and for Machines with schedules. Enum: no, always, on-failure |
## fly.MachineRootfs
| Property | Type | Required | Description |
| - | - | - | - |
| `persist` | `string` | No | Enum: never, always, restart |
| `size_gb` | `integer` | No | |
## fly.MachineSecret
A Secret needing to be set in the environment of the Machine. env\_var is required
| Property | Type | Required | Description |
| - | - | - | - |
| `env_var` | `string` | No | EnvVar is required and is the name of the environment variable that will be set from the secret. It must be a valid environment variable name. |
| `name` | `string` | No | Name is optional and when provided is used to reference a secret name where the EnvVar is different from what was set as the secret name. |
## fly.MachineService
| Property | Type | Required | Description |
| - | - | - | - |
| `autostart` | `boolean` | No | |
| `autostop` | `string` | No | Accepts a string (new format) or a boolean (old format). For backward compatibility with older clients, the API continues to use booleans for "off" and "stop" in responses. \* "off" or false - Do not autostop the Machine. \* "stop" or true - Automatically stop the Machine. \* "suspend" - Automatically suspend the Machine, falling back to a full stop if this is not possible. Enum: off, stop, suspend |
| `checks` | `fly.MachineServiceCheck[]` | No | An optional list of service checks |
| `concurrency` | `fly.MachineServiceConcurrency` | No | |
| `force_instance_description` | `string` | No | |
| `force_instance_key` | `string` | No | |
| `internal_port` | `integer` | No | |
| `min_machines_running` | `integer` | No | |
| `ports` | `fly.MachinePort[]` | No | |
| `protocol` | `string` | No | |
## fly.MachineServiceCheck
| Property | Type | Required | Description |
| - | - | - | - |
| `grace_period` | `string` | No | The time to wait after a VM starts before checking its health |
| `headers` | `fly.MachineHTTPHeader[]` | No | |
| `interval` | `string` | No | The time between connectivity checks |
| `method` | `string` | No | For http checks, the HTTP method to use to when making the request |
| `path` | `string` | No | For http checks, the path to send the request to |
| `port` | `integer` | No | The port to connect to, often the same as internal\_port |
| `protocol` | `string` | No | For http checks, whether to use http or https |
| `timeout` | `string` | No | The maximum time a connection can take before being reported as failing its health check |
| `tls_server_name` | `string` | No | If the protocol is https, the hostname to use for TLS certificate validation |
| `tls_skip_verify` | `boolean` | No | For http checks with https protocol, whether or not to verify the TLS certificate |
| `type` | `string` | No | tcp or http |
## fly.MachineServiceConcurrency
| Property | Type | Required | Description |
| - | - | - | - |
| `hard_limit` | `integer` | No | |
| `soft_limit` | `integer` | No | |
| `type` | `string` | No | |
## fly.MachineSpot
| Property | Type | Required | Description |
| - | - | - | - |
| `max_price_fraction` | `number` | No | MaxPriceFraction is the maximum fraction of the full Machine price you will pay for this Machine. Range: (0, 1.0] |
## fly.ProxyProtoOptions
| Property | Type | Required | Description |
| - | - | - | - |
| `fly_src` | `boolean` | No | |
| `version` | `string` | No | |
## fly.ReplayCache
| Property | Type | Required | Description |
| - | - | - | - |
| `allow_bypass` | `boolean` | No | |
| `name` | `string` | No | Name of the cookie or header to key the cache on |
| `path_prefix` | `string` | No | |
| `ttl_seconds` | `integer` | No | |
| `type` | `string` | No | Currently either "cookie" or "header" Enum: cookie, header |
## fly.Static
| Property | Type | Required | Description |
| - | - | - | - |
| `guest_path` | `string` | Yes | The path inside the Machine, or inside tigris\_bucket when set, where the files to serve are located |
| `index_document` | `string` | No | The document served for requests to the URL prefix or any of its subfolders. Only works with tigris\_bucket |
| `tigris_bucket` | `string` | No | An existing Tigris bucket belonging to the app to read the files from, instead of the Machine's filesystem. The files are still served through the Machine |
| `url_prefix` | `string` | Yes | The URL prefix under which the files are served |
## fly.StopConfig
| Property | Type | Required | Description |
| - | - | - | - |
| `signal` | `string` | No | Enum: SIGHUP, SIGINT, SIGQUIT, SIGKILL, SIGUSR1, SIGUSR2, SIGTERM |
| `timeout` | `string` | No | |
## fly.TCPHealthcheck
| Property | Type | Required | Description |
| - | - | - | - |
| `port` | `integer` | No | The port to connect to, often the same as internal\_port |
## fly.TLSOptions
| Property | Type | Required | Description |
| - | - | - | - |
| `alpn` | `string[]` | No | |
| `default_self_signed` | `boolean` | No | |
| `versions` | `string[]` | No | |
## fly.UnhealthyPolicy
Type: `string`
Enum: stop
## flydv1.ExecResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `exit_code` | `integer` | No | |
| `exit_signal` | `integer` | No | |
| `stderr` | `string` | No | |
| `stdout` | `string` | No | |
## flyio.Access
| Property | Type | Required | Description |
| - | - | - | - |
| `action` | `resset.Action` | No | |
| `app_feature` | `string` | No | |
| `appid` | `integer` | No | |
| `cluster` | `string` | No | |
| `command` | `string[]` | No | |
| `feature` | `string` | No | |
| `machine` | `string` | No | |
| `machine_feature` | `string` | No | |
| `mutation` | `string` | No | |
| `orgid` | `integer` | No | |
| `sourceApp` | `string` | No | |
| `sourceMachine` | `string` | No | |
| `sourceOrganization` | `string` | No | |
| `storage_object` | `string` | No | |
| `volume` | `string` | No | |
## forkPostgresClusterRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | Name for the forked cluster. Defaults to the source name with a -fork suffix. |
| `plan` | `string` | No | Plan slug selecting CPU and memory sizing for the fork. Defaults to the source cluster's plan. Matched case-insensitively. Enum: basic, starter, launch, scale, Performance |
## getPostgresUserCredentialsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresUserCredentials` | No | |
## ImageRef
| Property | Type | Required | Description |
| - | - | - | - |
| `digest` | `string` | No | |
| `labels` | `map` | No | |
| `registry` | `string` | No | |
| `repository` | `string` | No | |
| `tag` | `string` | No | |
## IPAssignment
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `egress` | `boolean` | No | |
| `ip` | `string` | No | |
| `network` | `IPAssignmentNetwork` | No | |
| `region` | `string` | No | |
| `service_name` | `string` | No | |
| `shared` | `boolean` | No | |
## IPAssignmentNetwork
The 6PN network a Flycast (private\_v6) address belongs to. Null for all other IP types.
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | Network name; empty for the organization's default network. |
| `org_slug` | `string` | No | |
## IPAssignmentType
Type of IP address to allocate. "egress-pair" allocates both v4 and v6 egress IP addresses (recommended when using egress IPs).
Type: `string`
Enum: v4, v6, shared\_v4, private\_v6, egress\_v4, egress\_v6, egress\_pair
## IPPair
| Property | Type | Required | Description |
| - | - | - | - |
| `v4` | `string` | No | |
| `v6` | `string` | No | |
## IssuedCertificate
| Property | Type | Required | Description |
| - | - | - | - |
| `certificate_authority` | `string` | No | |
| `expires_at` | `string` | No | |
| `type` | `string` | No | Enum: rsa, ecdsa |
## Lease
| Property | Type | Required | Description |
| - | - | - | - |
| `description` | `string` | No | Description or reason for the Lease. |
| `expires_at` | `integer` | No | ExpiresAt is the unix timestamp in UTC to denote when the Lease will no longer be valid. |
| `nonce` | `string` | No | Nonce is the unique ID autogenerated and associated with the Lease. |
| `owner` | `string` | No | Owner is the user identifier which acquired the Lease. |
| `version` | `string` | No | Machine version |
## ListAppsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `apps` | `App[]` | No | |
| `next_cursor` | `string` | No | Pagination cursor for the next page. Absent when no more apps remain. Cursors expire 30 minutes after the first page was requested. |
| `total_apps` | `integer` | No | The number of apps matching the request, across all pages. When paginating, it is counted once when the first page is requested and not updated afterwards, so it may differ slightly from the number of apps returned: it excludes apps created later and includes apps deleted while paginating. Apps created in the seconds before the first page may also be counted but missing from the pages. |
## listCertificatesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `certificates` | `CertificateSummary[]` | No | |
| `next_cursor` | `string` | No | |
| `total_count` | `integer` | No | |
## ListenSocket
| Property | Type | Required | Description |
| - | - | - | - |
| `address` | `string` | No | |
| `proto` | `string` | No | |
## listIPAssignmentsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `ips` | `IPAssignment[]` | No | |
## listPostgresActiveQueriesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresActiveQuery[]` | No | |
## listPostgresBackupsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresBackup[]` | No | |
## listPostgresClustersResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresClusterSummary[]` | No | |
## listPostgresDatabasesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresDatabase[]` | No | |
## listPostgresExtensionsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresExtension[]` | No | |
## listPostgresPlansResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresPlan[]` | No | |
## listPostgresSlowQueriesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresSlowQuery[]` | No | |
## listPostgresUsersResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresUser[]` | No | |
## macaroon.CaveatSet
| Property | Type | Required | Description |
| - | - | - | - |
| `caveats` | `object[]` | No | |
## macaroon.Nonce
| Property | Type | Required | Description |
| - | - | - | - |
| `kid` | `integer[]` | No | |
| `proof` | `boolean` | No | |
| `rnd` | `integer[]` | No | |
## Machine
| Property | Type | Required | Description |
| - | - | - | - |
| `checks` | `CheckStatus[]` | No | |
| `config` | `fly.MachineConfig` | No | |
| `cordoned` | `boolean` | No | |
| `created_at` | `string` | No | |
| `events` | `MachineEvent[]` | No | |
| `host_features` | `string[]` | No | |
| `host_status` | `string` | No | Enum: ok, unknown, unreachable |
| `id` | `string` | No | |
| `image_ref` | `ImageRef` | No | |
| `incomplete_config` | `fly.MachineConfig` | No | |
| `instance_id` | `string` | No | InstanceID is the same value as `version`. Deprecated: use `version`. |
| `lease` | `StrippedLease` | No | |
| `name` | `string` | No | |
| `nonce` | `string` | No | Nonce is only every returned on machine creation if a lease\_duration was provided. |
| `private_ip` | `string` | No | PrivateIP is the internal 6PN address of the machine. |
| `region` | `string` | No | |
| `state` | `string` | No | |
| `updated_at` | `string` | No | |
| `version` | `string` | No | Version is unique for each version of the machine. Pass it as `current_version` when updating the Machine to reject the update if the Machine has changed since this version. |
## MachineEvent
| Property | Type | Required | Description |
| - | - | - | - |
| `id` | `string` | No | |
| `request` | `object` | No | |
| `source` | `string` | No | |
| `status` | `string` | No | |
| `timestamp` | `integer` | No | |
| `type` | `string` | No | |
## MachineExecRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `cmd` | `string` | No | Deprecated: use Command instead |
| `command` | `string[]` | No | |
| `container` | `string` | No | |
| `machine` | `boolean` | No | Machine runs the command in the machine's own namespace instead of in a container. It is mutually exclusive with Container. |
| `stdin` | `string` | No | |
| `timeout` | `integer` | No | |
## MachineOverviewConfig
| Property | Type | Required | Description |
| - | - | - | - |
| `guest` | `fly.MachineGuest` | No | |
| `image` | `string` | No | |
| `metadata` | `map` | No | |
## MachineVersion
| Property | Type | Required | Description |
| - | - | - | - |
| `user_config` | `fly.MachineConfig` | No | |
| `version` | `string` | No | |
## main.getPlacementsRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `compute` | `fly.MachineGuest` | No | Resource requirements for the Machine to simulate. Defaults to a performance-1x machine |
| `count` | `integer` | No | Number of machines to simulate placement. Defaults to 0, which returns the org-specific limit for each region. |
| `org_slug` | `string` | Yes | |
| `region` | `string` | No | Region expression for placement as a comma-delimited set of regions or aliases. Defaults to "\[region],any", to prefer the API endpoint's local region with any other region as fallback. |
| `volume_name` | `string` | No | |
| `volume_size_bytes` | `integer` | No | |
| `weights` | `placement.Weights` | No | Optional weights to override default placement preferences. |
## main.getPlacementsResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `regions` | `placement.RegionPlacement[]` | No | |
## main.memoryResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `available_mb` | `integer` | No | |
| `limit_mb` | `integer` | No | |
## main.reclaimMemoryRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `amount_mb` | `integer` | No | |
## main.reclaimMemoryResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `actual_mb` | `integer` | No | |
## main.regionResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `nearest` | `string` | No | |
| `regions` | `main.regionRow[]` | No | |
## main.regionRow
| Property | Type | Required | Description |
| - | - | - | - |
| `code` | `string` | No | |
| `deprecated` | `boolean` | No | |
| `gateway_available` | `boolean` | No | |
| `geo_region` | `string` | No | |
| `latitude` | `number` | No | |
| `longitude` | `number` | No | |
| `mpg_available` | `boolean` | No | |
| `name` | `string` | No | |
| `requires_paid_plan` | `boolean` | No | |
## main.setMemoryLimitRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `limit_mb` | `integer` | No | |
## main.statusCode
Type: `string`
Enum: unknown, insufficient\_capacity, volume\_placement\_capacity, name\_taken
## main.TokenAccess
| Property | Type | Required | Description |
| - | - | - | - |
| `action` | `resset.Action` | No | Action is the action being taken on the specified resource. This is the combination of individual action characters (e.g "rw") - r: read - w: write - c: create - d: delete - C: control |
| `app_feature` | `string` | No | AppFeature is a named set of functionality associated with the app. If this is specified, the AppName field must be set. - images: images in the fly.io registry |
| `app_name` | `string` | No | AppName is the name of the app being accessed. |
| `command` | `string[]` | No | Command is the command being executed on a machine. If this is specified, the Machine must be set. |
| `machine_feature` | `string` | No | MachineFeature is a named set of functionality associated with the machine. If this is specified, the Machine field must be set. - metadata: machine metadata service - oidc: OIDC tokens - kmstoken: Petsem tokens for KMS access |
| `machine_id` | `string` | No | MachineID is the ID of the machine being accessed (e.g. 7811701f564258). |
| `mutation` | `string` | No | Mutation is the GraphQL mutation being performed. |
| `org_feature` | `string` | No | OrgFeature is a named set of functionality associated with the organization. If this is specified, the OrgSlug field must be set. - wg: WireGuard peers - builder: remote builders - addon: addons - membership: organization membership - billing: billing - litefs-cloud: LiteFS Cloud - authentication: authentication settings |
| `org_slug` | `string` | No | OrgSlug is the slug of the organization being accessed. |
| `source_machine` | `string` | No | SourceMachine is the machine ID of the actor attempting access. |
| `storage_object` | `string` | No | StorageObject is the storage object being accessed. If this is specified, the OrgSlug must be set. |
| `volume_id` | `string` | No | VolumeID is the encoded ID of the volume being accessed (e.g. vol\_r1p6pln1k9m9j7zr). |
## main.tokenInfo
| Property | Type | Required | Description |
| - | - | - | - |
| `apps` | `string[]` | No | |
| `org_slug` | `string` | No | |
| `organization` | `string` | No | |
| `restricted_to_machine` | `string` | No | Machine the token is restricted to (FromMachine caveat) |
| `source_machine_id` | `string` | No | Machine making the request |
| `token_id` | `string` | No | |
| `user` | `string` | No | User identifier if token is for a user |
## metadataValueResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `value` | `string` | No | |
## OkResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `ok` | `boolean` | No | |
## OrganizationRef
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | Organization name. |
| `slug` | `string` | No | Organization slug. |
## OrgMachine
| Property | Type | Required | Description |
| - | - | - | - |
| `app_name` | `string` | No | |
| `config` | `MachineOverviewConfig` | No | |
| `created_at` | `string` | No | |
| `id` | `string` | No | |
| `name` | `string` | No | |
| `private_ip` | `string` | No | |
| `region` | `string` | No | |
| `state` | `string` | No | |
| `updated_at` | `string` | No | |
| `version` | `string` | No | |
## OrgMachinesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `error_regions` | `string[]` | No | |
| `last_machine_id` | `string` | No | |
| `last_updated_at` | `string` | No | |
| `machines` | `OrgMachine[]` | No | |
| `next_cursor` | `string` | No | |
## OrgVolume
| Property | Type | Required | Description |
| - | - | - | - |
| `app_name` | `string` | No | |
| `attached_alloc_id` | `string` | No | |
| `attached_machine_id` | `string` | No | |
| `auto_backup_enabled` | `boolean` | No | |
| `block_size` | `integer` | No | |
| `blocks` | `integer` | No | |
| `blocks_avail` | `integer` | No | |
| `blocks_free` | `integer` | No | |
| `bytes_total` | `integer` | No | |
| `bytes_used` | `integer` | No | |
| `created_at` | `string` | No | |
| `encrypted` | `boolean` | No | |
| `fstype` | `string` | No | |
| `host_features` | `string[]` | No | |
| `host_status` | `string` | No | Enum: ok, unknown, unreachable |
| `id` | `string` | No | |
| `name` | `string` | No | |
| `region` | `string` | No | |
| `required_host_features` | `string[]` | No | |
| `size_gb` | `integer` | No | |
| `snapshot_retention` | `integer` | No | |
| `state` | `string` | No | |
| `type` | `string` | No | Enum: local, cache |
| `updated_at` | `string` | No | |
| `zone` | `string` | No | |
## OrgVolumesResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `last_updated_at` | `string` | No | |
| `last_volume_id` | `string` | No | |
| `next_cursor` | `string` | No | |
| `volumes` | `OrgVolume[]` | No | |
## OwnershipVerification
| Property | Type | Required | Description |
| - | - | - | - |
| `app_value` | `string` | No | |
| `name` | `string` | No | |
| `org_value` | `string` | No | |
## placement.RegionPlacement
| Property | Type | Required | Description |
| - | - | - | - |
| `concurrency` | `integer` | No | |
| `count` | `integer` | No | |
| `region` | `string` | No | |
## placement.Weights
Type: `map`
## PostgresActiveQuery
| Property | Type | Required | Description |
| - | - | - | - |
| `application_name` | `string` | No | Application connected to the backend. |
| `client` | `string` | No | Client address. |
| `database` | `string` | No | Database name. |
| `duration_seconds` | `number` | No | Query duration in seconds. |
| `pid` | `integer` | No | Process ID of the backend running the query. |
| `query` | `string` | No | Full query text. |
| `state` | `string` | No | Current backend state. |
| `state_duration_seconds` | `number` | No | Time spent in the current state, in seconds. |
| `user` | `string` | No | User running the query. |
| `wait_event` | `string` | No | Event the backend is waiting for. |
| `wait_event_type` | `string` | No | Type of event the backend is waiting for. |
## PostgresAttachedApp
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | Attached Fly app name. |
## PostgresAttachment
| Property | Type | Required | Description |
| - | - | - | - |
| `app_name` | `string` | No | Attached Fly app name. |
| `attached_at` | `string` | No | RFC 3339 timestamp when the attachment was created. |
| `postgres_cluster_id` | `string` | No | Cluster ID the app is attached to. |
## PostgresBackup
| Property | Type | Required | Description |
| - | - | - | - |
| `finished_at` | `string` | No | RFC 3339 stop timestamp. |
| `id` | `string` | No | Backup ID. |
| `size_bytes` | `integer` | No | Backup size in bytes. |
| `started_at` | `string` | No | RFC 3339 start timestamp. |
| `status` | `string` | No | Backup status. |
| `type` | `string` | No | Backup type. Enum: full, incr, diff |
## PostgresCluster
| Property | Type | Required | Description |
| - | - | - | - |
| `attached_apps` | `PostgresAttachedApp[]` | No | Apps attached to the cluster. |
| `cpu_kind` | `string` | No | CPU class. Enum: shared, performance |
| `cpus` | `integer` | No | vCPUs per node. |
| `created_at` | `string` | No | RFC 3339 creation timestamp. |
| `disk_size_gb` | `integer` | No | Disk size in gigabytes, for one replica. |
| `endpoints` | `PostgresClusterEndpoints` | No | Connection endpoints. Populated once the cluster is ready. |
| `id` | `string` | No | Cluster ID. |
| `memory_mb` | `integer` | No | Memory per node in megabytes. |
| `name` | `string` | No | Cluster name. |
| `organization` | `OrganizationRef` | No | |
| `pg_major_version` | `string` | No | Postgres major version. Enum: 16, 17 |
| `plan` | `string` | No | Plan slug. Enum: basic, starter, launch, scale, Performance |
| `postgis_enabled` | `boolean` | No | Whether PostGIS support is enabled. |
| `region` | `string` | No | Fly region code. |
| `replicas` | `integer` | No | Number of replicas. |
| `status` | `string` | No | Current lifecycle status. Enum: creating, initializing, ready, deleting, deleted, failed |
| `storage_provisioned_bytes` | `integer` | No | Storage provisioned, summed across the cluster's volumes and measured hourly. Null until first measured. |
| `storage_used_bytes` | `integer` | No | Storage used, summed across the cluster's volumes and measured hourly. Null until first measured. |
## PostgresClusterEndpoints
| Property | Type | Required | Description |
| - | - | - | - |
| `primary` | `PostgresNodeEndpoints` | No | Endpoints for connecting to the primary node. |
## PostgresClusterSummary
| Property | Type | Required | Description |
| - | - | - | - |
| `attached_apps` | `PostgresAttachedApp[]` | No | Apps attached to the cluster. |
| `created_at` | `string` | No | RFC 3339 creation timestamp. |
| `deleted_at` | `string` | No | RFC 3339 deletion timestamp; present for deleted clusters. |
| `id` | `string` | No | Cluster ID. |
| `name` | `string` | No | Cluster name. |
| `plan` | `string` | No | Plan slug. Enum: basic, starter, launch, scale, Performance |
| `region` | `string` | No | Fly region code. |
| `status` | `string` | No | Current lifecycle status. Enum: creating, initializing, ready, deleting, deleted, failed |
## PostgresDatabase
| Property | Type | Required | Description |
| - | - | - | - |
| `name` | `string` | No | Database name. |
## postgresDatabaseResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresDatabase` | No | |
## PostgresEndpoint
| Property | Type | Required | Description |
| - | - | - | - |
| `host` | `string` | No | Hostname to connect to. |
| `port` | `integer` | No | TCP port. |
## PostgresErrorResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `error` | `string` | No | Human-readable error message. |
| `status` | `string` | No | Optional machine-readable error status. |
## PostgresExtension
| Property | Type | Required | Description |
| - | - | - | - |
| `default_version` | `string` | No | Default version installed when enabled. |
| `description` | `string` | No | Human-readable description. |
| `installed` | `PostgresInstalledExtension` | No | Installation details, or null when the extension is not installed. |
| `name` | `string` | No | Extension name. |
| `system` | `boolean` | No | Whether this is a system extension that cannot be disabled. |
## PostgresInstalledExtension
| Property | Type | Required | Description |
| - | - | - | - |
| `schema` | `string` | No | Schema the extension is installed into. |
| `version` | `string` | No | Installed version. |
## PostgresNodeEndpoints
| Property | Type | Required | Description |
| - | - | - | - |
| `direct` | `PostgresEndpoint` | No | Unpooled connection to the cluster node. |
| `pooler` | `PostgresEndpoint` | No | Pooled connection to the cluster node (through PgBouncer). |
## PostgresPlan
| Property | Type | Required | Description |
| - | - | - | - |
| `cpu_kind` | `string` | No | CPU class. Enum: shared, performance |
| `cpus` | `integer` | No | vCPUs per node. |
| `memory_mb` | `integer` | No | Memory per node in megabytes. |
| `monthly_price_usd` | `integer` | No | Base monthly price in US dollars. |
| `plan` | `string` | No | Plan slug. Pass it as `plan` when creating or forking a cluster. Enum: basic, starter, launch, scale, Performance |
## PostgresSlowQuery
| Property | Type | Required | Description |
| - | - | - | - |
| `calls` | `integer` | No | Cumulative number of executions. |
| `database` | `string` | No | Database name. |
| `max_exec_time_seconds` | `number` | No | Maximum execution time in seconds. |
| `mean_exec_time_seconds` | `number` | No | Mean execution time in seconds. |
| `query` | `string` | No | Full query pattern. |
| `query_id` | `string` | No | Query identifier. |
| `total_exec_time_seconds` | `number` | No | Cumulative total execution time in seconds. |
| `user` | `string` | No | User running the query. |
## PostgresUser
| Property | Type | Required | Description |
| - | - | - | - |
| `role` | `string` | No | User role. Enum: schema\_admin, writer, reader |
| `username` | `string` | No | User name. |
## PostgresUserCredentials
| Property | Type | Required | Description |
| - | - | - | - |
| `password` | `string` | No | User password. |
| `username` | `string` | No | User name. |
## ProcessStat
| Property | Type | Required | Description |
| - | - | - | - |
| `command` | `string` | No | |
| `cpu` | `integer` | No | |
| `directory` | `string` | No | |
| `listen_sockets` | `ListenSocket[]` | No | |
| `pid` | `integer` | No | |
| `rss` | `integer` | No | |
| `rtime` | `integer` | No | |
| `stime` | `integer` | No | |
## resset.Action
Type: `integer`
Enum: 1, 2, 4, 8, 16, 31, 0
## restorePostgresClusterRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `backup_id` | `string` | No | Backup label to restore from, as returned by the backups list. Mutually exclusive with pitr\_time. |
| `name` | `string` | No | Name for the restored cluster. Defaults to a generated name derived from the source cluster and backup/point in time when omitted or blank. |
| `pitr_time` | `string` | No | Point in time to restore to, as an RFC3339 timestamp with an explicit offset from UTC (e.g. Z or +02:00). Normalized to UTC and must fall within the cluster's PITR recovery window. Mutually exclusive with backup\_id. |
## root.VerifiedToken
| Property | Type | Required | Description |
| - | - | - | - |
| `caveats` | `macaroon.CaveatSet` | No | |
| `header` | `string` | No | |
| `nonce` | `macaroon.Nonce` | No | |
| `permission_token` | `integer[]` | No | |
## rotatePostgresPasswordRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `kill_sessions` | `boolean` | No | Terminate the user's existing sessions after rotating. |
## rotatePostgresPasswordResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresUserCredentials` | No | |
## SecretKey
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `name` | `string` | No | |
| `public_key` | `integer[]` | No | |
| `type` | `string` | No | |
| `updated_at` | `string` | No | |
## SecretKeys
| Property | Type | Required | Description |
| - | - | - | - |
| `secret_keys` | `SecretKey[]` | No | |
## SetAppSecretRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `value` | `string` | No | |
## SetAppSecretResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `Version` | `integer` | No | DEPRECATED |
| `created_at` | `string` | No | |
| `digest` | `string` | No | |
| `name` | `string` | No | |
| `updated_at` | `string` | No | |
| `value` | `string` | No | |
| `version` | `integer` | No | The app's secrets version after this change. Pass it as `min_secrets_version` when creating or updating Machines so that they are guaranteed to see the change. |
## SetSecretkeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `type` | `string` | No | |
| `value` | `integer[]` | No | |
## SetSecretkeyResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `Version` | `integer` | No | DEPRECATED |
| `created_at` | `string` | No | |
| `name` | `string` | No | |
| `public_key` | `integer[]` | No | |
| `type` | `string` | No | |
| `updated_at` | `string` | No | |
| `version` | `integer` | No | |
## showPostgresClusterResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `data` | `PostgresCluster` | No | |
## SignalRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `signal` | `string` | No | Enum: SIGABRT, SIGALRM, SIGFPE, SIGHUP, SIGILL, SIGINT, SIGKILL, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM, SIGTRAP, SIGUSR1, SIGUSR2 |
## SignSecretkeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `plaintext` | `integer[]` | No | |
## SignSecretkeyResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `signature` | `integer[]` | No | |
## StartMachineResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `migrated` | `boolean` | No | Whether the Machine was moved to another host to be started |
| `new_host` | `string` | No | The host the Machine was moved to, if it was migrated |
| `previous_state` | `string` | No | The state the Machine was in before it was started |
## StopRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `signal` | `string` | No | Enum: SIGHUP, SIGINT, SIGQUIT, SIGKILL, SIGUSR1, SIGUSR2, SIGTERM |
| `timeout` | `string` | No | |
## StrippedLease
| Property | Type | Required | Description |
| - | - | - | - |
| `description` | `string` | No | |
| `expires_at` | `integer` | No | |
| `owner` | `string` | No | |
## UpdateMachineRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `config` | `fly.MachineConfig` | No | An object defining the Machine configuration |
| `current_version` | `string` | No | CurrentVersion is an optional optimistic-concurrency guard: the Machine's `version`, as returned by create, get or a previous update. When set and it is no longer the Machine's current version, the update is rejected with 409 Conflict and nothing changes. Omit it to update unconditionally. |
| `lease_ttl` | `integer` | No | |
| `min_secrets_version` | `integer` | No | The minimum version of the app's secrets the Machine must run with. Set it to the `version` returned by the endpoints that set or delete app secrets, so that the Machine is guaranteed to see those changes. This field belongs at the top level of the request body, not in `config`. If omitted, the app's current secrets version is looked up for you. |
| `name` | `string` | No | Unique name for this Machine. If omitted, one is generated for you |
| `region` | `string` | No | The target region. Omitting this param launches in the same region as your WireGuard peer connection (somewhere near you). |
| `skip_launch` | `boolean` | No | |
| `skip_secrets` | `boolean` | No | |
| `skip_service_registration` | `boolean` | No | |
## updateMetadataRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `machine_version` | `string` | No | |
| `metadata` | `map` | No | |
| `updated_at` | `string` | No | |
## updatePostgresUserRoleRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `role` | `string` | Yes | New role for the user. Enum: schema\_admin, writer, reader |
## UpdateVolumeRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `auto_backup_enabled` | `boolean` | No | |
| `snapshot_retention` | `integer` | No | |
## upsertMetadataKeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `updated_at` | `string` | No | |
| `value` | `string` | No | |
## VerifySecretkeyRequest
| Property | Type | Required | Description |
| - | - | - | - |
| `plaintext` | `integer[]` | No | |
| `signature` | `integer[]` | No | |
## Volume
| Property | Type | Required | Description |
| - | - | - | - |
| `attached_alloc_id` | `string` | No | |
| `attached_machine_id` | `string` | No | |
| `auto_backup_enabled` | `boolean` | No | |
| `block_size` | `integer` | No | |
| `blocks` | `integer` | No | |
| `blocks_avail` | `integer` | No | |
| `blocks_free` | `integer` | No | |
| `bytes_total` | `integer` | No | |
| `bytes_used` | `integer` | No | |
| `created_at` | `string` | No | |
| `encrypted` | `boolean` | No | |
| `fstype` | `string` | No | |
| `host_features` | `string[]` | No | |
| `host_status` | `string` | No | Enum: ok, unknown, unreachable |
| `id` | `string` | No | |
| `name` | `string` | No | |
| `region` | `string` | No | |
| `required_host_features` | `string[]` | No | |
| `size_gb` | `integer` | No | |
| `snapshot_retention` | `integer` | No | |
| `state` | `string` | No | |
| `type` | `string` | No | Enum: local, cache |
| `zone` | `string` | No | |
## VolumeSnapshot
| Property | Type | Required | Description |
| - | - | - | - |
| `created_at` | `string` | No | |
| `digest` | `string` | No | |
| `id` | `string` | No | |
| `retention_days` | `integer` | No | |
| `size` | `integer` | No | |
| `status` | `string` | No | |
| `volume_size` | `integer` | No | |
## WaitMachineResponse
| Property | Type | Required | Description |
| - | - | - | - |
| `event_id` | `string` | No | |
| `ok` | `boolean` | No | |
| `state` | `string` | No | |
| `version` | `string` | No | |
# List All Machines
Source: https://docs.fly.io/api/machines/organizations/list-all-machines
/api/machines/openapi.json get /v1/orgs/{org_slug}/machines
List all Machines associated with a specific organization. Machines are sorted by their `updated_at` timestamps, oldest to newest.
This API call represents "a point in time". Recent machine changes, including creations and destructions, may take time to propagate. When polling with `updated_after`, offset your timestamps to catch late-arriving events.
# List All Volumes
Source: https://docs.fly.io/api/machines/organizations/list-all-volumes
/api/machines/openapi.json get /v1/orgs/{org_slug}/volumes
List all volumes for an organization with optional filters and cursor-based pagination.
# Get Placements
Source: https://docs.fly.io/api/machines/platform/get-placements
/api/machines/openapi.json post /v1/platform/placements
Simulates placing the specified number of machines into regions, depending on available capacity and limits.
# Get Regions
Source: https://docs.fly.io/api/machines/platform/get-regions
/api/machines/openapi.json get /v1/platform/regions
List all regions on the platform with their details.
# Attach Cluster to App
Source: https://docs.fly.io/api/machines/postgres-clusters/attach-cluster-to-app
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/attachments
Record an attachment between a cluster and a Fly app. This endpoint records the relationship but does not set DATABASE_URL or other app secrets; configure the connection string separately. The attachment is created if it does not already exist.
# Create Backup
Source: https://docs.fly.io/api/machines/postgres-clusters/create-backup
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/backups
Create a backup for a specific cluster. The backup runs asynchronously and is rejected if one is already in progress.
# Create Database
Source: https://docs.fly.io/api/machines/postgres-clusters/create-database
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/databases
Create a database within a specific cluster.
# Create Postgres Cluster
Source: https://docs.fly.io/api/machines/postgres-clusters/create-postgres-cluster
/api/machines/openapi.json post /v1/postgres
Create a Managed Postgres cluster for the organization named in the request body. Provisioning is asynchronous: poll `GET /v1/postgres/{id}` until `status == ready` before calling database, user, extension, or credential endpoints.
# Create User
Source: https://docs.fly.io/api/machines/postgres-clusters/create-user
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/users
Create a Postgres user within a specific cluster. Fetch the generated password from the credentials endpoint.
# Delete Postgres Cluster
Source: https://docs.fly.io/api/machines/postgres-clusters/delete-postgres-cluster
/api/machines/openapi.json delete /v1/postgres/{postgres_cluster_id}
Delete a Managed Postgres cluster. The cluster is marked for deletion and removed asynchronously.
# Delete User
Source: https://docs.fly.io/api/machines/postgres-clusters/delete-user
/api/machines/openapi.json delete /v1/postgres/{postgres_cluster_id}/users/{username}
Delete a specific Postgres user within a cluster.
# Detach Cluster from App
Source: https://docs.fly.io/api/machines/postgres-clusters/detach-cluster-from-app
/api/machines/openapi.json delete /v1/postgres/{postgres_cluster_id}/attachments/{app_name}
Detach a specific cluster from a Fly app.
# Disable Extension
Source: https://docs.fly.io/api/machines/postgres-clusters/disable-extension
/api/machines/openapi.json delete /v1/postgres/{postgres_cluster_id}/databases/{database_name}/extensions/{extension_name}
Disable a specific Postgres extension within a database, optionally dropping objects that depend on it. System extensions cannot be disabled.
# Drop Database
Source: https://docs.fly.io/api/machines/postgres-clusters/drop-database
/api/machines/openapi.json delete /v1/postgres/{postgres_cluster_id}/databases/{database_name}
Delete a specific database within a cluster, along with objects that depend on it.
# Enable Extension
Source: https://docs.fly.io/api/machines/postgres-clusters/enable-extension
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/databases/{database_name}/extensions
Enable a Postgres extension within a specific database. PostGIS extensions require a PostGIS-enabled cluster.
# Fork Cluster to New Cluster
Source: https://docs.fly.io/api/machines/postgres-clusters/fork-cluster-to-new-cluster
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/fork
Fork a ready Managed Postgres cluster into a new cluster that inherits the source's settings, on the source's plan unless `plan` is given. The source cluster is left unchanged, and the fork is provisioned asynchronously.
# Get Postgres Cluster
Source: https://docs.fly.io/api/machines/postgres-clusters/get-postgres-cluster
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}
Get details of a specific Managed Postgres cluster.
# Get User Credentials
Source: https://docs.fly.io/api/machines/postgres-clusters/get-user-credentials
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/users/{username}/credentials
Get the connection credentials for a specific Postgres user within a cluster.
# List Active Queries
Source: https://docs.fly.io/api/machines/postgres-clusters/list-active-queries
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/queries/active
List active queries in the supplied database. The database query parameter is required and must not be empty or whitespace-only. Returns full query text and durations in seconds.
# List Backups
Source: https://docs.fly.io/api/machines/postgres-clusters/list-backups
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/backups
List all backups associated with a specific cluster.
# List Databases
Source: https://docs.fly.io/api/machines/postgres-clusters/list-databases
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/databases
List all databases within a specific cluster.
# List Extensions
Source: https://docs.fly.io/api/machines/postgres-clusters/list-extensions
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/databases/{database_name}/extensions
List all extensions available within a specific database, indicating which are installed.
# List Postgres Clusters
Source: https://docs.fly.io/api/machines/postgres-clusters/list-postgres-clusters
/api/machines/openapi.json get /v1/postgres
List Managed Postgres clusters for an organization.
# List Postgres Plans
Source: https://docs.fly.io/api/machines/postgres-clusters/list-postgres-plans
/api/machines/openapi.json get /v1/postgres/plans
List the Managed Postgres plans the caller can choose when creating or forking a cluster in an organization, cheapest first.
# List Slow Queries
Source: https://docs.fly.io/api/machines/postgres-clusters/list-slow-queries
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/queries/slow
List the top 20 queries by cumulative total execution time, excluding the postgres database. Uses the latest available metrics within the lookback range; statistics are cumulative PostgreSQL counters, not deltas over that range. Returns full query text and execution times in seconds.
# List Users
Source: https://docs.fly.io/api/machines/postgres-clusters/list-users
/api/machines/openapi.json get /v1/postgres/{postgres_cluster_id}/users
List all Postgres users within a specific cluster.
# Restore Backup or Point in Time to New Cluster
Source: https://docs.fly.io/api/machines/postgres-clusters/restore-backup-or-point-in-time-to-new-cluster
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/restore
Restore a backup, or a point in time within the cluster's PITR recovery window, into a new Managed Postgres cluster. The source cluster is left unchanged, and a new cluster is provisioned asynchronously.
# Rotate User Password
Source: https://docs.fly.io/api/machines/postgres-clusters/rotate-user-password
/api/machines/openapi.json post /v1/postgres/{postgres_cluster_id}/users/{username}/rotate_password
Rotate a specific Postgres user's password within a cluster, optionally terminating the user's existing sessions.
# Update User Role
Source: https://docs.fly.io/api/machines/postgres-clusters/update-user-role
/api/machines/openapi.json patch /v1/postgres/{postgres_cluster_id}/users/{username}
Update a specific Postgres user's role within a cluster.
# Create or update a secret key
Source: https://docs.fly.io/api/machines/secrets/create-or-update-a-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}
# Create or update Secret
Source: https://docs.fly.io/api/machines/secrets/create-or-update-secret
/api/machines/openapi.json post /v1/apps/{app_name}/secrets/{secret_name}
Running Machines don't pick up the change on their own: create or update Machines with `min_secrets_version` set to the returned `version` for them to see it.
# Decrypt with a secret key
Source: https://docs.fly.io/api/machines/secrets/decrypt-with-a-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}/decrypt
# Delete an app secret
Source: https://docs.fly.io/api/machines/secrets/delete-an-app-secret
/api/machines/openapi.json delete /v1/apps/{app_name}/secrets/{secret_name}
Running Machines don't pick up the change on their own: create or update Machines with `min_secrets_version` set to the returned `version` for them to see it.
# Delete an app's secret key
Source: https://docs.fly.io/api/machines/secrets/delete-an-apps-secret-key
/api/machines/openapi.json delete /v1/apps/{app_name}/secretkeys/{secret_name}
# Encrypt with a secret key
Source: https://docs.fly.io/api/machines/secrets/encrypt-with-a-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}/encrypt
# Generate a random secret key
Source: https://docs.fly.io/api/machines/secrets/generate-a-random-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}/generate
# Get an app secret
Source: https://docs.fly.io/api/machines/secrets/get-an-app-secret
/api/machines/openapi.json get /v1/apps/{app_name}/secrets/{secret_name}
# Get an app's secret key
Source: https://docs.fly.io/api/machines/secrets/get-an-apps-secret-key
/api/machines/openapi.json get /v1/apps/{app_name}/secretkeys/{secret_name}
# List app secrets belonging to an app
Source: https://docs.fly.io/api/machines/secrets/list-app-secrets-belonging-to-an-app
/api/machines/openapi.json get /v1/apps/{app_name}/secrets
# List secret keys belonging to an app
Source: https://docs.fly.io/api/machines/secrets/list-secret-keys-belonging-to-an-app
/api/machines/openapi.json get /v1/apps/{app_name}/secretkeys
# Sign with a secret key
Source: https://docs.fly.io/api/machines/secrets/sign-with-a-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}/sign
# Update app secrets belonging to an app
Source: https://docs.fly.io/api/machines/secrets/update-app-secrets-belonging-to-an-app
/api/machines/openapi.json post /v1/apps/{app_name}/secrets
Running Machines don't pick up the change on their own: create or update Machines with `min_secrets_version` set to the returned `version` for them to see it.
# Verify with a secret key
Source: https://docs.fly.io/api/machines/secrets/verify-with-a-secret-key
/api/machines/openapi.json post /v1/apps/{app_name}/secretkeys/{secret_name}/verify
# Check DNS and re-validate certificate
Source: https://docs.fly.io/api/machines/tls-certificates/check-dns-and-re-validate-certificate
/api/machines/openapi.json post /v1/apps/{app_name}/certificates/{hostname}/check
# Get certificate details
Source: https://docs.fly.io/api/machines/tls-certificates/get-certificate-details
/api/machines/openapi.json get /v1/apps/{app_name}/certificates/{hostname}
# List certificates for app
Source: https://docs.fly.io/api/machines/tls-certificates/list-certificates-for-app
/api/machines/openapi.json get /v1/apps/{app_name}/certificates
# Remove ACME certificates
Source: https://docs.fly.io/api/machines/tls-certificates/remove-acme-certificates
/api/machines/openapi.json delete /v1/apps/{app_name}/certificates/{hostname}/acme
# Remove certificate
Source: https://docs.fly.io/api/machines/tls-certificates/remove-certificate
/api/machines/openapi.json delete /v1/apps/{app_name}/certificates/{hostname}
# Remove custom certificate
Source: https://docs.fly.io/api/machines/tls-certificates/remove-custom-certificate
/api/machines/openapi.json delete /v1/apps/{app_name}/certificates/{hostname}/custom
# Request ACME certificate
Source: https://docs.fly.io/api/machines/tls-certificates/request-acme-certificate
/api/machines/openapi.json post /v1/apps/{app_name}/certificates/acme
# Upload custom certificate
Source: https://docs.fly.io/api/machines/tls-certificates/upload-custom-certificate
/api/machines/openapi.json post /v1/apps/{app_name}/certificates/custom
# Authenticate token header
Source: https://docs.fly.io/api/machines/tokens/authenticate-token-header
/api/machines/openapi.json post /v1/tokens/authenticate
Verify a token header without checking resource access.
# Authorize token for resource access
Source: https://docs.fly.io/api/machines/tokens/authorize-token-for-resource-access
/api/machines/openapi.json post /v1/tokens/authorize
Verify a token header and validate it against a requested access scope.
# Get Current Token Information
Source: https://docs.fly.io/api/machines/tokens/get-current-token-information
/api/machines/openapi.json get /v1/tokens/current
Get information about the current macaroon token(s), including organizations, apps, user identity hashes, and machine restrictions
# Request a Petsem token for accessing KMS
Source: https://docs.fly.io/api/machines/tokens/request-a-petsem-token-for-accessing-kms
/api/machines/openapi.json post /v1/tokens/kms
# Request an OIDC token
Source: https://docs.fly.io/api/machines/tokens/request-an-oidc-token
/api/machines/openapi.json post /v1/tokens/oidc
Request an Open ID Connect token for your machine. Customize the audience claim with the `aud` parameter. This returns a JWT token. Learn more about [using OpenID Connect](/docs/reference/openid-connect/) on Fly.io.
# Create Snapshot
Source: https://docs.fly.io/api/machines/volumes/create-snapshot
/api/machines/openapi.json post /v1/apps/{app_name}/volumes/{volume_id}/snapshots
Create a snapshot for a specific volume within an app.
# Create Volume
Source: https://docs.fly.io/api/machines/volumes/create-volume
/api/machines/openapi.json post /v1/apps/{app_name}/volumes
Create a volume for a specific app using the details provided in the request body.
# Destroy Volume
Source: https://docs.fly.io/api/machines/volumes/destroy-volume
/api/machines/openapi.json delete /v1/apps/{app_name}/volumes/{volume_id}
Delete a specific volume within an app by volume ID.
# Extend Volume
Source: https://docs.fly.io/api/machines/volumes/extend-volume
/api/machines/openapi.json put /v1/apps/{app_name}/volumes/{volume_id}/extend
Extend a volume's size within an app using the details provided in the request body.
# Get Volume
Source: https://docs.fly.io/api/machines/volumes/get-volume
/api/machines/openapi.json get /v1/apps/{app_name}/volumes/{volume_id}
Retrieve details about a specific volume by its ID within an app.
# List Snapshots
Source: https://docs.fly.io/api/machines/volumes/list-snapshots
/api/machines/openapi.json get /v1/apps/{app_name}/volumes/{volume_id}/snapshots
List all snapshots for a specific volume within an app.
# List Volumes
Source: https://docs.fly.io/api/machines/volumes/list-volumes
/api/machines/openapi.json get /v1/apps/{app_name}/volumes
List all volumes associated with a specific app.
# Update Volume
Source: https://docs.fly.io/api/machines/volumes/update-volume
/api/machines/openapi.json put /v1/apps/{app_name}/volumes/{volume_id}
Update a volume's configuration using the details provided in the request body.
# Chat: A 6PN Example with NATS
Source: https://docs.fly.io/app-guides/6pndemochat
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# 6pn-demo-chat
A chat server that uses Fly 6PN networking to connect to a NATS cluster enabling global chat.
## Rationale
This is the code for a very simple chat example that uses bare websockets in Node and a Browser to connect to it.
Instead of immediately broadcasting the message to all users, the server sends the message on to a NATS server, using 6PN private networking, which broadcasts the message to all interested parties.
## Preparation
As this example relies on a NATS server (or cluster) to be available, it should be deployed alongside the [`nats-cluster`](https://github.com/fly-examples/nats-cluster) example, within the same Fly organization.
Make a note of the app-name used to deploy the NATS cluster.
Select a name for your chat app.
## Initializing
Create an app on Fly with your chosen chat app name, using `fly init`:
```bash theme={null}
fly init YOURAPPNAME --import example-fly.toml
```
## Deploying
To deploy this app, we have one detail we have to pass it, the name fo the NATS cluster. The Dockerfile we use takes a build argument (NATSAPP) and turns it into an environment variable for the app to pick up when it runs. So if your NATS cluster app is called `myclusterapp` then running:
```bash theme={null}
fly deploy --build-arg NATSAPP=myclusterapp
```
Note: You can make that setting permanent by adding
```toml theme={null}
[build.args]
NATSAPP="myclusterapp"
```
to the fly.toml file.
Once deployed, open up the web page - `fly open` will get you there quickly - enter a chat handle and begin chatting.
## Notes
The more interesting parts of this app happen out of sight of the users. It's worth following the path of the NATSAPP build argument to find out more.
* NATSAPP's value is passed as a build argument to the Dockerfile at deployment time.
* Inside the Dockerfile, the build argument is converted into an Environment variable, `NATSAPPHOST` which is retained with the image's configuration.
* When the `server.js` app starts up, it picks up the `NATSAPPHOST` variable:
```
let natsapphost=process.env.NATSAPPHOST
```
* It then uses that value to connect to the NATS cluster:
```
const nc = NATS.connect(natsapphost+".internal");
```
The combination of an app name and `.internal` in 6PN networking resolves to the IPv6 addresses of the first instance of all instances if that app currently running. So, if we set NATSAPP to `myclusterapp` then scaling and adding nodes to `myclusterapp` would automatically be reflected in `myclusterapp.internal`. 6PN has a number of `.internal` addresses an app can query which you can read about in the [6PN documentation](/networking/private-networking#fly-io-internal-dns).
Once the chat server has connected to the NATS cluster, it subscribes to a topic `msg` and any message written to `msg` will be broadcast to all web socket clients that are connected. If any of those web socket clients sends a message to the chat server, then it will publish that message to the `msg` topic on the NATs cluster. It doesn't even try to echo it locally, relying on the NATS cluster to publish to subscribers.
The NATS cluster can be scaled up and down according to demand. Chat servers will always be able to locate an available node thanks to Fly 6PN networking advertising homogenous services through DNS and `.internal` addresses. You can then deploy the NATS cluster and/or the chat server to any region and know that both will have predictable names; the NATS cluster through Fly's 6PN networking, and the chat server through Fly's AnyCast Edge network.
## Discuss
You can discuss this example (and the nats-cluster example) on the [dedicated Fly Community topic](https://community.fly.io/t/new-examples-nats-cluster-and-6pn-demo-chat/562).
# EdgeDB on Fly
Source: https://docs.fly.io/app-guides/edgedb
Run EdgeDB on Fly.io backed by a Fly Postgres app: set EDGEDB_SERVER_* secrets, attach Postgres, grant role permissions, and persist the TLS cert.
[EdgeDB](https://www.edgedb.com) is a [graph-relational database](https://www.edgedb.com/blog/the-graph-relational-database-defined) that runs on top of Postgres and is designed as a spiritual successor to SQL. This guide explains how to perform a single-region deployment of EdgeDB with persistent storage. It will require two apps: one for Postgres and one for the EdgeDB container.
## Create the Postgres app
Create a Postgres app. You'll need to replace `mypostgres` with a unique name of your choosing.
```bash cmd theme={null}
flyctl postgres create \
--name mypostgres \
--vm-size performance-1x \
--volume-size 10 \
--initial-cluster-size 1
```
```output output theme={null}
Automatically selected personal organization:
? Select region: sea (Seattle, Washington (US))
Creating postgres cluster mypostgres in organization personal
Postgres cluster mypostgres created
Username: postgres
Password:
Hostname: mypostgres.internal
Proxy Port: 5432
PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!
```
## Create the EdgeDB app
Create a directory and create a new app. Be sure to replace `myedgedb` with a custom name. When prompted, pick a region and a Fly organization.
```bash theme={null}
flyctl launch --name myedgedb --image edgedb/edgedb --no-deploy
```
Since we're using `flyctl launch`, this will create a `fly.toml`. All `flyctl` commands will now be applied against our `myedgedb` application unless otherwise specified with `--app`.
EdgeDB needs more RAM than the default `256MB` so let's scale the RAM to `1024MB`.
```bash theme={null}
flyctl scale memory 1024
```
Set a password for the EdgeDB instance as a Fly secret. Keep track of this password; it won't be visible again.
```bash theme={null}
flyctl secrets set EDGEDB_PASSWORD=mysecretpassword
```
We need to configure a few other environment variables too.
```bash theme={null}
flyctl secrets set \
EDGEDB_SERVER_BACKEND_DSN_ENV=DATABASE_URL \
EDGEDB_SERVER_TLS_CERT_MODE=generate_self_signed \
EDGEDB_SERVER_PORT=8080
```
Let's discuss what's going on with all these secrets.
* `EDGEDB_SERVER_BACKEND_DSN_ENV` tells EdgeDB which environment variable holds the Postgres connection string.
* `EDGEDB_SERVER_TLS_CERT_MODE` tells EdgeDB to auto-generate a self-signed TLS certificate.
* You may instead choose to provision a custom TLS certificate. In this case, you should instead create two other secrets: assign your certificate to `EDGEDB_SERVER_TLS_CERT` and your private key to `EDGEDB_SERVER_TLS_KEY`.
* `EDGEDB_SERVER_PORT` tells EdgeDB to listen on port `8080` instead of the default 5656, as Fly uses `8080` for its default health checks.
## Attach and deploy
Let's attach `mypostgres` to `myedgedb`.
```bash theme={null}
flyctl postgres attach mypostgres
```
This sets the value of `DATABASE_URL` in the `myedgedb` app and creates a new role called `myedgedb` in our Postgres database. By default, this role doesn't have the full set of permissions EdgeDB needs to run. To change that, open a connection to your Postgres instance.
```bash cmd theme={null}
flyctl postgres connect --app mypostgres
```
```output output theme={null}
Connecting to mypostgres.internal... complete
psql (14.4 (Debian 14.4-1.pgdg110+1))
Type "help" for help.
postgres=#
```
Then run the following REPL command.
```
postgres=# alter role myedgedb createrole createdb;
ALTER ROLE
postgres=# \quit
```
Finally, our EdgeDB app is ready to deploy!
```bash theme={null}
flyctl deploy
```
It may take a few minutes to fully deploy. Debug with `flyctl logs` and check the deployment status with `flyctl status`.
```bash theme={null}
flyctl status
```
## Persist certificates
We need to persist the auto-generated TLS certificate to make sure it survives EdgeDB app restarts. (If you’ve provided your own certificate, skip this step.)
```bash theme={null}
flyctl ssh console \
-C "edgedb-show-secrets.sh --format=toml EDGEDB_SERVER_TLS_KEY EDGEDB_SERVER_TLS_CERT" \
| tr -d '\r' | flyctl secrets import
```
## Connection
You'll be able to connect to your EdgeDB instance using the following DSN from *other Fly.io apps* in the same organization:
`edgedb://edgedb:mysecretpassword@myedgedb.internal:8080`
To pass this DSN to another app, create a secret called `EDGEDB_DSN` containing this value. EdgeDB's client libraries will read this environment variable and connect to your instance automatically.
```bash theme={null}
flyctl secrets set \
EDGEDB_DSN=edgedb://edgedb:mysecretpassword@myedgedb.internal:8080 \
--app my-other-app
```
If you've [configured a Wireguard tunnel](/networking/private-networking) on your local machine, you'll be able to open a REPL to your new EdgeDB instance with the `edgedb` CLI.
```bash cmd theme={null}
edgedb --dsn edgedb://edgedb:mysecretpassword@myedgedb.internal:8080 --tls-security insecure
```
```output output theme={null}
EdgeDB 2.x (repl 2.x.x)
Type \help for help, \quit to quit.
edgedb>
```
# Fauna and Fly.io
Source: https://docs.fly.io/app-guides/fauna
Pair a Fly.io app with a Fauna database: Fauna starter kits, US and EU Region Groups, and which Fly regions to scale into with fly scale count.
[Fauna](https://fauna.com/) is a distributed relational database with a document data model, and delivered as an API. Databases created in Fauna automatically get three geographically distributed replicas with active-active writes and ACID compliance, making it a powerful complement to Fly.io in serving low latency reads and writes for dynamic global applications.
## Starter kit
Fauna provides [starter kits](https://github.com/orgs/fauna-labs/repositories?q=fly-io-starter) to help you quickly launch an app on Fly.io, using Fauna as the database backend. Before you start, read the short section below to learn how Fauna and Fly.io makes for a great combination when building low latency, dynamic global applications.
## Region Groups
A Fauna Region Group refers to the locality footprint of where the replicas are located, and allows you to control where your data resides, making it possible to comply with data locality legislation, such as the General Data Protection Regulation ([GDPR](https://gdpr-info.eu/)).
When you create a database in Fauna, you choose which Region Group you want the database created in. When you query the database, your request is automatically routed to the closest replica (within the Region Group) based on latency. This functionality comes out-of-the-box without any special configuration or deployment strategy.
## Scale your deployment to match the Fauna footprint
Currently, Fauna provides 2 choices of Regions Groups, US and EU. The table below lists the Fly.io regions that are closest to the Fauna replicas of each respective Region Group:
| Fauna Region Group | Deploy on Fly Regions |
| - | - |
| EU | lhr, arn, fra |
| US | sjc, ord, iad |
To take full advantage of Fauna's distributed footprint, you should deploy your app on 3 Fly.io regions as well, and as close as possible to the Region Groups' replicas.
For example, let's say you created your Fauna database in the US Region Group and you used the Fauna [Python Fly.io starter kit](https://github.com/fauna-labs/python-fly-io-starter) with the default configuration to create your Fly app. Your app will be in the `sjc` region, as determined by `primary_region` in the `fly.toml` config file. You can add Machines in the other closest US regions using this command:
```
fly scale count 2 --region ord,iad
```
Run `fly scale show` to see where your app's Machines are running. For example:
```bash cmd theme={null}
fly scale show
```
```output output theme={null}
VM Resources for app: my-app-name
Groups
NAME COUNT KIND CPUS MEMORY REGIONS
app 3 shared 1 256 MB iad,ord,sjc
```
# Appkata: Gogs - standalone Git Server
Source: https://docs.fly.io/app-guides/git-gogs-server
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# appkata-gogs
Deploying Gogs using Fly.
[Read this guide on Fly.](/app-guides/git-gogs-server)
## Rationale
**Appkata**: The application fighting style - and a series of application-centric examples for Fly.io.
The fundamental tool of modern development is a version control system, most often Git. Let's get some Git on Fly. If we just skip past creating a bare Git server because that's no fun, we find Gogs. Gogs bills itself as "a painless self-hosted Git service" and comes ready-packed as an easier to install, and complete Git service, including a web front end for issues and pull requests.
Let's get Gogs going on Fly. Our starting point is [gogs/docker](https://github.com/gogs/gogs/tree/main/docker) which has all the configuration and files related to the official Gogs docker image. It's that image we are going to build with. We are going to use that image to deploy on Fly.
## Databases
Gogs needs the support of a database, typically MySQL or Postgres, which would involve deploying a whole separate process. But it also supports using SQLite3 as a local database. The database will need persistent disk space. We'll use the Fly Volumes feature to give our application that disk space.
## Ports
Gogs runs a web service on port 3000 and an SSH service on port 22. We'll get Fly to redirect the web service to port's 80 and 443, and put SSH on port 10022. First though, some initialization
```bash cmd theme={null}
> fly init appkata-gogs --image gogs/gogs --org personal --port 3000
```
```out out theme={null}
Selected App Name: appkata-gogs
New app created
Name = appkata-gogs
Organization = personal
Version = 0
Status =
Hostname =
App will initially deploy to lhr (London, United Kingdom) region
```
This generates a `fly.toml` file with the basic components for configuring our gogs:
```toml theme={null}
# fly.toml file generated for appkata-gogs on 2020-11-02T16:00:06Z
app = "appkata-gogs"
[build]
image = "gogs/gogs"
[[services]]
internal_port = 3000
protocol = "tcp"
[services.concurrency]
hard_limit = 25
soft_limit = 20
[[services.ports]]
handlers = ["http"]
port = "80"
[[services.ports]]
handlers = ["tls", "http"]
port = "443"
[[services.tcp_checks]]
interval = 10000
timeout = 2000
```
This is the barebones of a configuration. It only redirects port 3000. The next step is to add a service to handle SSH communications from Gogs. Append this to `fly.toml`:
```toml theme={null}
[[services]]
internal_port = 22
protocol = "tcp"
[[services.ports]]
handlers = []
port = 10022
[[services.tcp_checks]]
interval = 10000
timeout = 2000
```
This passes traffic on 10022 through to Gogs on port 22. There's no handler needed for this, but it is checked as part of the Fly health checks.
## Persistent Disks
With the other port set up, we then move to the persistent data storage. Make a Fly volume in the region where the app will be running. When we "inited" the app, it told us it would initially be deployed to London, LHR region. Let's create a default (10GB) sized volume called `data` there:
```bash cmd theme={null}
fly volumes create data --region lhr
```
```out out theme={null}
ID: JD20D7DMOVAequGbLPR
Name: data
Region: lhr
Size GB: 10
Created at: 04 Nov 20 09:45 UTC
```
The Gogs image expects to see its data volume mounted on /data. To do that, it's back to `fly.toml` to add a `mounts` section for our new volume:
```toml theme={null}
[[mounts]]
source="data"
destination="/data"
```
## Deploying
We can now deploy Gogs using the command `fly deploy.`
```bash cmd theme={null}
❯ fly deploy
```
```out out theme={null}
Deploying appkata-gogs
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 3000
TCP 10022 ⇢ 22
Deploying image: gogs/gogs
--> docker.io/gogs/gogs
==> Optimizing Image
--> Done Optimizing Image
==> Creating Release
Release v0 created
Deploying to : appkata-gogs.fly.dev
Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 2 total, 2 passing]
--> v0 deployed successfully
```
Note the two health-checks which match up with the two health-checks in `fly.toml`.
Gogs is now up and running and you can use `fly open` to go straight to the site where you'll be greeted by the first time run screen. Now it's time to configure our Gogs deployment.
## Custom Domains
If you have a custom domain you want to use for your Gogs server, now is the time to set it up. You'll need to edit some DNS records so kick off the process with `fly certs add yourdomain` and it'll give you instructions as you go.
## Configuring Gogs
The first stop on the install screen for Gogs is the database settings. As mentioned at the start, we want to set this to SQLlite3, which is nicely self-contained. We'll also want to make sure that the path is `/data/gogs.db` (an absolute path, not a relative one).
Scroll down and you'll come the application settings. We'll want to change the domain to the Fly domain (or your custom domain name). You'll also want to change the ssh port to the one we set in the fly.toml services (10022). The HTTP port doesn't need to change but the application URL should change to the same domain as the application URL and there's no need to set a port in the URL.
Finally, make sure to create your admin account otherwise anyone logging in first to the server will get access and thats not good.
Click to install Gogs and it'll be set up with this configuration. You can now go back to the front page - run `fly open` if you can't remember the URL - and log in. You now have your own Git repository, issues tracker and more.
## Notes
* Once set up, the configuration is effectively frozen - remember this when setting up. The clear the data and configuration, `fly suspend` the app, destroy the existing volume and create a new similarly named volume, then `fly resume` the app.
## Discuss
* You can discuss this example on the [community.fly.io](https://community.fly.io/t/gogs-standalone-git-service-as-a-fly-example/358) topic.
# Run an NGINX proxy on Fly
Source: https://docs.fly.io/app-guides/global-nginx-proxy
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
Everything starts with a Fly app on Fly. If you already have an application running somewhere else, and want to get some of the benefits of running it close to users, you can launch a proxy app on Fly. This will forward requests to your *origin* application for processing.
### Configuring NGINX
NGINX is a powerful HTTP proxy server you can use to direct traffic to its appropriate destination, apply routing rules, or even implement caching. We start with a Dockerfile that'll allow us to create a NGINX image with our own `nginx.conf` configuration file:
```
FROM nginx
COPY nginx.conf /etc/nginx/conf.d/nginx.conf
```
That's it. It copies a local configuration file into the image. In this example config, we want to proxy requests from example.com to realexample.com and proxy requests from exemplum.com to realexemplum.com.
```
server {
listen 8080;
listen [::]:8080;
server_name example.com;
location / {
proxy_pass https://realexample.com/;
proxy_set_header X-Forwarded-Host $http_host;
}
}
server {
listen 8080;
listen [::]:8080;
server_name exemplum.com;
location / {
proxy_pass https://realexemplum.com/;
proxy_set_header X-Forwarded-Host $http_host;
}
}
```
Both server sections are set to listen on port 8080. The `server_name` directive says which hostname the server directive will be handling and the `location` directive uses `proxy_pass` to send all requests on to the appropriate site.
### Deploying NGINX to Fly
We now need to create a slot for our new Fly app. Assuming you are signed up and logged in (if not see our [hands-on for getting started](/getting-started/launch-demo)), then we'll run this:
```bash theme={null}
flyctl apps create
```
This command will prompt us for an app name - we'll use `nginxproxy` but we recommend you use the autogenerated name to ensure a unique name. As well as creating the slot for the app in the Fly infrastructure, a `fly.toml` file will be created with the app name and configuration settings in it. To deploy our NGINX, we literally issue the deploy command:
```bash theme={null}
flyctl deploy
```
Our app will now be built and deployed onto the Fly infrastructure. When that's complete, get the status of the application with `flyctl status`. You'll see it has a hostname which you can use to connect. The host name will match none of the hosts in the `nginx.conf` file and so it will proxy for the default server - the first server listed in that configuration file.
### Choose your own adventure
With the proxy app in place, you now have a foundation for adding features "at the edge". Maybe try HTTP caching in NGINX, or look at our [OpenResty guide and learn to write Lua](/app-guides/openresty-nginx-plus-lua). Or if you're feeling extra spicy, let your customers [add their own domains](/networking/custom-domain) to your application.
# Apollo GraphQL + Edge Caching
Source: https://docs.fly.io/app-guides/graphql-edge-caching-apollo
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Edge GraphQL with Apollo Server and Fly.io
> Run Apollo Server + Caching close to users with [Fly](https://fly.io/).
## Overview
This application demonstrates how to use Fly, Redis and Apollo to enable rapid GraphQL queries from a variety of data sources.
Fly runs tiny virtual machines close to your users, and includes a global Redis cache. Apollo is a GraphQL server that integrates with a variety of data sources and can be run in Node. This application combines these three technologies to create a GraphQL server on Fly.
It uses the [Open Library API](https://openlibrary.org/developers/api) as a backend source to serve two GraphQL endpoints, and caches responses in Fly's Redis instance.
## Deploying to Fly
* Install [flyctl](/flyctl/install)
* Login to Fly: `flyctl auth login`
* Create a new Fly app: `flyctl apps create`
* Deploy the app to Fly: `flyctl deploy`
Once deployed, you can launch the GraphQL Playground by running `flyctl open`.
## Running GraphQL queries
To search books, enter:
```
{
search(search:"for whom the bell tolls") {
isbn
author_name
contributor
title
first_publish_year
}
}
```
To get a single book by ISBN, enter:
```
{
book(bib_key:"ISBN:0385472579") {
bib_key,
thumbnail_url,
preview_url,
info_url
}
}
```
Learn more about GraphQL playground [in the documentation](https://www.apollographql.com/docs/apollo-server/testing/graphql-playground/).
If you prefer to use to command line, there's two scripts, [`scripts/bookfind.sh`](https://github.com/fly-examples/edge-apollo-cache/blob/master/scripts/bookfind.sh) and [`scripts/booksearch.sh`](https://github.com/fly-examples/edge-apollo-cache/blob/master/scripts/booksearch.sh) you can use. Set the environment variable APPNAME to your applications name- you can find the application name with `flyctl info`.
## What Happens Inside The Application
There are a number of parts of the server which are mostly defining the configuration of the packages used. `apollo-server` handles the web interactions, `apollo-server-cache-redis` is a component for that server which manages Redis as a general purpose cache and `apollo-server-plugin-response-cache` uses that cache for the particular job of caching responses from API requests.
When a query arrives at the ApolloServer, it is filtered through the GraphQL types definition `typeDefs` and then matched with resolvers defined in `resolvers`. Both define the endpoints in terms of GraphQL types - for parsing the query - and in terms of datasources where the query can be resolved.
In this case there are two queries, book and search. The resolvers definition maps each of these to the same RESTDataSource which is configured in books-api.js. It's there that the connection to the openlibrary.org REST API is defined.
Once the REST query has been fulfilled, the server will return the results as JSON to the client that made the query. And in the background if its been configured to cache that result, it'll cache that response. Needless to say, it will also check on incoming queries to see if it already has a cached result to return.
In `index.js`, after the definition of the `typeDefs` and `resolvers`, the application has code to create the RedisCache, either locally or using Fly's global Redis service. Environment variables with a connection string to the server determine which.
Finally an ApolloServer is created, is given all the configured items, the `typedefs`, `resolvers`, a list of datasources, a cache with parameters and the `responseCachePlugin` and the server is set off listening for incoming requests.
If you are wondering where the GraphQL Playground comes from in this, that's an integrated part of the Apollo Server.
## How Does Fly Fit Into This?
Traditionally, an application like this would run somewhere in the cloud and usually distant from its users thanks to geography. The biggest problem then is the latency of the connection between the user and the application.
As we previously mentioned, Fly runs tiny virtual machines close to users in datacenters around the world. Some of these machines handle connections, terminating TLS as close to the user as possible. As TLS needs a couple of connections to get going, thats a big boost, especially with an API. The connection handlers then pass the connection on to the application itself.
Applications are also run all over the world and Fly lets you put your applications in the most appropriate regions for your task. An edge cache like this could be located in the locations where you are finding the most traffic. When there is no immediatly local application, Fly automatically finds the closest place the application is running and directs connections to that place.
Edge hosting (and Fly specifically) offers significant performance benefits over traditional centralized hosting. Because requests can be served from the node closest to the user, responses will be much faster. This example uses Redis to cache responses so after the first request is made to a region, duplicate requests in the next hour will be significantly faster.
## Testing latency with cURL
If you're running on MacOS or Linux, you likely already have the cURL command line tool installed. cURL can print timing data for requests, here are examples that show total request time, TCP connect time, and TLS handshake time.
#### GraphQL cURL request
See [`scripts/timegraphqlapi.sh`](https://github.com/fly-examples/edge-apollo-cache/blob/master/scripts/timegraphqlapi.sh)
```curl theme={null}
curl 'https://.fly.dev/' \
-o /dev/null -sS \
-X POST \
-H "Content-Type: application/json" \
-d '{"operationName":null,"variables":{},"query":"{ book(bib_key: \"ISBN:0385472579\") {\n bib_key\n thumbnail_url\n preview_url\n info_url\n }\n}\n"}' \
-w "Timings\n------\ntotal: %{time_total}\nconnect: %{time_connect}\ntls: %{time_appconnect}\n"
```
#### Source API cURL request
See [`scripts/timesourceapi.sh`](https://github.com/fly-examples/edge-apollo-cache/blob/master/scripts/timesourceapi.sh)
```curl theme={null}
curl 'https://openlibrary.org/api/books?bibkeys=ISBN:0385472579' \
-o /dev/null -sS \
-w "Timings\n------\ntotal: %{time_total}\nconnect: %{time_connect}\ntls: %{time_appconnect}\n"
```
## Latency Results
Here are some sample response times using this app hosted on Fly vs. making requests to the Open Library API directly:
| Request Method | Test 1 | Test 2 | Test 3 |
| - | - | - | - |
| [Open Library API](#source-api-curl-request) | 2.06s | 1.70s | 1.24s |
| [Fly.io (uncached)](#graphql-curl-request) | 1.01s | 1.03s | 2.27s |
| [Fly.io (cached)](#graphql-curl-request) | 0.13s | 0.11s | 0.09s |
On uncached requests, Fly.io must connect to the Open Library API, but subsequent requests will load data from the regional Fly cache. You can try these tests locally using [cURL](#testing-latency-with-curl)
# GraphQL on Fly with Hasura
Source: https://docs.fly.io/app-guides/graphql-on-fly-with-hasura
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
So you have a PostgreSQL database somewhere in the cloud and you want to modernize how you access it. GraphQL is likely on your list of things you want to implement, but tools like Hasura - which take care of the backend complexity - need to be installed on a server somewhere. Fly can help by putting that server where it needs to be and making API requests faster than ever.
Let's assume you have a Postgres database somewhere...
*We made life easy and have our own sample app [FlyDictionary](https://github.com/fly-apps/flydictionary) with its own Deploy to Heroku button. Click that to create your own Heroku app and a Postgres backend.*
And that you can get a URL with PostgreSQL user credentials to connect to that database.
*For Heroku users, you can find those credentials by going to the Data view and selecting the database you want to use in the list. Then select the *Settings* tab and click on *View Credentials*. This will reveal a panel of various settings; the one you want is labelled *URI*.*
Let's say that we got something like:
```
postgres://user:pass@serverhost.com:5432/databasename
```
We're now ready to deploy Hasura.
## Create an App
For any Fly application, you need to reserve an app name. So let's create an app using `flyctl` the command line for Fly.
```bash theme={null}
flyctl apps create
```
**If you don't have `flyctl`, you'll need to install it. Check out our [Install Flyctl](/flyctl/install) guide. If you haven't got a Fly account yet, run `flyctl auth signup` to get your own free account.**
You will be asked for an app name at this point. Hit return to have one auto-generated for you. You'll see the app name when the create has been completed like this:
```
❯ flyctl apps create
? App Name (leave blank to use an auto-generated name)
? Select organization: demo (demo)
New app created
Name = still-dust-14
Owner = demo
Version = 0
Status =
Hostname =
Wrote config file fly.toml
```
So, now we have a Fly app named `still-dust-14`.
## Set the Stage
Now, before we deploy Hasura, there's one more step for a smooth deployment. We are going to set some environment variables and secrets in the app. In particular, we're going to pass over our database URL to the soon to be deployed application. For this we use `flyctl secrets set`.
```bash theme={null}
flyctl secrets set \
HASURA_GRAPHQL_DATABASE_URL="postgres://user:pass@serverhost.com:5432/databasename" \
HASURA_GRAPHQL_ADMIN_SECRET="OurAdminSecret" \
HASURA_GRAPHQL_ENABLE_CONSOLE=true
```
The Admin secret helps secure the Hasura GraphQL endpoint and console. For finer tuned authentication and role based security, consult the Hasura documentation.
The `HASURA_GRAPHQL_ENABLE_CONSOLE` environment variable, set to true, will allow you to log in to Hasura's interactive console where you can create queries.
Now, technically this whole step could take place after the app had deployed but that would mean allowing the app to spin its wheels looking for a database until we did set the values. This way, Hasura can start running properly as soon as it is deployed.
## Fly Deploy
We can now deploy Hasura. We'll use the -i option of `flyctl deploy` which allows you to install publicly available images from Docker hub.
```bash theme={null}
flyctl deploy -i hasura/graphql-engine:v1.1.0
```
The Fly platform will now go and deploy the Hasura docker image on one of the Fly nodes around the world.
When the deployment process has completed, running `flyctl info` will give the details of our Hasura instance:
```
❯ flyctl info
App
Name = still-dust-14
Owner = dj
Version = 2
Status = running
Hostname = still-dust-14.fly.dev
Services
TASK PROTOCOL PORTS
app tcp 80 => 8080 [HTTP]
443 => 8080 [TLS, HTTP]
IP Addresses
TYPE ADDRESS CREATED AT
v4 77.83.141.82 1h16m ago
v6 2a09:8280:1:ad2e:fa50:9986:a226:99d3 1h16m ago
```
The important part here is the App hostname - here it's `still-dust-14.fly.dev`.
## Connecting to Hasura
Open your browser and go to the `/console` path on that host - `still-dust-14.fly.dev/console`. You'll be prompted for the admin secret we set in the secrets and once entered, you'll be in Hasura's GraphQL console. From there, you'll be able to create new tables in PostgreSQL, track existing tables and model your GraphQL API. Once you've done that, you can then create GraphQL queries. Of course, this wouldn't be how you would use Hasura day to day. You'd likely be using it as a GraphQL proxy for your applications.
```
❯ curl -X POST -H "Content-Type: application/json" -H "x-hasura-admin-secret:OurAdminSecret" --data '{ "query": "{ example { id name } }" }' https://still-dust-14.fly.dev/v1/graphql
{"data":{"example":[{"id":1,"name":"Fred"}, {"id":2,"name":"Wilma"}, {"id":3,"name":"Barney"}, {"id":4,"name":"Betty"}]}}
```
Once you do have your API configured, you can shut down the Hasura console by changing the console secret:
```bash theme={null}
flyctl secrets set HASURA_GRAPHQL_ENABLE_CONSOLE=false
```
Fly will automatically redeploy Hasura with the console disabled, while still responding to your GraphQL API requests.
# Deploy gRPC and gRPC-Web Services
Source: https://docs.fly.io/app-guides/grpc-and-grpc-web-services
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# gRPC & gRPC-Web Services on Fly.io
> Run gRPC & gRPC-Web services close to users with [Fly](https://fly.io/).
## Overview
This application demonstrates how to run a gRPC service on Fly.io. Fly runs tiny virtual machines at edge datacenters close to your users.
gRPC is designed for fast, low overhead services with client and server code generation tools for many languages and VMs.
gRPC servers run their own HTTP/2-based protocol, so they offer excellent performance benefits. This will not work out of the box with most hosting platforms. In fact, you can't even call the service directly from a web browser. In this example, we'll consume the gRPC service with a special web adapter. This allows you to create the client stubs and use them in browser-based JavaScript.
This example uses the [gRPC](https://grpc.io) libraries for the main gRPC service and [grpcwebproxy](https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy) for browser support.
## What we'll deploy
Since this is an example, we'll write a quick gRPC service definition in [`hello.proto`](https://github.com/fly-examples/grpc-service/blob/master/hello.proto) that has the following methods:
```protobuf theme={null}
service MainService {
rpc Hello(Empty) returns (Greeting);
rpc Clock(Empty) returns (stream Time);
}
```
## Deploying the gRPC service to Fly
* Install [flyctl](/flyctl/install)
* Login to Fly using `fly auth login`
* Create and deploy the app with `fly launch`
## Using the gRPC service
With gRPC, you'll usually use the generated client libraries in the language of your choice.
To make sure our service is working, we'll use the [`grpcurl`](https://github.com/fullstorydev/grpcurl) tool:
```shell theme={null}
grpcurl -proto hello.proto grpc-test.fly.dev:443 MainService/Hello
```
You can also test streaming by calling the `Clock` method. It sends a timestamp every second:
```shell theme={null}
grpcurl -proto hello.proto grpc-test.fly.dev:443 MainService/Clock
```
## What's Happening Inside
The gRPC servers that we're running have one special feature that makes them difficult to deploy on many systems – they use the relatively new HTTP/2 protocol with a special feature called HTTP Trailers. Most HTTP/2 load balancers will terminate the HTTP/2 connection at the load balancer, and then make a more compliant HTTP/1.1 connection on the backend to the application server. This won't work with gRPC.
Luckily, Fly supports connecting to your service using HTTP/2, with the following configuration in your `fly.toml`:
```toml theme={null}
[http_service]
internal_port = 54321
force_https = true
auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 0
processes = ['app']
[http_service.http_options]
h2_backend = true
[http_service.tls_options]
alpn = ["h2"]
```
Once the TCP request reaches your gRPC application, the server will then take over and provide threading / event loop / goroutine management, depending on your chosen language, along with serialization and deserialization.
## Using the Web Proxy
Because the gRPC services uses HTTP/2, you can't make requests to it from inside a browser — there are currently no browser APIs that allow direct and full control of a HTTP/2 connection. To enable use inside a browser, there's a [gRPC-Web](https://grpc.io/docs/languages/web/) specification that converts a normal HTTP/1.1 request to and from the gRPC HTTP/2 format. Multiple proxy servers that implement the spec are available, like [Envoy](https://grpc.io/docs/languages/web/basics/#configure-the-envoy-proxy) and [grpcwebproxy](https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy) (which we've deployed here). We've generated JS code for our gRPC definition using the instructions [on the official gRPC-Web example page](https://github.com/grpc/grpc-web/tree/master/net/grpc/gateway/examples/helloworld#generate-protobuf-messages-and-client-service-stub), so let's see how to use it.
In the `web-proxy` folder, the [`client.js`](https://github.com/fly-examples/grpc-service/blob/master/web-proxy/client.js) file calls the two methods of our service and writes the output into `console.log`. You can change the following [line](https://github.com/fly-examples/grpc-service/blob/master/web-proxy/client.js#L4):
```js theme={null}
var client = new MainServiceClient('https://grpc-web-proxy-test.fly.dev:443');
```
to update it with your web proxy app's hostname and then run:
```shell theme={null}
npm install
npx webpack client.js
open index.html
```
This will re-compile the client and open up your browser, where the method repsonses are being printed into the console.
You'll notice that the clock stops streaming after exactly 10 seconds – this is based on a [flag](https://github.com/improbable-eng/grpc-web/blob/b16a11b6e855a48b6bc6e369a85f3d46fcfe1a77/go/grpcwebproxy/main.go#L45) in the proxy configuration that you'll want to set if your services have longer-running methods. For example, you can set them to 1 hour by adding `--server_http_max_write_timeout=3600` and `--server_http_max_read_timeout=3600` to the `ENTRYPOINT` command in [`web-proxy/Dockerfile`](https://github.com/fly-examples/grpc-service/blob/master/web-proxy/Dockerfile#L40).
## How Does Fly Fit Into This?
While gRPC works really well when communicating between servers on the same rack or same datacenter, the principles it uses apply equally well to clients on websites or mobile devices. The HTTP/2 communication channel remains open as much as possible to provide a low-latency and low-energy way to speak to your server, and the underlying [Protocol Buffers](https://developers.google.com/protocol-buffers) serialization/de-serialization system is very low-overhead, so it's great for cellular or metered-bandwidth connections.
Running your backend services on Fly lets you put your services really close to every user in your global client base. If your data is in a central location, your services can use the local Redis cache that Fly provides at each edge location to cache data, and broadcast changes or invalidations via the [global Redis command broadcast](/upstash/redis). There are also data solutions from other cloud providers like [DynamoDB Global Tables](https://aws.amazon.com/dynamodb/global-tables/) (NoSQL), [Aurora Multi-Master](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) (SQL) or [Google Cloud Spanner](https://cloud.google.com/spanner) (SQL) that can help spread your data across the world and closer to your application servers.
The reduced latency from having your servers close to your clients is especially useful in gRPC where you want all your calls to finish as quickly as possible. If your clients are mobile applications, they'll benefit even more from the latency reduction when they're running on cellular networks, where bandwidth is low and re-connections are common.
## Testing gRPC performance with `ghz`
Because gRPC applications use a special wire protocol, we can't use `cURL`, `wget` or `siege` like we would on normal servers. Instead, there are special gRPC-aware tools like [ghz](https://ghz.sh) that work great. After you enable the Fly regions you'd like to run your service in and set the container CPU & memory configuration to the size you need, you can use `ghz` to test your service:
```
ghz grpc-test.fly.dev:443 --call=MainService/Hello --proto=hello.proto
```
Keep in mind that this test will run against your closest enabled Fly region, which may or may not be the region that your users will use. gRPC testing also works a little different, because a single HTTP/2 connection is re-used across many requests. Testing against dummy methods that don't do much is really just a test of single connection bandwidth. The best way to make sure your service is running great would be to enable the instrumentation provided in your client and service libraries and exercise your service in a real-world scenario.
# Deploy Livebook on Fly.io
Source: https://docs.fly.io/app-guides/livebook
Deploy Livebook to one Fly Machine with a 1GB volume: a ready fly.toml, LIVEBOOK_PASSWORD secret, shared-cpu-2x sizing, and auto stop and start notes.
In this guide we'll deploy [Livebook](https://livebook.dev), the interactive Elixir notebook app, to a single Fly Machine using the [Fly Launch](/launch/index) approach.
We'll give Livebook a 1GB [persistent volume](/volumes) on which to store notebooks and other data.
## Speedrun
1. Create a `fly.toml` file with the contents in the next section.
2. `fly launch --no-deploy`
3. `fly secrets set LIVEBOOK_PASSWORD=""`
4. `fly deploy --vm-size shared-cpu-2x --volume-initial-size 1`
5. `fly apps open`
## Prepare a Fly Launch config
Fly Launch uses a TOML configuration file—called `fly.toml` by default.
Prepare a directory for your Livebook app config and place a `fly.toml` file inside, with the following contents:
```toml theme={null}
[build]
image = "ghcr.io/livebook-dev/livebook:0.13.0"
# use a prebuilt Livebook Docker image
[env]
ERL_AFLAGS = "-proto_dist inet6_tcp" # enable ipv6 for Fly.io private networking
LIVEBOOK_DATA_PATH = "/data" # /data is the mount point of the persistent volume
LIVEBOOK_HOME = "/data"
LIVEBOOK_ROOT_PATH = "/data"
LIVEBOOK_IP = "::" # listen on all ipv6 interfaces
PORT = "8080" # listen on port 8080
[[mounts]]
source = "data" # first deployment creates a volume with this name
destination = "/data" # mount location on the Machine file system
[http_service]
internal_port = 8080 # the Livebook app listens on [::]:8080
force_https = true
auto_stop_machines = true # when no http connections, the Machine stops
auto_start_machines = true # Fly proxy restarts the Machine on request
min_machines_running = 0 # allow all Machines to stop
processes = ["app"] # the default process group
```
This is the app-wide config that gets applied to all the Machines in the app when you run `fly deploy`.
## Create a new Fly app
Create a new app by running `fly launch` from the directory containing the app's `fly.toml` config file.
```bash theme={null}
fly launch --no-deploy
```
Follow the prompts. Respond `Y` to copy the configuration from your existing `fly.toml` to the new app. You'll be asked for an app name and a primary region, which will be added to your local `fly.toml`. The app name must be unique across Fly.io. The primary region is the region where Machines are created for this app when not otherwise specified.
## Set a Livebook password
Livebook stores your password in an environment variable. Set that variable securely using `fly secrets set`.
```bash theme={null}
fly secrets set LIVEBOOK_PASSWORD=""
```
Livebook requires that this password be at least 12 characters long. Don't use this example password as a real password.
## Deploy the app
Now that the app has been created and the `LIVEBOOK_PASSWORD` secret is set, you can deploy to a Machine.
The `fly deploy` command creates a new release of an app using the `fly.toml` in your working directory, and rolls that out to its Machines.
The first time it's run on a given app, and when it's run on an app that currently doesn't have any Machines, `fly deploy` does some extra work. For our Livebook app, it provisions public IP addresses for the HTTP service and creates the volume referred to in the `mounts` section in `fly.toml`, before creating and starting the app's Machine(s).
```
fly deploy --vm-size shared-cpu-2x --volume-initial-size 1
```
The `fly deploy` command accepts some options to tune the initial resource provisioning.
Livebook seems to want something a bit heftier than the default `shared-cpu-1x` Machine with 256MB RAM, so for the purpose of demonstration we chose a `shared-cpu-2x` size, which comes with 512MB. You can experiment with CPU and memory to see what you need for your project.
We explicitly set the size of the volume to be created, although 1GB happens to be the default size for a new volume created on `fly deploy`.
> [!NOTE]
>
> At the time of writing, you may see this error message as the first deployment finishes:
>
> `WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.`
>
> If you can reach your Livebook in the browser, then you can ignore this message. Occasionally, provisioning the storage volume can delay startup of Livebook until after this check times out.
## Visit your Livebook app
To use your Livebook, visit `https://.fly.dev` in the browser, or type `fly apps open`, and enter your password where prompted. Create a new notebook, and save it in the `/data` directory.
You can test that your notebook was saved onto the persistent volume by running `fly apps restart` and clicking "Open" on the Livebook home page to find and open it from storage.
## A note on auto start and stop settings
The `auto_stop_machines` and `auto_start_machines` config options are set to `true` in the app config. The Fly proxy will shut down the Machine when there's nobody connected to it. Livebook does try to save even "unsaved notebooks," but principle you can lose data if it shuts down and you have not saved your work.
The proxy will also start the Machine when a connection is requested. This means bots may wake up your app.
When you visit the app when it's stopped, it will take a moment before it's running again.
If you'd rather start and stop your Livebook app manually, you can do so directly with flyctl. Set `auto_stop_machines` and `auto_start_machines` both to `false`, and redeploy. Use the `fly machine stop` and `fly machine start` commands to stop the Machine when you're not using it and to start it up again when you're ready.
## A note on persistent storage
Fly volumes don't automatically synchronize across an app, so the simplest way to run Livebook with volume storage is as a single Machine, with the availability repercussions that implies. Other deployment configurations are beyond the scope of this guide, but Livebook does provide the option to connect with an object storage service.
# Deploy MinIO Object Storage to Fly.io
Source: https://docs.fly.io/app-guides/minio
Self-host S3-compatible MinIO object storage on Fly.io with a volume, reach the console via fly proxy, use the mc client, and set up a 3-node cluster.
Object storage is useful when your apps need access to unstructured data like images, videos, or documents. Amazon's S3 is an obvious solution, but you can also host your own S3-compatible object storage using the AGPLv3-licensed [MinIO](https://min.io/).
We're going to set up a single instance of MinIO on Fly.io, backed by [Fly Volume storage](/volumes).
A note: it's been observed that some features of the MinIO browser console are memory-intensive enough to crash a VM with 256MB of RAM. For even small-scale production use you'll probably need a larger (non-free) VM. 512MB seems to work for poking around and uploading small objects. (For perspective, [MinIO docs recommend a minimum of 128GB RAM](https://min.io/docs/minio/container/operations/checklists/hardware.html#minio-hardware-checklist) for deployment at scale.)
This is a guide to a demonstration deployment of the official standalone MinIO Docker image, and isn't tuned in any way. Check out the [MinIO docs](https://min.io/docs/minio/linux/index.html) for more sophisticated use.
## Dockerfile
We'll use the official [minio/minio](https://hub.docker.com/r/minio/minio) Docker image, but with a custom start command. Here's the Dockerfile to build that:
```docker theme={null}
FROM minio/minio
CMD [ "server", "/data", "--console-address", ":9001"]
```
This just takes the MinIO image and runs it with the command `server /data --console-address :9001`, so that the server runs using the `/data` directory for storage. We'll get to that directory in a moment. The `--console-address` flag specifies which port to run the web-based admin console on. Otherwise, MinIO chooses a random port. See below for how to access this panel after MinIO has been deployed.
## Initializing the app
Initialize the app with `fly launch`. Here we use flags to specify an app name and the organization it belongs to. The most important flag is `--no-deploy`, because we still have Things to Do before we deploy:
```bash cmd theme={null}
fly launch --name my-minio --org personal --no-deploy
```
```out out theme={null}
Creating app in /Users/jfent/fly-tests/my-minio
Scanning source code
Detected a Dockerfile app
Some regions require a paid plan (bom, fra, maa).
See https://fly.io/plans to set up a plan.
? Choose a region for deployment: London, United Kingdom (lhr)
App will use 'lhr' region as primary
Created app 'my-minio' in organization 'personal'
Admin URL: https://fly.io/apps/my-minio
Hostname: my-minio.fly.dev
Wrote config file fly.toml
Validating /Users/jfent/fly-tests/my-minio/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
```
(Hold your horses. Your app is not, in fact, ready.)
## (Un)configure networking
There's no need for this app to be accessible from the public internet, if you run your object storage within the same [Fly IPv6 private network](/networking/private-networking) as the app(s) that want to connect to it.
Delete the `[http_service]` block in `fly.toml`. Now the Fly proxy won't pass any requests to the app from outside your private network.
## Disk storage
The application's VM and image are ephemeral. When the app is stopped or moved it loses any data written to its file system. For persistent storage, provision a volume with a name and a size in the same region as the app.
```bash cmd theme={null}
fly vol create miniodata --region lhr
```
```out out theme={null}
Warning! Every volume is pinned to a specific physical host. You should create two or more volumes per application to avoid downtime. Learn more at https://fly.io/docs/volumes/
? Do you still want to use the volumes feature? Yes
ID: vol_6vjd0dkm5ny038mv
Name: miniodata
App: my-minio
Region: lhr
Zone: cbfa
Size GB: 3
Encrypted: true
Created at: 13 Nov 23 12:07 UTC
Snapshot retention: 5
```
We didn't specify a size, so we got the default: 3GB.
Tell the app to mount this volume onto its `/data` directory, by appending to `fly.toml`:
```toml theme={null}
[mounts]
source = "miniodata"
destination = "/data"
```
## Secrets
MinIO uses environment variables `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` to store administrator login information. Instead of using normal environment variables, use `fly secrets set` to pass these sensitive values to the server in encrypted form. They'll only be decrypted at runtime.
```bash theme={null}
fly secrets set MINIO_ROOT_USER= MINIO_ROOT_PASSWORD=
```
## Deployment
Now the app is, in fact, ready to deploy:
```bash theme={null}
fly deploy
```
## Accessing the web-based MinIO admin panel
MinIO has a web interface. It's served on the port specified by `--console-address` in the Dockerfile, which we set to `9001`. We can access this panel over our private WireGuard network.
One way to do this is to set up a [regular WireGuard tunnel](/networking/private-networking), and visit `http://my-minio.internal:9001` in the browser.
A simpler way is to use flyctl's user-mode WireGuard to proxy a local port to the app:
```bash cmd theme={null}
fly proxy 9001
```
```out out theme={null}
Proxying local port 9001 to remote [my-minio.internal]:9001
```
Leave this running and visit `localhost:9001` with the browser.
Log into the admin panel with the `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` values set using Fly Secrets above, and you can create buckets, do administration, and upload and download files right from the browser.
## Using the `mc` MinIO Client
You can connect to your MinIO with the `mc` command-line [MinIO Client](https://min.io/docs/minio/linux/reference/minio-mc.html).
MinIO listens for non-browser connections on port 9000, by default. If you're connecting using `fly proxy`, you'll have to proxy port 9000 to use `mc`.
You can set up an alias to connect more conveniently to your MinIO.
If you're using `fly proxy`:
```bash theme={null}
mc alias set proxy-to-minio http://localhost:9000
```
If you're hooked up with WireGuard:
```bash theme={null}
mc alias set miniotest http://my-minio.internal:9000
```
Test the alias by checking your MinIO's status:
```bash theme={null}
mc admin info miniotest
```
Create a new non-admin user with `readwrite` permissions. This user won't have full admin privileges, but will be able to create and save files to buckets.
```bash theme={null}
mc admin user add miniotest
```
At this point `` can log in and read the contents of your buckets. Make that read-write access:
```bash theme={null}
mc admin policy set miniotest readwrite user=
```
## Multi-node deployment
What if you want high availability? You can deploy MinIO in a multi-node configuration!
Each node needs to know where to find the other nodes in the configuration. This can be done using either hostnames or IP addresses, but MinIO requires that they're sequential. We'll use hostnames since we can modify `/etc/hosts` to ensure their "sequential-ness". Ensuring sequential IP addresses is much more tricky!
There's going to be a bit of back-and-forth here since we don't have any private IPs to put into `/etc/hosts` until we create the Machines, but we need a Dockerfile to create them in the first place. Bear with me!
### Dockerfile
We need to change the `CMD` in the Dockerfile to declare that we intend to run a multi-node deployment:
```docker theme={null}
FROM minio/minio
ENV MINIO_SERVER_URL="http://minio1.local:9000"
CMD [ "server", "http://minio{1...3}.local:9000/data", "--console-address", ":9001"]
```
We're using [expansion notation](https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#sequential-hostnames) to tell MinIO that there will be 3 nodes in the cluster.
### Disk Storage
Create 3 volumes, one for each node:
```bash cmd theme={null}
fly vol create miniodata --count 3 --region lhr
```
```output output theme={null}
ID: vol_8r6kj28xq7e7qw14
Name: miniodata
App: my-minio
Region: lhr
Zone: 81b8
Size GB: 3
Encrypted: true
Created at: 13 Nov 23 12:38 UTC
Snapshot retention: 5
ID: vol_9vle38y30eog2p8r
Name: miniodata
App: my-minio
Region: lhr
Zone: d88f
Size GB: 3
Encrypted: true
Created at: 13 Nov 23 12:38 UTC
Snapshot retention: 5
ID: vol_7r11e3jk1p8g17pr
Name: miniodata
App: my-minio
Region: lhr
Zone: cbfa
Size GB: 3
Encrypted: true
Created at: 13 Nov 23 12:38 UTC
Snapshot retention: 5
```
### Initial deployment
You need to deploy and scale the app in order for private IPs to be assigned. First, deploy:
```bash theme={null}
fly deploy
```
...then scale up to 3 Machines:
```bash theme={null}
fly scale count 3
```
### Private networking
One of the great things about Fly.io is the [private networking](/networking/private-networking) we get for free. We'll use our Machines' private IPs to define hostnames in `/etc/hosts` which our nodes will use to communicate with each other.
Get the private IPs of your Machines:
```bash cmd theme={null}
fly ip private ls
```
```out out theme={null}
ID REGION IP
4d89703f427258 lhr fdaa:3:7c6d:a7b:8f:143c:79a5:2
080e475b1d2758 lhr fdaa:3:7c6d:a7b:172:ac21:8674:2
683d479cde4368 lhr fdaa:3:7c6d:a7b:172:21da:b0bc:2
```
Ideally we'd update `/etc/hosts` directly in the Dockerfile, but Docker won't let us do this at build time. Instead, we'll create a script called `run.sh` that will add our host definitions at runtime:
```sh theme={null}
#!/bin/sh
cat <> /etc/hosts
fdaa:3:7c6d:a7b:8f:143c:79a5:2 minio1.local
fdaa:3:7c6d:a7b:172:ac21:8674:2 minio2.local
fdaa:3:7c6d:a7b:172:21da:b0bc:2 minio3.local
EOF
# This script comes with the minio/minio image
/usr/bin/docker-entrypoint.sh server http://minio{1...3}.local:9000/data --console-address :9001
```
### Final deployment
Now update your Dockerfile to execute `run.sh`:
```docker theme={null}
FROM minio/minio
ENV MINIO_SERVER_URL="http://minio1.local:9000"
COPY run.sh run.sh
ENTRYPOINT [ "/bin/sh" ]
CMD [ "run.sh" ]
```
When we `fly deploy`, we should have a working multi-node MinIO deployment!
If you want to see it in action, try proxying to two different Machines. In one terminal:
```bash cmd theme={null}
fly proxy 9001:9001 -s
```
```out out theme={null}
? Select instance: lhr.my-minio.internal (fdaa:3:7c6d:a7b:172:21da:b0bc:2)
Proxying local port 9001 to remote [fdaa:3:7c6d:a7b:172:21da:b0bc:2]:9001
```
...and in a different terminal:
```bash cmd theme={null}
fly proxy 9002:9001 -s
```
```out out theme={null}
? Select instance: lhr.my-minio.internal (fdaa:3:7c6d:a7b:172:ac21:8674:2)
Proxying local port 9002 to remote [fdaa:3:7c6d:a7b:172:ac21:8674:2]:9001
```
Visit `http://localhost:9001` and `http://localhost:9002` in separate browser tabs, and log in to both with your `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`.
Create a bucket in one tab and then refresh the other tab. If everything is working, you should see the bucket in both!
## What's next?
This was a basic guide for getting an instance of MinIO deployed on Fly.io, scratching the surface of what you might want to do with an S3-compatible object store. You can use your MinIO bucket storage right from the web interface. Your apps can talk to it from within the same private network. [MinIO docs](https://min.io/docs/minio/linux/index.html) cover more advanced usage.
# Appkata: MQTT - Event driven messaging
Source: https://docs.fly.io/app-guides/mqtt
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Appkata-mqtt
Run your own MQTT server for event-based messaging on Fly
[Read this guide on Fly.](/app-guides/mqtt)
## Rationale
Your modern application is more reactive than ever and with an event-based messaging platform like MQTT, you can communicate and react more than ever. MQTT allows for publish and subscribe messages to be sent to topics and broadcast to listening clients with options to persist data and enforce quality of service on connections.
We'll be using Mosquitto, a popular free MQTT broker, to create a persistent messaging hub on Fly.
## Preparing
You will need [Mosquitto](https://github.com/eclipse/mosquitto) installed locally as we'll be using some of its tools in the process of creating and testing our server. The other thing we'll need is [mkcert](https://github.com/FiloSottile/mkcert), which allows you to create your own certificate authority and certificates. If you are in macOS, install [Homebrew](https://brew.sh/) and then run `brew install mosquitto mkcert`.
## Initialising
We're ready to begin building on the official Mosquitto image, eclipse-mosquitto and making some changes to it with a Dockerfile:
```
FROM eclipse-mosquitto:openssl
RUN mkdir /etc/mosquitto
ADD mosquitto.conf /etc/mosquitto
ADD passwd /etc/mosquitto
ADD certs /etc/mosquitto/certs
CMD ["/usr/sbin/mosquitto", "-c", "/etc/mosquitto/mosquitto.conf"]
```
We select the openssl build of Mosquitto and then add a mosquitto config directory to it in `/etc/mosquitto`. We then copy over a configuration file, passwords, and certificates to the image. We wrap up with a startup command which runs Mosquitto using our configuration.
Now we need to create a slot for the application on Fly. Run
```bash cmd theme={null}
fly init appkata-mqtt --dockerfile --port 1883 --org personal
```
```out out theme={null}
Selected App Name: appkata-mqtt
New app created
Name = appkata-mqtt
Organization = personal
Version = 0
Status =
Hostname =
App will initially deploy to lhr (London, United Kingdom) region
Wrote config file fly.toml
```
And it will create a Fly app called `appkata-mqtt` (though you'll have to find another name because fly app names are unique - if you are having trouble making one miss out the app name and Fly will generate one for you). It will also create a `fly.toml` file which will use the Dockerfile to build the app. There are to the most important things to know are the app name and the region where the app will deploy - in this case, lhr.
All that's missing now are all those files. Let's go fill in the gaps.
## mosquitto.conf
This file sets up the server:
```
persistence true
persistence_file mosquitto.db
persistence_location /mosquitto/data/
allow_anonymous false
password_file /etc/mosquitto/passwd
cafile /etc/mosquitto/certs/rootCA.pem
certfile /etc/mosquitto/certs/mqtt-server.crt
keyfile /etc/mosquitto/certs/mqtt-server.key
require_certificate false
tls_version tlsv1.2
```
A quick run through this file is in order. First, we set up where Mosquitto will persist data; specifically in a file called `/mosquitto/data/mosquitto.db`.
Then we are on to password configuration: we require them and the password file will be in `/etc/mosquitto/passwd`. Third, we set up the TLS/SSL parameters. Mostly the locations of files we will create soon, and some settings, like `require_certificate` set to false to turn off client authentication and a reasonable selection of TLS version for communications.
## Passwords
Let's create that password file. This is simply a matter of coming up with a username and password and then running
```bash theme={null}
mosquitto_passwd -c ./passwd username
```
It'll prompt you for a password (twice) and generate out password file. If you want more users and passwords, check out the mosquitto\_passwd docs where it covers batch modes and converting plaintext passwords into hashed passwords. Now we have that in place, that just leaves the certificates.
## Certificates
This is where we use `mkcert`. In one command, we'll ask it to generate
```bash theme={null}
mkcert -key-file certs/mqtt-server.key -cert-file certs/mqtt-server.crt appkata-mqtt.fly.dev
```
This will create a set of certificates for our server at `appkata-mqtt.fly.dev` - remember to replace the hostname with your app name followed by .fly.dev to make your own certificates.
As mkcert generates its own CA root, to use these certificates, we need a copy of that root.
```bash theme={null}
cp "$(mkcert -CAROOT)/rootCA.pem" certs/rootCA.pem
```
This set of three files will allow the server to use TLS. Incoming clients will need the certs/rootCA.pem file to work with this server, so make sure you have a copy.
## Services and Volumes
With the Mosquitto configuration done, we now need to prepare the Fly configuration. The first part of this is creating the volume to store the persistent data. We've already mentioned the directory name in `mosquitto.conf` and the persistence location setting. Now we need to create a Fly disk volume to hold that data in the region where the app will deploy. We got that information when we initialized the app at the start. We'll make a `mosquitto_data` volume.
```bash theme={null}
fly volumes create mosquitto_data --region lhr
```
As it stands, that volume is not connected to our application. That can be set up in the `fly.toml` file. Open it and add these lines:
```toml theme={null}
[[mounts]]
source="mosquitto_data"
destination="/mosquitto/data"
```
This will mount the volume at `/mosquitto/data`. Before you close, there is something else to do in `fly.toml` and thats configure the services. By default, the `fly.toml` file directs traffic on port 80 and port 443 to the internal port like so:
```
[[services.ports]]
handlers = ["http"]
port = "80"
[[services.ports]]
handlers = ["tls", "http"]
port = "443"
```
Well, we aren't going to set a handler for MQTT traffic and we're going to use port 10000 for it. So replace those two sections of config with:
```toml theme={null}
[[services.ports]]
handlers = [ ]
port = "10000"
```
Save the file and we're ready to deploy.
## Deploying and Testing
To deploy our server is simple:
```bash theme={null}
fly deploy
```
Once deployed, we'll want to test it. Open up two terminals. In one terminal we'll set up a subscribe to listen to all topics:
```bash theme={null}
mosquitto_sub -L mqtts://username:password@appkata-mqtt.fly.dev:10000/# --cafile certs/rootCA.pem
```
Insert the username and passwords you used when creating the passwd file with mosquitto\_passwd.
Now go to the other terminal and run:
```bash theme={null}
mosquitto_pub -L mqtts://username:password@appkata-mqtt.fly.dev:10000/tests --cafile certs/rootCA.pem -l
```
Now type something and hit return and it should show up in the sub terminal. You now have a working, TLS encrypted, username/password authorized Mosquitto server, ready to rebroadcast your message traffic.
## Discuss
* You can discuss this example on its dedicated [community.fly.io](https://community.fly.io/t/appkata-mqtt/390) topic.
# Multiple processes inside a Fly.io app
Source: https://docs.fly.io/app-guides/multiple-processes
Run multiple processes in a Fly.io app using process groups in fly.toml, a Bash script, supervisord, or overmind with a Procfile, or use separate apps.
> [!NOTE]
>
> This guide discusses different ways to run multiple processes in your app. To learn about process groups in Fly Apps and the `[processes]` configuration in `fly.toml`, see [Run multiple process groups in an app](/launch/processes). For process group configuration with the Machines API, see the `config.processes` object in the [Machine config](/machines/api/machines-resource#machine-config-object-properties).
This comes up a lot: how can you run multiple programs in an app on Fly.io? Recall that Fly.io apps are shipped to us in OCI images, usually built by Docker, and Docker has… opinions… about running multiple things in a container.
Well, [we don't use Docker to run images](https://fly.io/blog/docker-without-docker/). Your app is running in a [Machine](/machines), a fast-launching VM with its own kernel. You can do pretty much anything you want inside of it, including running as many programs as you like. Most of the time, the trick is just telling Docker how to do that.
There are a couple different ways to run multiple processes in a Fly.io app. All of them address the first rule of programs running in a Fly Machine: when your entrypoint program exits, our `init` kills the Machine and we start a new one. So ultimately, *something* has to keep running "in the foreground".
### Setting the scene
Let's use the world's dumbest Golang app as our test case. Here's a Dockerfile:
```Dockerfile theme={null}
FROM golang
RUN apt-get update && \
apt-get install -y \
bash \
curl
RUN mkdir /app
WORKDIR /app
ADD go.* *.go /app/
RUN go build
CMD tail -f /dev/null
```
I'll spare you the 10 lines of Go code, but it's just a server that runs "Hello Foo" on 8080/tcp if you run it without arguments, and "Hello Bar" on 8081/tcp if you run it `-bar`. Use your imagination!
And we're not running anything right now; it's just going to sit there inert while we figure out how to run both instances of the program.
With a working Dockerfile, we can spin this up with `fly launch`. We'll edit `fly.toml` to tell Fly.io about the two internal ports we're listening on, and `fly deploy` to effect that change.
Now, some options to actually run this stuff:
### Process groups
[Process groups](/launch/processes) are the recommended way to run multiple processes. This method runs each process in its own Machine or group of Machines.
You can define multiple process groups in your `fly.toml`, giving each a name. Each defined process runs in its own Machine or group of Machines within the one app.
You can see that in action in the below (truncated) `fly.toml` file:
```toml theme={null}
[processes]
web = "/app/server"
bar_web = "/app/server -bar"
[[services]]
processes = ["web"] # this service only applies to the web process
http_checks = []
internal_port = 8080
protocol = "tcp"
script_checks = []
```
Here we define two processes: `web` and `bar_web`. The command defined for each process group is setting the *command* passed to your Dockerfile *entrypoint*. That means your entrypoint needs to be able to handle a command being passed to it!
Here's an example of such an entrypoint:
```bash theme={null}
#!/bin/bash
if [[ $# -gt 0 ]]; then
# If we pass a command, run it
exec "$@"
else
# Else default to starting the server
exec /app/server
fi
```
Note that under the `[[services]]` section of the `fly.toml` file, we define a service for process `web`. The `processes = [...]` array acts as a filter - it will apply only to the processes listed.
You can also scale Machines horizontally and vertically by process group. See the [process groups](/launch/processes) docs for details.
### Just use Bash
This is gross, but a suggestion Docker makes in its own documentation, so it must be OK. We boot our Docker container into a shell script. That shell script is:
```bash theme={null}
#!/bin/bash
set -m # to make job control work
/app/server &
/app/server -bar &
fg %1 # gross!
```
The trick here is we can run any number of programs in the background, as long as we foreground one of them at the end of the script, so the script itself doesn't exit (and tell us to kill the VM).
Call this `gross.sh` and change the Dockerfile entrypoint:
```dockerfile theme={null}
ADD gross.sh /app/
CMD /app/gross.sh
```
This works well enough to connect the app to Fly.io's Anycast network, so here's where we'd crack open `fly.toml`:
```toml theme={null}
[[services]]
internal_port = 8080
protocol = "tcp"
[[services.ports]]
handlers = ["http"]
port = "80"
[[services.ports]]
handlers = ["tls", "http"]
port = "443"
```
(It's a contrived example, so I'm only lighting one of the ports up, but whatever).
### Use Supervisord
The other Docker documentation suggestion: use `supervisor`. `supervisor` is an actual process manager; it'll run in the foreground and manage all our processes and, most importantly, when our processes exit, `supervisor` will (configurably) restart it.
`supervisor` has [a lot of configuration options](http://supervisord.org/configuration.html). But because we're running on Fly.io, we mostly don't care about them; the platform is doing a lot of this work for us. So the `supervisor` configuration we want is pretty simple:
```toml theme={null}
[supervisord]
logfile=/dev/stdout
logfile_maxbytes=0
loglevel=info
pidfile=/tmp/supervisord.pid
nodaemon=true
user=root
[unix_http_server]
file=/tmp/supervisor.sock
[program:foo]
command=/app/server
[program:bar]
command=/app/server -bar
```
If you're running an app on Fly.io, chances are you don't want it composting its own logs in `/var/log`, so we just configure `supervisor` to log to `stdout` like a normal person. The rest, apart from our command lines, is just boilerplate.
(You could drop privileges by changing the `user` line if you wanted; whether you want to depends on whether there's meaningful security to get by running multi-user in your particular container. If basically everything in the container shares the same data, there probably isn't.)
Our Dockerfile derives from the `golang` base image, and to add `supervisor` to it, we just need to add the dep:
```Dockerfile theme={null}
FROM golang
RUN apt-get update && \
apt-get install -y \
bash \
supervisor \
curl
# ...
ADD supervisor.conf /app/
CMD supervisord -c /app/supervisor.conf
```
### Use a Procfile Manager
`supervisor` works, and lots of people use it. It's a bit fussy, though, and it pulls Python in as a dependency. There are lots of other process managers. One useful genus of them are the `Procfile` managers. `Procfiles` are so dirt simple you don't need me to explain them; just look:
```
foo: /app/server
bar: /app/server -bar
```
A Procfile manager I like a lot is [overmind](https://github.com/DarthSim/overmind). `overmind` is a Go program, so it's got a small runtime, and you could, if you were fussy, build a container that just takes the `overmind` binary and none of its build deps. We won't bother, though, since we're already bringing those deps in. So:
```Dockerfile theme={null}
FROM golang
RUN apt-get update && \
apt-get install -y \
bash \
tmux \
curl
# ...
RUN GO111MODULE=on go get -u github.com/DarthSim/overmind/v2
ADD Procfile /app/
CMD ["overmind", "start"]
```
`overmind` has a bunch of interesting features, most notable among them that it wraps `tmux`, so you can get a terminal window on any of your running programs. Deploy, and then you can:
```bash theme={null}
$ flyctl ssh console
root@30952b25:/# cd /app
root@30952b25:/# overmind connect foo
# (clouds part, screen fills with foo's terminal)
```
If you're not a `tmux` person, `ctrl-b d` detaches you from that window.
### There are so many other process managers
Use whichever you like! The only limitation you're going to run into is that Fly.io owns your `init` (sorry!); we have to be PID 1. There are a bunch of process managers that want to replace `init`, and those'll be tricky to get to work in a Fly.io app.
### Maybe you want multiple apps, though
There are good reasons to run multiple programs in a single container, but sometimes when you're doing that you're actually working against the platform (which is why Docker keeps telling you not to do that).
If you're running multiple heavy-weight things in a single container, you might explore just running them as separate Fly.io apps. The advantage to doing this, apart from the apps not fighting over the same CPUs and memory, is that you can scale separate apps independently, and put them in different sets of regions.
Fly.io apps can talk to each other over a [private network](/networking/private-networking) that's always available. They can find each other under the `.internal` top-level domain (if your apps are `foo` and `bar`, they'll be in the DNS as `foo.internal` and `bar.internal`). Because the network connection is private and encrypted, you can generally just talk back and forth without extra authentication until you know you need it; in other words, you can keep things simple.
The main conveniences you give up when splitting into separate apps are that secrets are shared across all process groups in a single app, and all process groups deploy from the same Docker image. There are also cases where separate apps are the only option. For instance, the [metrics-based autoscaler](/launch/autoscale-by-metric) operates on a whole app, so workloads that need independent scaling behavior have to live in their own apps.
You can still share a single built image across several apps. Build it once without deploying:
```bash theme={null}
fly deploy --build-only --push
```
This outputs a tag of the form `registry.fly.io/app:deployment-xxxx`. Any app in the same organization can then deploy from that image:
```bash theme={null}
fly deploy --image registry.fly.io/app:deployment-xxxx
```
The main thing to watch out for is that secrets have to be set on each app individually, which can mean some duplication.
### Maybe you don't need multiple processes
There are a bunch of reasons people want to run multiple things in a container that Fly.io already takes care of for you. For instance: [metrics are a built-in feature of the platform](https://fly.io/blog/hooking-up-fly-metrics/), as are logs. You might not need to run helper processes for this kind of stuff.
Another thing people run multiple programs to get is static asset HTTP service; you'll run your app server, and a small webserver alongside it to serve assets. You don't need to do this if you don't want! Instead, you can ask the Fly.io platform to do it for you, by adding `statics` directives to your `fly.toml`:
```toml theme={null}
[[statics]]
guest_path = "/app/public"
url_prefix = "/public"
```
When Fly.io sees that in your `fly.toml`, it'll fish your static assets out of your container and serve them directly from our proxies at the `url_prefix` you specify. This is faster and, of course, doesn't demand you package an HTTP server into your container.
# Use a MySQL Database
Source: https://docs.fly.io/app-guides/mysql-on-fly
Run self-managed MySQL on Fly.io from the official image: volume, secrets, [processes] datadir and auth plugin flags per version, fly proxy, and backups.
If you just want to run a quick, self-managed MySQL instance on Fly.io, here's how to do it. It's pretty basic, with one caveat around using a [Fly Volume](/volumes) to persist data.
Most Fly apps use a `Dockerfile` to define an application and its dependencies. However, in this case we can use MySQL's official container directly - no need for a custom `Dockerfile`!
Here's how to run MySQL.
## Create the App
We'll run MySQL as a new app:
```bash theme={null}
# Make a directory for the mysql app
mkdir my-mysql
cd my-mysql
# Run `fly launch` to create an app
# Use the --no-deploy option since we'll make changes before first deploy
# Use the --image option to specify a MySQL Docker image
fly launch --no-deploy --image mysql:8.0.37
```
Type `y` when prompted to tweak the default app settings. Then, on the Fly Launch page:
* Name the app whatever you'd like. The name will become a hostname our application uses to connect to the database, such as `my-mysql.internal`.
* If you're using MySQL 8, it's a good idea to add some additional RAM to the VM: we recommend selecting 2GB of VM Memory.
Confirm the settings and return to your terminal.
## Configure the App
Let's create a volume straight-away. If you don't create a volume, you'll lose all of your data on each deployment.
```bash theme={null}
# Create a volume named "mysqldata" within our app "my-mysql"
fly volumes create mysqldata --size 10 # gb
```
We also need to set some secrets required by the [MySQL container](https://hub.docker.com/_/mysql):
```bash theme={null}
# Set secrets:
# MYSQL_PASSWORD - password set for user $MYSQL_USER
# MYSQL_ROOT_PASSWORD - password set for user "root"
fly secrets set MYSQL_PASSWORD=password MYSQL_ROOT_PASSWORD=password
```
Save these secrets somewhere, because they're not accessible after you set them.
Finally, edit the `fly.toml` file that `fly launch` generated.
```toml theme={null}
# Keep your own app name
app = 'my-mysql'
# Keep your own primary region
primary_region = "bos"
[build]
image = 'mysql:8'
[[vm]]
cpu_kind = 'shared'
cpus = 1
memory_mb = 2048
# The [processes] section is different for 8+, 8.4, and 5.7. Use the one that matches your version.
# Use the following for versions 8 to 8.3:
[processes]
app = """--datadir /data/mysql \
--default-authentication-plugin mysql_native_password"""
# Uncomment and use the following for 8.4:
# [processes]
# app = """--datadir /data/mysql \
# --mysql-native-password=ON"""
# Uncomment and use the following for 5.7:
# [processes]
# app = "--datadir /data/mysql"
# Add the following sections for all versions
[mounts]
source = "mysqldata"
destination = "/data"
[env]
MYSQL_DATABASE = "some_db"
MYSQL_USER = "non_root_user"
```
There are a few important things to note:
1. We deleted the `[[http_service]]` block and everything under it. We don't need it!
2. In the `[env]` section, we added two not-so-secret environment variables that MySQL will need to initialize itself.
* The `MYSQL_USER` here should be any user but `root`, which already exists.
3. We added the `[processes]` section for the default `app` process, which lets us pass custom commands (overriding Docker's `CMD`).
* For MySQL 8+, you'll want to use the `mysql_native_password` authentication plugin. As shown in the fly.toml above, there are different syntax for setting this plugin based on the MySQL version.
* If you're using **MySQL 5.7**, your `app` process can be simplified as shown in the `fly.toml` example above, as it uses `mysql_native_password` [by default](https://dev.mysql.com/doc/refman/5.7/en/native-pluggable-authentication.html#:~:text=client%20programs%20use-,mysql_native_password,-by%20default.%20The).
* **MySQL 8.0.32 to 8.3** must explicitly set this with the `--default-authentication-plugin` flag.
* Starting from **MySQL 8.4**, the `--default-authentication-plugin` option is [deprecated](https://github.com/docker-library/mysql/issues/1048#issuecomment-2088836076). If you're using 8.4+, you can make use of the `--mysql-native-password` flag instead.
> [!NOTE]
>
> **Important**: Set MySQL's data directory to a subdirectory of your mounted volume. Mounting a disk in Linux usually results in a `lost+found` directory being created. However, MySQL won't initialize into a data directory unless it's completely empty. That's why you need to use a subdirectory of the mounted location: `/data/mysql`.
## Deploy the App
We're now ready to deploy the MySQL app! Go ahead and run:
```bash theme={null}
fly deploy
```
After a minute of MySQL initializing itself (on its first run), you should now have an app running MySQL!
Your other apps can access the MySQL service by its name. In my case, I would use `my-mysql.internal` as the hostname. Any app that needs to access the database should set the hostname and username as environment variables, and create a secret for the database password.
## Debugging Possible Errors
### `mysqld: Table 'mysql.plugin' doesn't exist`
This error is an indicator that the [mysql System Schema](https://dev.mysql.com/doc/refman/8.4/en/system-schema.html) inside the [MySQL data directory](https://dev.mysql.com/doc/refman/8.4/en/data-directory.html) is empty. This can happen due to different reasons. One of them is when a MySQL app is previously deployed with `--default-authentication-plugin`, and it gets re-deployed with the updated syntax `--mysql-native-password=ON`.
The quickest way to fix this is to revise the `--datadir` flag to a different directory from what was previously set:
```toml theme={null}
# Please revise the MySQL data directory to another location
# For example if you had:
[processes]
app = "--datadir /data/mysql"
# You can change the path to a different directory
[processes]
app = "--datadir /data/mysql_
```
Changing the [MySQL data directory](https://dev.mysql.com/doc/refman/8.4/en/data-directory.html) will allow MySQL to completely initialize the new directory with a complete `mysql system schema.` Once done, redeploy your changes with `fly deploy`.
## Access the database from outside
To connect to your MySQL database from outside of your Fly organization, you need a WireGuard connection. However, `fly` on your local machine can connect using [user-mode WireGuard](https://fly.io/blog/our-user-mode-wireguard-year/) magic, without you having to set up your own WireGuard tunnel.
You can forward the MySQL server port to your local machine using [`fly proxy`](/flyctl/cmd/fly_proxy):
```bash theme={null}
flyctl proxy 3306 -a my-mysql
```
You can also set a different local port, if your `3306` port is already in use:
```bash theme={null}
flyctl proxy 13306:3306 -a my-mysql
```
Then connect to your MySQL server at `localhost:3306` and the username and password credentials from above:
```bash theme={null}
mysql --protocol=tcp -h localhost -P 3306 -u non_root_user -ppassword some_db
```
## Backups
We'll take a snapshot of the created volume every day. We retain 5 days of snapshots.
To restore a snapshot, make sure you have the latest version of the flyctl, and then create a new volume using the `--snapshot-id` flag.
```bash theme={null}
# Get a volume ID
fly volumes list -a my-mysql
# List snapshots for a volume
fly volumes snapshots list vol_xxx
# Create a new volume from a snapshot
fly volumes create --snapshot-id vs_xxx --size 10
```
# Global NATS Cluster
Source: https://docs.fly.io/app-guides/nats-cluster
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Global NATS Cluster
[NATS](https://docs.nats.io/) is an open source messaging backend suited to many use cases and deployment scenarios. We use it for internal communications at Fly. This repo shows how to use it for your application.
This example creates a federated mesh of NATS servers that communicate over the private, encrypted IpV6 network available to all Fly organizations.
## Setup
1. `fly launch --no-deploy`
> You'll be prompted for an app name. Hit return to let Fly generate an app name for you. Pick your target organizatio and a starting region.
2. `flyctl deploy`
> This will start NATS with a single node in your selected region.
3. Add more regions with `flyctl regions add ` or
> For this demo, we set `ord`, `syd`, `cdg` regions.
```bash theme={null}
fly regions set ord syd cdg
```
4. Scale the application so it can place nodes in the regions.
```bash theme={null}
fly scale count 3
```
Then run `flyctl logs` and you'll see the virtual machines discover each other.
```
2020-11-17T17:31:07.664Z d1152f01 ord [info] [493] 2020/11/17 17:31:07.646272 [INF] [fdaa:0:1:a7b:abc:21de:af5f:2]:4248 - rid:1 - Route connection created
2020-11-17T17:31:07.713Z 21deaf5f cdg [info] [553] 2020/11/17 17:31:07.704807 [INF] [fdaa:0:1:a7b:81:d115:2f01:2]:34902 - rid:19 - Route connection created
2020-11-17T17:31:08.123Z 82fabc30 syd [info] [553] 2020/11/17 17:31:08.114852 [INF] [fdaa:0:1:a7b:81:d115:2f01:2]:4248 - rid:7 - Route connection created
2020-11-17T17:31:08.259Z d1152f01 ord [info] [493] 2020/11/17 17:31:08.241644 [INF] [fdaa:0:1:a7b:b92:82fa:bc30:2]:45684 - rid:2 - Route connection created
```
## Testing the cluster
While the cluster is only accessible from inside the Fly network, you can use Fly's [Wireguard support](/networking/private-networking) to create a VPN into your Fly organization and private network.
Then you can use tools such as [natscli](https://github.com/nats-io/natscli) to subscribe to topics, publish messages to topics and perform various tests on your NATS cluster. Install the tool first.
Once installed, create a context that points at your NATS cluster:
```bash theme={null}
nats context add fly.demo --server appname.internal:4222 --description "My Cluster" --select
```
You can subscribe to a topic with `nats sub topicname`:
```bash theme={null}
nats sub fly.demo
```
And then, in another terminal sessions, we can use `nats pub topicname` to send either simple messages to that topic:
```bash theme={null}
nats pub fly.demo "Hello World"
```
Or send multiple messages:
```bash theme={null}
nats pub fly.demo "fly.demo says {{.Cnt}} @ {{.TimeStamp}}" --count=10
```
You're ready to start integrating NATS messaging into your other Fly applications.
## What to try next
1. [NATS streaming](https://docs.nats.io/nats-streaming-concepts/intro) offers persistence features, you can create a NATS streaming app by modifying this demo and adding volumes: `flyctl volume create`
2. Create a [NATS super cluster](https://docs.nats.io/nats-server/configuration/gateways) let you join multiple NATS clusters with gateways. If you want to run regional clusters, you can query the Fly DNS service to with `..internal` to find server in specific regions.
## Discuss
You can discuss this example (and the paired 6pn-demo-chat example) on the [dedicated Fly Community topic](https://community.fly.io/t/new-examples-nats-cluster-and-6pn-demo-chat/562).
# Appkata: Node-RED - Flow Programming
Source: https://docs.fly.io/app-guides/node-red
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Appkata-Node-RED
Bringing Node-RED to your Fly applications
[Read this guide on Fly.](/app-guides/node-red)
## Rationale
If you are looking for a quick way to interconnect your various applications then [Node-RED](https://nodered.org/) may be the low code option you've been looking for. Node-RED began its life as a way of putting together Internet Of Things applications, modeling systems as a flow of events which entered through an input node, passed through processing nodes, and eventually arrived at an output node or nodes.
For example, an MQTT event could arrive through an MQTT node, be passed through some processing nodes, and be output as a new record in a database or forwarded to another messaging system. If that sounds complex, Node-RED visualizes these flows as an interactive graph you can manipulate.
But enough about Node-RED, you can [install it and try it locally](https://nodered.org/docs/getting-started/local) yourself. We're here today to show you how to launch Node-RED on Fly. And in the process, how to refine an application on Fly for best fit.
## Preparing
The first thing to do is to [install Node-RED locally](https://nodered.org/docs/getting-started/local), as there are some command-line operations we'll need later.
## Initialising
The simplest way to get up and running with Node-RED on Fly is to use the nodered/node-red docker image. You can do this in one line:
```bash cmd theme={null}
fly init appkata-node-red --image nodered/node-red --port 1880 --org personal
```
```out out theme={null}
Selected App Name: appkata-node-red
New app created
Name = appkata-node-red
Organization = personal
Version = 0
Status =
Hostname =
App will initially deploy to lhr (London, United Kingdom) region
Wrote config file fly.toml
```
The port is set to 1880 because that's the default port for Node-RED to communicate on. Take a note of where the application says it will initially deploy to too.
We can deploy this app right now with:
```
fly deploy
```
and we can open the app with:
```
fly open
```
Thanks to Fly's edge network, your connection will be upgraded automatically to HTTPS and you'll be talking to Node-RED. Or at least observing this warning:
## Volumes
What that error is about is that the /data directory in the Node-RED image doesn't have isn't writable. We need to give our application a writable, persistent storage space, which is where Fly volumes come in. We can create a volume, local to our application, and get Fly to mount it on /data. First step in that process is to create a volume with a name:
```bash cmd theme={null}
fly vol create node_red_data --region lhr
```
```out out theme={null}
fly vol create node_red_data --region lhr
ID: bjR5jQjvZwNYVCVDxX9R
Name: node_red_data
Region: lhr
Size GB: 10
Created at: 25 Nov 20 11:03 UTC
```
That region setting comes from when we inited the app earlier; it was going to deploy in lhr, so we've made the volume in lhr. The second part of this process is to tell the app about this new volume. For that we edit the `fly.toml` file and where we see:
```toml theme={null}
[build]
image = "nodered/node-red"
```
we add, after it:
```toml theme={null}
[[mounts]]
source="node_red_data"
destination="/data"
```
This tells Fly that the volume named `node_red_data` should be mounted on the `/data` directory. With that in place, we can `fly deploy` and `fly open` our app. There's no warning and Node-RED flows are safe over redeploys.
There's only one problem...
## Securing Node-RED
You may have noticed that there's no password authentication when we connect to Node-RED. There is an option to turn it on in the Node-RED settings.js file, but that file is notionally on the /data volume. How can we get a file there with our settings?
Let's get the settings.js file first. You can find the default settings file [in the Node-Red repository](https://raw.githubusercontent.com/node-red/node-red/master/packages/node_modules/node-red/settings.js). Download that and save it as `settings.js`.
Next, we take control of the build process.
### Switching to a Dockerfile
This is where Fly's flexibility comes into play. We can switch deployment from just deploying an image to using a Dockerfile to create our own image. We'll create a Dockerfile to reproduce what we're currently doing first:
```
FROM nodered/node-red
```
And then tell Fly to use the Dockerfile by removing the `[build]` instruction from `fly.toml`, deleting or commenting out the part that reads:
```
[build]
image = "nodered/node-red"
```
If we now run `fly deploy` you'll see this instruction being run and the app deploy as it previously did.
### Customizing Startup
When images startup, they have a command to be run and its that command we want to take over. The Node-RED image calls npm to start the Node-RED service like so:
```
npm start --cache /data/.npm -- --userDir /data
```
We'll add this command to the end of a short script which will copy our `settings.js` file over to /data/ before we start up Node-RED. This can all go in a script called `entrypoint.sh`.
```
#!/bin/bash
cp /etc/settings.js /data/settings.js
npm start --cache /data/.npm -- --userDir /data
```
But wait! How did settings.js get there and how are we going to run this script? Back to our Dockerfile, to ask it to do that work. Let's add this to the Dockerfile and:
```
COPY settings.js /etc/settings.js
USER root
COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh
USER node-red
ENTRYPOINT ["/entrypoint.sh"]
```
So, step by step:
* We copy the settings.js file into the image we are creating
* We switch to the root user
* We then copy the entrypoint.sh script to the file system root
* And we make it executable
* Then we switch back to the node-red user
* And set our entrypoint to this image to our entrypoint script.
Two little details to know: One is that the Node-RED Docker image switches to a `node-red` user as its last act, which is why we do the user switcheroo. The other is that yes, if our script was already executable, then it would be executable in the image. But just in case, and because invisible effects are invisible, we explicitly set the script to executable.
Now we can `fly deploy` and `fly open` the app again.
And everything is exactly the same! Which isn't surprising because we haven't changed the `settings.js` file so it's all still running of defaults.
### Adding a Password
Open up the `settings.js` file and find the "Securing Node-RED" section. You'll see an example of what we need to add immediatly below that: it's an `adminAuth` object.
Add to the file a section that should look like this:
```javascript theme={null}
adminAuth: {
type: "credentials",
users: [{
username: "admin",
password: "hashedpassword",
permissions: "*"
}]
},
```
This will create us an admin user, but now we need to generate a hashed version of the password. If you installed Node-Red at the start of this example, it'll be easy. Just run `node-red admin hash-pw` and enter your password when prompted. It'll return a string which you should copy in where `hashedpassword` appears above.
Now we can `fly deploy` and `fly open` and be greeted with:
Now we can login as admin and the password we set and we've secured access to our Node-RED.
## Notes
We've put together a deployable Node-RED image for Fly. In the process, we've seen how we can take control of existing images and supplement their build process to add in functionality, and how we can use scripts to take over the startup of an image to make modifications we want in place.
For Node-RED administration, editing the `setting.js` file locally and deploying will update our installed settings, so you can explore all the optional controls. For Node-RED usage, think of it as a graphically configurable glue layer within your system. You can install many of the third party nodes available and plug into many more applications and platforms.
## Discuss
* You can discuss this example on the [community.fly.io](https://community.fly.io/t/node-red-on-fly-now-an-appkata-example/413) topic.
# OpenResty: NGINX + Lua
Source: https://docs.fly.io/app-guides/openresty-nginx-plus-lua
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Running OpenResty on Fly
If you've been developing web applications for long, you've probably heard of [Nginx](https://nginx.org/en/). It's a widely used open-source HTTP server, but it has some limitations. By default, Nginx doesn't provide a way to program logical operators or write custom logic. Developers can circumvent this by using modules like [Nginx JavaScript](https://www.nginx.com/blog/introduction-nginscript/) or [Lua](https://github.com/openresty/lua-nginx-module), but that takes extra work to install and configure.
[OpenResty](https://openresty.org/en/) allows you to build full-fledged web applications by bundling Nginx with a Lua compiler and several common modules. This makes OpenResty more broadly useful than vanilla Nginx, but depending on your use case, it could be overkill.
For example, if you want to [run a simple reverse proxy](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) on your server, Nginx can handle it. If you need to add rate limiting, authentication, advanced caching, or a connection to a database, you'll need a solution like OpenResty. With the increase in distributed computing and microservices, OpenResty has become an excellent option for complex firewalls, API gateways, and even full-fledged web applications.
Typically, OpenResty is deployed to a central server. Web requests go through OpenResty before being routed to the relevant backing services, but this model isn't great for performance or redundancy. While you can set up several servers and run OpenResty on each, you have to figure out how to maintain and sync data across them.
A better solution is to use a distributed hosting platform like [Fly](https://fly.io) to run OpenResty at the edge. Using Fly will decrease latency while ensuring that a single node's failure doesn't make your website unavailable.
## How to Deploy OpenResty to Fly
In this tutorial, you'll see how to create an OpenResty application and deploy it to Fly. You'll create a reverse proxy endpoint that uses the [JSON Placeholder API](https://jsonplaceholder.typicode.com/) as a backend service. You'll use a custom Lua script to add rate limiting, and the Fly Redis connection to add API key authentication to your endpoint. All the steps you need are in this tutorial, but if you'd like to download the final application, it's [available on Github](https://github.com/karllhughes/fly-openresty) as well.
### Prerequisites
* [Flyctl command line tool](/flyctl/install).
### Preparing To Fly
To start, create a new directory called `fly-openresty` and we'll create your new app inside of it:
```bash theme={null}
mkdir fly-openresty && cd fly-openresty
```
### Configuring the Dockerfile
Fly will build and run your Docker image in a [Firecracker VM](/reference/architecture#compute) on the edge, but you need to select and configure a Dockerfile first.
OpenResty provides [several Docker images](https://github.com/openresty/docker-openresty) you can use for your application. I opted for Centos, but because of [an apparent bug in their Docker image](https://github.com/openresty/docker-openresty/issues/124), specified the `1.15.8.1-4-centos` tag.
After selecting the base image, add a `RUN` command to append `env FLY_REDIS_CACHE_URL` to the top of your `/usr/local/openresty/nginx/conf/nginx.conf` file. This line ensures that Nginx has access to the `FLY_REDIS_CACHE_URL` environment variable.
Finally, remove the default Nginx site configuration files and add `default.conf` file to the `/etc/nginx/conf.d/` directory. When done, your Dockerfile should look something like this:
```dockerfile theme={null}
# Using this base image because of: https://github.com/openresty/docker-openresty/issues/124
FROM openresty/openresty:1.15.8.1-4-centos
# Add the REDIS connection URL as an env variable in NGINX
RUN echo -e "env FLY_REDIS_CACHE_URL;\n$(cat /usr/local/openresty/nginx/conf/nginx.conf)" > /usr/local/openresty/nginx/conf/nginx.conf
# Add the configuration file
RUN rm /etc/nginx/conf.d/*
COPY default.conf /etc/nginx/conf.d/default.conf
```
Now that your Dockerfile is ready, you need to create the `default.conf` file before you can deploy your application to Fly.
### Setting up the Nginx Configuration
Before you add rate limiting and Redis to your Nginx configuration, start with a simple reverse proxy configuration.
To set up your reverse proxy, create a `default.conf` file in your root directory and add the following:
```conf theme={null}
server {
listen 80;
location /api/ {
proxy_pass http://jsonplaceholder.typicode.com/posts/;
}
}
```
This minimal Nginx configuration listens to requests on port 80 and reverse proxies any request to `/api/` to the JSON Placeholder API. Now that your Dockerfile and Nginx configuration are ready, you can create and deploy your reverse proxy to Fly.
### Creating a New Fly Application
You'll need to use `flyctl` to create a new application. If you haven't already, install the appropriate version of `flyctl` for your operating system using the [instructions here](/flyctl/install).
Next, [sign up](/getting-started/sign-up-sign-in) or [sign in](/getting-started/sign-up-sign-in) to your Fly account via the command line:
```bash theme={null}
# Sign up
flyctl auth signup
# Or sign in
flyctl auth login
```
You can now create your new app in the fly-openresty directory using the fly `init` command:
```bash theme={null}
flyctl init --port 80
```
The `--port 80` says our application communicates on port 80.
You'll be prompted for an app name. Use the auto-generated app name, select your organization, and select `Dockerfile` as your builder. You should see output similar to this in your console:
```bash theme={null}
? App Name (leave blank to use an auto-generated name)
? Select organization: YOUR ORGANIZATION
? Select builder: Dockerfile
(Create an example Dockerfile)
New app created
Name =
Owner =
Version = 0
Status =
Hostname =
Wrote config file fly.toml
```
Fly will create a `fly.toml` file in the your project's directoy and set the `internal_port` to 80 so that requests are properly routed to OpenResty.
Now you're ready to deploy your application using the command line:
```bash theme={null}
flyctl deploy
```
You should see Fly build your Docker image and push it to Fly to deploy it. When finished, you'll see output similar to the following:
```bash theme={null}
Deploying
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 80
Deploy source directory '/Users/karl/fly-openresty'
Docker daemon available, performing local build...
==> Building with Dockerfile
Using Dockerfile: /Users/karl/fly-openresty/Dockerfile
...
--> Done Pushing Image
==> Optimizing Image
--> Done Optimizing Image
==> Creating Release
Release v1 created
Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v1 deployed successfully
```
Your reverse proxy is now live on Fly! You can visit it and see the JSON Placeholder data using the Fly command line tool:
```bash theme={null}
flyctl open /api
```
In the next two sections, you'll see how to add rate limiting and authentication using a Redis store and custom Lua scripts.
### Adding Rate Limiting
Nginx reads and applies all the configuration files in the `/etc/nginx/conf.d/` directory. Because OpenResty adds the Lua compiler to Nginx, you can write [Lua code](http://www.lua.org/) inside your `default.conf` file. To add rate limiting, you can use the [lua-resty-limit-traffic](https://github.com/openresty/lua-resty-limit-traffic) library that comes with OpenResty and customize its behavior in your Nginx configuration file.
Open your `default.conf` file and replace it with the following:
```conf theme={null}
lua_shared_dict my_limit_req_store 100m;
server {
listen 80;
location /api/ {
access_by_lua_block {
-- RATE LIMITER --
local limit_req = require "resty.limit.req"
-- Allow .5 requests per second --
local lim, err = limit_req.new("my_limit_req_store", .5, .5)
if not lim then
ngx.log(ngx.ERR, "failed to instantiate a resty.limit.req object: ", err)
return ngx.exit(500)
end
-- Use the visitor's IP addres as a key --
local key = ngx.var.http_fly_client_ip
local delay, err = lim:incoming(key, true)
-- Throw an error when the limit is reached --
if err == "rejected" then
ngx.log(ngx.ERR, "Limit reached: ", err)
return ngx.exit(503)
end
}
proxy_pass http://jsonplaceholder.typicode.com/posts/;
}
}
```
This configuration passes your request through using the `proxy_pass` directive at the end, as the previous version did. Before it does, it checks if the visitor has reached their request limit using the `Fly-Client-IP` header [attached to the request by Fly](/networking/services#http-handler). If the IP address has called the endpoint in the past 2 seconds, it returns a 503 response and logs the error.
You can re-deploy this configuration file to Fly (again using `flyctl deploy`) and call the endpoint twice in quick succession to test it.
You've now got a working rate limiter, but you're still not finished. In the last section, you'll see how to connect your OpenResty application to Redis to store API keys that can be used for authentication.
### Checking API Keys Against a Redis Store
Fly offers a [region-local Redis instance](/upstash/redis) to all deployments, which can be used to persist data for longer periods. While this volatile datastore is not meant for permanent use, you can use it to cache data so that it's accessible on the edge.
In this last step, you'll connect to the Fly Redis instance using the OpenResty [Redis driver](https://github.com/openresty/lua-resty-redis). You'll authenticate requests using a `key` passed in by the user through a query string argument and return a 401 response code if authentication fails. In a real application, you would probably push data to the Fly Redis instance using [their global data store](/upstash/redis), but because this is a demonstration app, you'll hard code a few sample API keys.
First, write a Lua script to parse the connection string. Fly's `FLY_REDIS_CACHE_URL` must be split at the `:` and `@` characters, so you can write a function that takes any number of characters as possible delimiters ([credit to Walt Howard on Stack Overflow](https://stackoverflow.com/a/29497100/977192) for this one). Create a new file called `split.lua` and add the following:
```lua theme={null}
local _M = {}
function _M.split(source, delimiters)
local elements = {}
local pattern = '([^'..delimiters..']+)'
string.gsub(source, pattern, function(value) elements[#elements + 1] = value; end);
return elements
end
return _M
```
Next, you need to make sure this file is copied into your Docker image, so open up your `Dockerfile` and add the following line to the end of it:
```dockerfile theme={null}
...
COPY split.lua /etc/nginx/conf.d/split.lua
```
Finally, you're ready to update your `default.conf` file. Open it and edit it as shown below:
```conf theme={null}
lua_shared_dict my_limit_req_store 100m;
lua_package_path "/etc/nginx/conf.d/?.lua;;";
-- Ensures that the Redis connection's DNS resolves --
resolver 8.8.8.8;
server {
listen 80;
location /api/ {
access_by_lua_block {
-- RATE LIMITER --
...
-- REDIS CACHE --
splitter = require("split")
local redis_client = require "resty.redis"
local redis = redis_client:new()
redis:set_timeouts(1000, 1000, 1000)
-- Split the connection string env variable --
parts = splitter.split(os.getenv("FLY_REDIS_CACHE_URL"), ":@")
-- Connect to Redis --
local res, err = redis:connect(parts[4], parts[5])
if not res then
ngx.log(ngx.ERR, "failed to connect: ", err)
return
end
-- Authorize using the password --
local res, err = redis:auth(parts[3])
if not res then
ngx.log(ngx.ERR, "failed to authenticate: ", err)
return
end
-- Set some allowed API keys --
-- Note: This should be done outside this script in a real app --
ok, err = redis:set("oVr0mDgJejSmb9jwXp6B", 1)
ok, err = redis:set("AstIqxOHpyAToCwh8qeL", 2)
ok, err = redis:set("eaFW03Pjp27ZbgqpgqJQ", 3)
-- Lookup the `key` --
local res, err = redis:get(ngx.var.arg_key)
if (not res) or (res == ngx.null) then
ngx.log(ngx.ERR, "Invalid key: ", err)
return ngx.exit(401)
end
-- Close the connection --
local res, err = redis:close()
if not res then
ngx.say("failed to close: ", err)
return
end
}
proxy_pass http://jsonplaceholder.typicode.com/posts/;
}
}
```
Deploy this updated configuration file using `flyctl deploy` and visit `https://.fly.dev/api/` again. This time, you will get a 401 response from OpenResty. Add one of the API keys you hard-coded as a `key` in the query string: `https://.fly.dev/api/?key=AstIqxOHpyAToCwh8qeL` and you'll see the JSON placeholder data again.
## Conclusion
In this post, you've seen how to create an OpenResty application to extend Nginx's functionality. You've added rate-limiting to ensure that users don't abuse your API and simple authentication using data cached in Redis. Finally, by deploying the application on Fly, you can take advantage of their globally distributed edge hosting environment to make your app faster and more reliable than it would be on traditional hosting.
If you have any questions about using OpenResty with [Fly.io](https://fly.io/), be sure to reach out so we can help you get started.
# Fly.io and PlanetScale
Source: https://docs.fly.io/app-guides/planetscale
Connect a Node app on Fly.io to PlanetScale read-only regions, pick the closest replica per region, and replay writes to the primary with fly-replay.
> [!WARNING]
>
> We haven't yet updated this page for Fly Machines, which are the VMs now at the core of all Fly Apps. You might notice that some features don't work the same way anymore. Visit our [community forum](https://community.fly.io/) if you need help with your app.
PlanetScale is a MySQL-compatible serverless database. It now supports read replicas so you can perform low-latency reads wherever your users are located. This guide walks you through setting up PlanetScale with Fly.io apps.
We'll work with a [sample Express app](https://github.com/fly-apps/nodejs-planetscale-read-replicas/) for the [Fly platform](https://fly.io) which uses a [PlanetScale](https://planetscale.com) database.
## Create a database
The sample app assumes a database already exists. You'll need to create that database before running the app. Since read-only regions can currently only be created using the PlanetScale UI, this guide does not use their CLI.
[Sign in to PlanetScale](https://auth.planetscale.com/sign-in), creating an account if you don't already have one. You can use a free account to create a database however you will need to upgrade to their paid *Scaler*, *Team*, or *Enterprise* [plan](https://planetscale.com/pricing) if you would like to add read-only regions.
Click on the black 'New database' button and pick 'Create new database'. Give it a name and choose its region. This is your **primary** database (we'll refer to it in multiple places later on). Choose the closest region to where you will deploy your app for the best performance:
PlanetScale uses branches for schemas. An initial `main` branch will be automatically created. It will take a few seconds to initialise so wait until that notification disappears. You should see a page similar to this:
Click on the *Connect* button (or on the blue *"connect to your database"* link). A dialog will open showing your username and password. Handily PlanetScale makes connection strings for various clients. For this app we'll choose the one listed to connect with Node.js. When we mention using the connection string for your primary database below, *this* is the one we mean. Make a temporary note of this as it will only show you it once:
There are many SQL clients you could use to manage your database. We'll use [ArcType](https://arctype.com/). That comes with a PlanetScale integration. However to use its PlanetScale integration you need to create an Arctype account. For now we won't. So instead we'll use the standard *MySQL* option. Click the appropriate option in your SQL client to create a new connection:
To save typing in all the details (a username, a password, and so on) we'll use that connection string noted above. Copy the entire value (starting from *"mysql://"*) into this box and then click the "Test connection" button. After a moment you should see a message to say it successfully connected. Click the blue "Save" button to proceed:
Click the blue "New Query" button in the top-right ready to run some basic SQL. Again, your editor will look slightly different:
We'll use the names of some fruits as some sample data. So paste this into your query editor and click the "Run" (or equivalent) button:
```sql theme={null}
CREATE TABLE `fruits` (
`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
`name` varchar(255) NOT NULL
);
```
You should see it succeeded:
Add a few rows to that table:
```sql theme={null}
INSERT INTO `fruits` (id, name) VALUES (1, 'banana');
INSERT INTO `fruits` (id, name) VALUES (2, 'apple');
INSERT INTO `fruits` (id, name) VALUES (3, 'strawberry');
```
Promote the main branch to production. The production branch is protected, highly available and is automatically backed up. You can promote a branch within the PlanetScale UI by simply clicking this button:
Assuming you have a *Scaler*, *Team*, or *Enterprise* account, you will be able to add additional read-only regions. They can greatly improve the performance of your application, reducing the latency of reads from regions close to them. Click on the *main* branch (the production one). You should see the option to *Add region*:
Choose the region you would like to create it in:
After a few moments those additional read-only regions should be ready. They may take slightly longer if there is a large amount of data to be replicated. Each region has its own, unique credentials. So click the "..." and then on *Connect*. It will show you a username, password and host. Make a connection string for each region that will look like this: `"mysql://**username**:**password**@**host**/**database-name**?ssl={"rejectUnauthorized":true}"`:
Your database is now ready to be queried by our sample app!
## Run the app locally
To run it locally you will need [NodeJS](https://nodejs.org/en/download/).
1. Clone the [`nodejs-planetscale-read-replicas`](https://github.com/fly-apps/nodejs-planetscale-read-replicas) repo.
2. Run `npm install` to install its dependencies.
3. Duplicate `.env.example`, naming it `.env`.
4. The application needs to know about all the available regions. There are different ways this could be done. We've opted to use a single comma-separated string of connection strings. In the `.env`, set `DATABASE_URL` as *all* of the connection URLs you have, separated by a comma. The **first** one should be the one for your **primary** (and possibly only) database. Follow it with those for any read-only regions you've added. It will look something like this. Note the comma between them:
```
DATABASE_URL='mysql://user1:password1@region1/name?ssl={"rejectUnauthorized":true},mysql://user2:password2@region2/name?ssl={"rejectUnauthorized":true}'
```
5. Run `npm start`.
6. You should be able to visit `http://localhost:3000` and see `hello world`: that confirms the app is running.
7. Visit `http://localhost:3000/read` and you should see a list of fruits fetched from PlanetScale. Note when run locally, this sample app does not know where you are in the world and so it can't pick the closest database. It will default to using the *first* connection string (which must be the primary).
8. Visit `http://localhost:3000/write` and you should see a row has been added to the PlanetScale database. Normally data is added by a non-GET request however using a GET makes it easier to check using a browser.
You've successfully connected to a PlanetScale database!
## Deploy the app to Fly
If you haven't already done so, [install the Fly CLI](/flyctl/install) and then [log in to Fly](/getting-started/sign-up-sign-in).
We've added a few files to make this sample app ready to run on Fly: a `Dockerfile`, `.dockerignore` and `fly.toml`. The `Dockerfile` tells Fly how to *package* the application. The `.dockerignore` file specifies the files/folders we want included (else it would include ones like `.env` that we do not want). The `fly.toml` file *configures* the application. The Fly CLI can make that for you. But since we know which variables, ports, services and so on our app needs, we made one.
**Note:** You'll need to make two changes to the provided `fly.toml`:
1. Update the app's name at the top to one of your choice `app = "your-name-here"`.
2. Update the region to the one nearest *your* primary database: `PRIMARY_REGION = "lhr"`. We created *our* database in `eu-west`. But Fly uses different region names than PlanetScale. And so you need to use the Fly region you would want your primary PlanetScale database to always be used from. As an example, let's say you have a PlanetScale database in *eu-west-1*. You plan on having a Fly app deployed to `lhr` and `iad`. You would set `PRIMARY_REGION = "lhr"` in the `fly.toml`, because that is the closest Fly region to the primary database's region.
To deploy the app, run `fly launch` from the application's directory.
The Fly CLI will spot the existing `fly.toml`:
```
An existing fly.toml file was found for app-name
? Would you like to copy its configuration to the new app? (y/N)
```
Type *y* (yes).
The CLI will then spot the `Dockerfile`:
```
Scanning source code
Detected a Dockerfile app
```
You'll be asked to give the app a name. Type a name using lowercase characters and hyphens, for example **your-app-here**.
You'll be prompted to pick an organization. Since every Fly user has a personal organization, let's pick that for now.
You'll be asked for the region to deploy the first vm to. That will be the same value you set for `PRIMARY_REGION` above. For example, `lhr`.
It will ask if you want a PostgreSQL database. Type *N* (no) as you are using PlanetScale for its database!
Finally you are asked if you want to deploy. Type *N* (no). Why not deploy now? This sample application expects a `DATABASE_URL` connection string to be available as a secret environment variable. Now that your app is staged on Fly we can set that. So run this command (setting its value as a comma-separated list of *all* of the connection strings you have from PlanetScale). The sample app expects the **first one** to be the connection string for your **primary database** (the region you created the PlanetScale database in). Any *subsequent* ones are the connection strings for any read-only regions. The resulting command you need to run will look something like this. Note the comma *between* the connection strings:
```bash theme={null}
fly secrets set DATABASE_URL='mysql://your-primary-region-one,mysql://optional-for-read-region,mysql://optional-for-another-read-region'
```
You should see `Secrets are staged for the first deployment`.
Now you can go ahead and deploy the app:
```bash theme={null}
fly deploy
```
You should see the build progress and after a couple of minutes the application will be successfully deployed:
```
==> Monitoring deployment
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 2 total, 2 passing]
--> v0 deployed successfully
```
If you open its URL `https://your-app-here.fly.dev` you should see the "hello world" text to confirm the app is working.
If you now open its *read* page `https://your-app-here.fly.dev/read` you should see a list of fruits read from the *closest* PlanetScale database. And since the primary region is the closest, well it makes sense to use that:
```json theme={null}
{
"time": 15,
"usingFlyRegion": "lhr",
"usingPrimaryRegion": true,
"usingDatabaseHost": "random.eu-west-3.psdb.cloud",
"data": [
{
"id": 7,
"name": "grape"
},
{
"id": 6,
"name": "grape"
},
{
"id": 5,
"name": "lemon"
}
]
}
```
If you open its *write* page `https://your-app-here.fly.dev/write` you should see a random row of data has been written. Again, this request is handled by a vm in the primary region and so the write is sent to the *primary* PlanetScale database:
```json theme={null}
{
"time": 18,
"usingFlyRegion": "lhr",
"usingPrimaryRegion": true,
"usingDatabaseHost": "random.eu-west-3.psdb.cloud",
"data": "Added a row with ID 8"
}
```
## Scaling the app
So far the app has *only* used the primary database. Since there is only *one* vm.
Now we'll test the read-only regions. Start by permitting the app to run in more regions. You can see which regions the app can run in using:
```bash theme={null}
fly regions list
```
You'll see something like:
```
Region Pool:
lhr
Backup Region:
```
Adjust that to permit more regions. For our test database, we made a primary in `eu-west-1` (Ireland), and two read-only regions in `us-west-2` (US) and `ap-southeast-2` (Australia). Suitably distributed to test the latency. And so we'll adjust the regions our sample Fly app can run in to use roughly matching Fly regions:
```bash theme={null}
fly regions set lhr sea syd
```
Now we need to scale the app to put a vm in each of those regions:
```bash theme={null}
fly scale count 3 --max-per-region=1
```
**Note:** If you run `fly scale show` you may see `Max Per Region: Not set`. This is a known issue however the value *is* set.
Confirm the new vms have been placed by waiting a few moments and then running `fly status`. Wait a minute until all three have a status of `running`:
```
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
abcdefgh app 2 syd run running 2 total, 2 passing 0 33s ago
abcdefgh app 2 sea run running 2 total, 2 passing 0 33s ago
abcdefgh app 2 lhr run running 2 total, 2 passing 0 1m2s ago
```
**Note:** You may want to run `fly logs`. If you have not edited our sample app to remove our debug logs, you should see each vm making a choice at run-time for which database it should connect to. For example for the vm launched in `sea` *we* saw *... the closest available database should be this one: us-west.connect.psdb.cloud*. For the vm launched in `syd` we *instead* saw: *...the closest available database should be this one: aws-ap-southeast-2.connect.psdb.cloud*. Perfect! Each of those vms connected to the *closest* database. You may need to adjust your region preferences (please take a look at `database.js` to see how we picked them).
Back to the test.
Now those two new vms are running, try calling the `https://app-name-here.fly.dev/read` and the `https://app-name-here.fly.dev/write` again. This time look at the value of `"usingFlyRegion"` in the response to see where the request was handled. If that value is your primary region (in our case that was `lhr`) any read-only regions *won't* be used. As there is no *need* for them to be. So to test any read-only regions you need to make a request from somewhere *else*, closer to one of the other vms.
In our case, we want to make a request that will be served by the vm we have in the US. That could be done in a variety of ways. You could use an online tool. We opted to use another Fly app we have to simulate requests from a particular location. It has a vm in the `sjc` (US) region. We connected to that vm using `fly ssh console` and ran a couple of curl commands ...
#### Read
```
#curl https://app-name-here.fly.dev/read
{"time":19,"usingFlyRegion":"sea","usingPrimaryRegion":false,"usingDatabaseHost":"us-west.connect.psdb.cloud","data":[{"id":9,"name":"lemon"},{"id":8,"name":"blueberry"},{"id":7,"name":"grape"}]}
```
Perfect! We can see from that response that this request worked. It was handled by the vm in `sea` (the closest vm *this* app has to our simulated user in `sjc`). The response shows which database it fetched the data about our fruits from: the read-only region `us-west.connect.psdb.cloud`! This confirms it didn't need to connect to the primary database, far away in Europe. A fast, low-latency read.
#### Write
```
#curl https://app-name-here.fly.dev/write
#{"time":20,"usingFlyRegion":"lhr","usingPrimaryRegion":true,"usingDatabaseHost":"random.eu-west-3.psdb.cloud","data":"Added a row with ID 10"}
```
Perfect! We can see from that response that this request worked. It was handled in our primary region (`lhr`). And so the write succeeded. Despite the closest database to `sea` actually being a read-only replica. But wait ... how does this work? Well, we can see by looking at the logs for this app:
```
2022-05-19T19:09:42Z app[abcdefgh] sea [info]Replaying this attempt to write to a read replica (from Fly region: sea) in the primary Fly region: lhr
```
That line is logged by our app's custom error handler, in `server.js`. Initially the write request was received by the closest vm to the simulated user (in the US). And that vm connected to the *closest* database (in the US). But ... that's a read-only replica. And so that write failed, throwing an error. That error was caught by the error handler, analysed and matched. And so the app used some Fly magic to *replay* the HTTP request *in* the primary region (in our case that's `lhr`). And a vm in that region connects to the primary database. Which *can* of course be written to. And so the write worked.
## How does it know which database is closest?
If there is only one connection string in the `DATABASE_URL`, no decision needs to be made. That is used regardless of where in the world the app is running. That would be the case if you don't have any read replicas.
If there are multiple connection strings in the `DATABASE_URL`, the app needs to decide which one of them it should connect to. If you take a look at our sample app's `database.js` you will see it takes a *rough* guess at which regions it would *prefer* to connect to (ideally the closest). And then it sees which of those regions actually has a database. By looking for a connection string that uses a host in that region. If it can't find one, it defaults to connecting to the primary.
If your app only uses a couple of regions, it would probably be simpler for you to instead have *more* environment variables and *less* code. You could add an environment variable per region, such as `DATABASE_URL_LHR`, `DATABASE_URL_FRA` ... and so on. And then pick the appropriate one based on the value of `FLY_REGION` that is provided at run-time as an environment variable. You don't have to do it the same way we did!
## Issues
If you try deploying the app to Fly and that fails, the most *likely* reason is your `DATABASE_URL` contains at least one invalid connection string. When each vm starts, the app tries to connect to the database. If it can't (such as the provided password is wrong) it will throw an error. You can confirm that is the case by running `fly logs` after a failed deploy. If you see a line such as `sea [info]Error: Access denied for user 'username-here'` please check all your connection strings are correct and there is no space between the commas.
## Notes
* If you have any read-only replicas and use this approach (replaying a write to them using Fly's `fly-replay` header) make sure to not enable auto-scaling for your app. There is currently an issue where vms are not evenly distributed and so you can be sure there will be a vm in your primary region. And if not, writes to the database will fail.
* For this example we used the same branch for local development *and* on Fly. In reality you would have a development branch and a separate production branch. As such, the connection string(s) in your local `.env` file would differ from the one your application uses on Fly. And in fact would likely instead be a local installation of MySQL to reduce costs.
* Adding additional regions in PlanetScale increases the cost of storage. For example if your primary database is 10GB, adding an additional region (for a read replica) would double the cost of storage.
* Make sure to keep an eye on usage. If you exceed your included monthly allowance you are billed an additional fee:
* Since the usage is based on rows read (not rows fetched) make sure all of your queries are optimised. If they don't use an index, they could result in a table scan which will increase your costs.
* Your application may *not* benefit from adding read-only regions if it does substantially more writes than reads. In which case the saving on latency for reads may not justify the increased storage cost.
## View your application on Fly
Use `fly apps open` as a shortcut to open the app's URL in your browser.
Use `fly logs` to see its log files.
Use `fly status` to see its details:
```
App
Name = your-app
Owner =
Version = 1
Status = running
Hostname = your-app.fly.dev
Deployment Status
ID = 67c6d3a4-963f-46cf-b694-d7e28aea6600
Version = v1
Status = successful
Description = Deployment completed successfully
Instances = 1 desired, 1 placed, 1 healthy, 0 unhealthy
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
abcdefgh app 1 lhr run running 2 total, 2 passing 0 0h1m ago
```
## Delete the app
To avoid charges you might want to delete the app and the PlanetScale database:
* To delete the sample Fly app (this is irreversible), run `fly apps destroy the-app-name-here`
* To delete the PlanetScale database, click the red delete button within the database's *Settings* tab:
# Postgres: A single server deployment for Fly
Source: https://docs.fly.io/app-guides/postgres
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Postgres on Fly
Deploy a Postgres database on Fly, ready for your Fly apps.
# Rationale
Postgres is a useful database to have as part of your Fly installation. To make it simple to install, this example takes you through all the steps necessary.
The resulting Postgresql installation will available over the Fly 6PN internal network to other applications and to users who have configured a Fly Wireguard VPN on their local system.
This Postgres configuration is good for development and small services. For mission-critical workloads, you'll want a HA configuration.
## Initialize the Fly app
Create your `fly.toml` file by importing the template from the repository. Run:
`fly init --import fly.source.toml`
You will be prompted for an app name and organization. You can hit return to generate an app name when asked.
Do make a note of the name of the app and the region that it will initially deploy to. You'll need the app name to connect to it, and more immediately, the region for the next step.
## Create a volume
Postgres needs to be able to store and persist data on disk. For that Fly has volumes which can be created in all of the Fly regions. Each volume is region-specific and is identified by name.
`fly volumes create postgres --region lhr`
Would create a 10GB volume named `postgres` in the `lhr` region. Add `--size 20` to create a 20GB volume.
## Set an initial password
When Postgres initialised the database for the first time, it will use the `POSTGRES_PASSWORD` to set the password for the `postgres` super-user. To pass that to the database, we set a Fly secret with whatever password we have chosen.
`fly secrets set POSTGRES_PASSWORD=`
## Deploy Postgres
We are now ready to deploy Postgresql. Run:
`fly deploy`
## Connecting to Postgres
As configured, this Postgres has no direct external connectivity.
### Other Fly Apps
When 6PN private networking is enabled for an app, the Postgres instance will be accessible on `.internal`. It will not require TLS as connections between apps within Fly are already encrypted.
### Wireguard
If you have configured Fly's Wireguard, then you can securely connect to the Postgres server using that. The address of the server will be `.internal`. For example, your first connection to your new Postgresql with psql would be:
`psql postgres://postgres:@appname.internal/`
## Notes
* This configuration has no direct external connectivity and is specifically for the use of other Fly apps
* This configuration is good for development and small services.
## Discuss
* You can discuss this example on the [postgres example topic](https://community.fly.io/t/new-postgres-example/458) on community.fly.io.
# Puppeteer JS Renderer
Source: https://docs.fly.io/app-guides/puppeteer-js-renderer
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Puppeteer JS Renderer on Fly
JavaScript is the bane of a web scraper's life. Scraping is all about extracting data from a web page and JavaScript is there adding content, hiding blocks, and moving the DOM around. Reading the HTML from the server is just not enough. What you ideally want is a way to run all that JavaScript on the page so you can see what's left after that. Then you can get down to some serious scraping.
There are tools to do this out there but most have their own complications or restrictions that make them difficult to deploy. Puppeteer has none of those problems and with [Fly](https://fly.io), you can deploy puppeteer based applications close to your users. At its core, this project is a puppeteer-based service. [Puppeteer](https://pptr.dev/) is a package that renders pages using a headless Chrome instance, executing the JavaScript within the page.
## Quick Start
A typical YouTube page will add the views count to itself only when the JavaScript on that page executes. These are the types of use cases where Puppeteer comes in very handy. Puppeteer can execute the JavaScript on the YouTube page and, from the rendered page, extract the view count and the title of the video.
Lets get started
### Install example project
To install this project locally, execute the following commands:
```
git clone https://github.com/fly-examples/puppeteer-js-renderer.git
cd puppeteer-js-renderer
npm install
```
### Your first scraper
To quickly try the project out after installing it, run the command below:
```
node yt-views.js
```
This will show something like below:
```
Pulling views from YouTube, please wait...
Luis Fonsi - Despacito ft. Daddy Yankee has 6,881,463,846 views
```
If you want to try any other video just pass the YouTube video URL as a parameter to the script like below:
```
node yt-views https://www.youtube.com/watch?v=XqZsoesa55w
```
It should give you an output like this:
```
Pulling views from YouTube, please wait...
Baby Shark Dance | Sing and Dance! | @Baby Shark Official | PINKFONG Songs for Children has 6,077,338,169 views
```
### What just happened here?
To show the YouTube video views count, a small scraper written with [Axrio](https://www.npmjs.com/package/@geshan/axrio) npm package was executed. Axrio combines the popular [Axios](https://www.npmjs.com/package/axios) library and [Cheerio](https://www.npmjs.com/package/cheerio) to create a mini scraper. Axios is used to make requests and Cheerio acts like DOM navigator parsing the markup and giving us an API for traversing/manipulating the resulting data structure. You can kickstart a small scraper with Axrio too.
The `yt-views.js` is a basic scraper which performs a GET request for the given YouTube URL with Puppeteer. This will return the final DOM after the page's JavaScript executes. Then it parses the title and views count out of the rendered page's markup and prints it on the console.
It uses the puppeteer-js-renderer service that is already deployed and running on Fly.io. Have a look at he "[how to use it as a service](#how-to-use-it-as-a-service)" section for more information. To start your own instance on Fly.io jump to the "[how to deploy on fly.io](#how-to-deploy-it-on-fly-io)" section.
## Run locally
If you have node installed on your machine, you already cloned this repository and ran `npm install`. Now run `npm start` to get this service running locally.
The next step is to navigate to `http://localhost:8080/api/render?url=https://www.youtube.com/watch?v=kJQP7kiw5Fk` on your browser to view the JavaScript rendered output.
### Run with Docker
If you want to run with Docker, execute the following after you clone the repository:
```sh theme={null}
cd puppeteer-js-renderer
docker-compose up
```
Then go to `http://localhost:8080/api/render?url=https://www.youtube.com/watch?v=kJQP7kiw5Fk` in your browser to see the output.
## How to use it as a service
If you want to use puppeteer-js-renderer for scraping, you can use the following URL on Fly.io:
```
https://js-renderer-fly.fly.dev/api/render?url=https://www.youtube.com/watch?v=kJQP7kiw5Fk
```
The YouTube URL is an example above, you can use any URL that needs javascript to be executed to be functional.
### Styles broken
Styles and images will look broken but the HTML tags will be there. Happy Web Scraping!
## How to deploy it on fly.io
[Fly.io](https://fly.io) is a platform for applications that need to run globally. It runs your code close to users and scales compute in cities where your app is busiest. Write your code, package it into a Docker image, deploy it to Fly's platform, and let that do all the work to keep your app snappy.
Fly.io has great [documentation](/) to get started. You can find a quick speed run showing how to get your app running closer to your users with this [guide](/getting-started/launch).
Please follow these steps to deploy your own puppeteer-js-renderer service on Fly.io:
### Prerequisites
1. [Install](/flyctl/install) the flyctl CLI command.
2. Register on fly with `flyctl auth signup`, if you already have a fly account log in with `flyctl auth login`.
### Steps
1. Clone this repo with `git clone git@github.com:fly-examples/puppeteer-js-renderer.git` if you are logged in with SSH support enabled. Otherwise try `git clone https://github.com/fly-examples/puppeteer-js-renderer.git`.
2. Then run `cd puppeteer-js-renderer`.
3. After that execute `flyctl init --dockerfile` and when asked for an app name, hit return to have one generated (unless there's a name you really want). I ran it with js-renderer-fly as the app name for this example.
4. Subsequently, you can select an organization. Usually this will be your first name-last name on the prompt.
5. It should create a fly.toml file in the project root (I have not committed it, it is in .gitignore).
6. Now run `flyctl deploy` to deploy the app. It will build the docker image, push it to the fly docker image registry and deploy it. In the process of deploying it, it will display information about the number of instances and their health.
7. You can then run `flyctl info`. This will give the details of the app including hostname.
8. You can view your app in the browser with `flyctl open`. For me, it opened `https://js-renderer-fly.fly.dev` and displays `{message: "alive"}`.
9. Add `/api/render?url=https://www.youtube.com/watch?v=kJQP7kiw5Fk` on the address bar of your browser after your puppeteer-js-renderer URL. Mine looked like `https://js-renderer-fly.fly.dev/api/render?url=https://www.youtube.com/watch?v=kJQP7kiw5Fk`. This will render the final DOM of that YouTube page after the JavaScript is executed.
10. Enjoy!
The YouTube video URL is an example and it can, of course, be replaced with other web pages which need JavaScript to work.
## More fly commands
You can suspend your service with `flyctl suspend`; this will pause your service until you resume it. If you try `flyctl status` after suspending it will not show any instances running. Suspending the service means all resources allocated to run the service will be deallocated, resulting in the application running nowhere. To get the instances running again execute `flyctl resume`.
### Fly default resources
I wanted to see what resources were allocated to the App on Fly by default. The scale commands allowed me to find it out pretty easily like below:
1. `flyctl scale show` - showed me VM Size: micro-2x
2. `flyctl scale vm` - showed me micro-2x is a 0.25 CPU cores with 512 MB of memory.
If you want to increase CPU/memory or run more instances in a particular region please refer to the official Fly docs on [scaling](/launch/scale-count).
### Flying the app on 3 continents
Our service is running in one data center. For me, it's iad (Ashburn, Virginia) but yours will likely be different based on where you are working from. We can add instances around the world to speed up responses, let's get rolling:
1. To see the regions available run `flyctl platform regions`, I could see regions all over the world from Oregon to Sydney.
2. Let's add an instance to Australia in Sydney, to do this run `flyctl regions add syd`, yes it is that easy.
3. Now check `flyctl status` and you will see an instance running in Sydney
4. Let’s add one more in Europe in Amsterdam with `flyctl regions add ams`. So now we are mostly covered with the app running in 3 continents.
5. Of course, you can run `flyctl status` again to see your app shining on 3 continents.
# Redis - standalone Redis Server
Source: https://docs.fly.io/app-guides/redis
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
The official repository for Running Redis on Fly.io. Find the accompanying Docker image at [flyio/redis](https://hub.docker.com/repository/docker/flyio/redis).
## Usage
This installation requires setting a password on Redis. To do that, run `fly secrets set REDIS_PASSWORD=mypassword` before deploying. Keep
track of this password - it won't be visible again after deployment!
If you need no customizations, you can deploy using the official Docker image. See `fly.toml` in this repository for an example to get started with.
## Runtime requirements
By default, this Redis installation will only accept connections on the private IPv6 network, on the standard port 6379.
If you want to access it from the public internet, add a `[[services]]` section to your `fly.toml`. An example is included in this repo for accessing Redis on port 10000.
We recommend adding persistent storage for Redis data. If you skip this step, data will be lost across deploys or restarts. For Fly apps, the volume needs to be in the same region as the app instances. For example:
```bash cmd theme={null}
flyctl volumes create redis_server --region ord
```
```out out theme={null}
Name: redis_server
Region: ord
Size GB: 10
Created at: 02 Nov 20 19:55 UTC
```
To connect this volume to the app, `fly.toml` includes a `[mounts]` entry.
```
[mounts]
source = "redis_server"
destination = "/data"
```
When the app starts, that volume will be mounted on /data.
## Cutting a release
If you have write access to this repo, you can ship a prerelease or full release with:
```
scripts/bump_version.sh
```
or
```
scripts/bump_version.sh prerel
```
# Appkata: Redistls - Redis Server with TLS
Source: https://docs.fly.io/app-guides/redistls
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Redis (with TLS)
Deploying Redis with TLS encryption using Fly.
[Read this guide on Fly.](/app-guides/redistls)
## Rationale
A key/value database like Redis is useful to have to support caching, session management and, well, anything which needs simple fast storage. That's why there's Redis already built into Fly. But there are some things that Redis configuration doesn't do, like Publish and Subscribe, and that's when you want to deploy your own Redis on Fly.
For this example, we are going to customize a Redis docker image to tune it for running on Fly and deploy it with persistent disk storage for Redis to save its data on.
## Preparing
There's a couple of components to this example. We're going to use the official Redis image, `redis:alpine`, but we want to change some system settings before Redis starts running. To do that, we'll use a script, `start-redis-server.sh`
```shell theme={null}
#!/bin/sh
sysctl vm.overcommit_memory=1
sysctl net.core.somaxconn=1024
# pass env vars as server args
redis-server /usr/local/etc/redis/redis.conf --requirepass $REDIS_PASSWORD
```
The two `sysctl` calls set up the environment so that Redis doesn't throw warnings about memory and connections. The script then starts up the Redis server, giving it a password to require and a config file to boot with. The Redis config is simple – it defines IPs to listen on, persistence options:
```conf theme={null}
# listen on all ipv4 and ipv6 addresses
bind 0.0.0.0 ::
# write to /data/
dir /data/
# appendonly settings
appendonly yes
auto-aof-rewritepercentage 100
auto-aof-rewrite-min-size 64mb
```
Now we need to make those changes apply to a new Redis deployment. For that we use this `Dockerfile`:
```dockerfile theme={null}
FROM redis:alpine
ADD redis.conf /usr/local/etc/redis/redis.conf
ADD start-redis-server.sh /usr/bin/
ADD certs /etc/certs
RUN chmod +x /usr/bin/start-redis-server.sh
CMD ["start-redis-server.sh"]
```
It adds our new shell script to the image, makes it executable, adds the redis config, and boots the container with the shell script.
With these two files in place we're ready to put Redis onto Fly.
## Configuring
First, we need a configuration file - and a slot on Fly - for our new application. We'll use the `fly init` command. The parameter is the app name we want - names have to be unique so choose a new unique one or omit the name in the command line and let Fly choose a name for you.
```bash cmd theme={null}
fly init redis-example
```
```out out theme={null}
Selected App Name: redis-example
? Select organization: Demo Sandbox (demo-sandbox)
? Select builder: Dockerfile
(Do not set a builder and use the existing Dockerfile)
? Select Internal Port: 6379
New app created
Name = redis-example
Organization = demo-sandbox
Version = 0
Status =
Hostname =
App will initially deploy to ord (Chicago, Illinois (US)) region
Wrote config file fly.toml
```
The important choices here are we select a Dockerfile (the one we created) as the builder for this app and we set the internal port to 6379, the default port for Redis. Do take note of which region here the init command says the app will initially deploy into. We'll need that in a moment.
This will generate a new fly.toml file which we'll need to edit. By default, the `fly.toml` takes the internal port and connects it to an HTTP only handler on port 80 and a combined TLS/HTTP handler on port 443. We don't need that at all, we just need a straight-through connection to the internal port.
Edit your generated `fly.toml`, removing both `[[services.ports]]` entries for ports 80 and 443 and replacing them with a single entry:
```
[[services.ports]]
handlers = []
port = "10000"
```
This will direct external traffic on port 10000 to internal port 6379. Save the file for now.
## Keeping a secret
Our script takes a password from an environment variable to secure Redis. We need to set that now using the `fly secrets` command. This encrypts the value so it can't leak out; the only time it is decoded is into the Fly application as an environment variable.
```
flyctl secrets set REDIS_PASSWORD=
```
And of course, remember that password because you won't be able to get it back.
## Persisting Redis
The last step is to create a disk volume for Redis to save its state on. Then the Redis can be restarted without losing data. For Fly apps, the volume needs to be in the same region as the app. We saw that region when we initialized the app; here it's `ord`. We'll give the volume the name `redis_server`.
```bash cmd theme={null}
flyctl volumes create redis_server --region ord
```
```out out theme={null}
Name: redis_server
Region: ord
Size GB: 10
Created at: 02 Nov 20 19:55 UTC
```
To connect this volume to the app, pop back to editing fly.toml and add:
```
[[mounts]]
source = "redis_server"
destination = "/data"
```
When the app starts, that volume will be mounted on /data.
## Deploy
We're ready to deploy now. Run `fly deploy` and the Redis app will be created and launched on the cloud. Once complete you can connect to it using the `redis-cli` command or any other Redis client. Just remember to use port 10000, not the default port.
```
redis-cli -h redis-example.fly.dev \
-p 10000 \
--askpass
```
You'll immediately be prompted to enter the password you previously set.
## Lock it down with TLS
So far, our Redis configuration is good for a demo. But running Redis on an unencrypted public port with password authentication is insecure.
Redis has built-in TLS support, we generate certificates and configure the server to require client certificate validation. This is roughly the equivalent security level as running behind a VPN.
Generating certs is simple with the excellent [`mkcert`](https://github.com/FiloSottile/mkcert) utility. Once you install `mkcert`, you'll get a local certificate authority to create your own certificates with.
Here are the commands you'll need to create certificates for this Redis example:
1. Create `certs` directory for the Docker image:
```
mkdir -p certs
```
2. Generate server certificates:
```
mkcert -key-file certs/redis-server.key -cert-file certs/redis-server.crt redis-example.fly.dev
```
3. Generate a client certificate:
```
mkcert --client -key-file redis-client.key -cert-file redis-client.crt redis-example.fly.dev
```
4. Copy the root CA certificate:
```
cp "$(mkcert -CAROOT)/rootCA.pem" certs/rootCA.pem
```
When you're done you should see a directory layout like this:
```
├── certs
│ ├── redis-server.crt
│ ├── redis-server.key
│ └── rootCA.crt
├── redis-client.crt
└── redis-client.key
```
Then, we can pop into our `redis.conf` to configure TLS by adding:
```conf theme={null}
tls-port 7379
tls-cert-file /etc/certs/redis-server.crt
tls-key-file /etc/certs/seredis-server.key
tls-ca-cert-file /etc/certs/rootCA.pem
```
This configures Redis to listen on port 7379, which means changing the internal port in `fly.toml`:
```toml theme={null}
[[services]]
internal_port = 7379
...
```
Now run `fly deploy` again, and enjoy the relaxation of a locked-down Redis service.
The `redis-cli` CLI has support for TLS, you can connect with this command:
```
redis-cli -h redis-example.fly.dev \
-p 10000 \
--cert redis-client.crt \
--key redis-client.key \
--tls \
--cacert certs/rootCA.pem \
--askpass
```
As before, you will be prompted for a password, but now the connection has already been authenticated by the TLS handshaking exchanging the client certificate.
## Discuss
* You can discuss this example on the [community.fly.io](https://community.fly.io/t/new-redistls-appkata-example/405) topic.
# Global Image Processing Service
Source: https://docs.fly.io/app-guides/run-a-global-image-service
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
For many web applications, the ability to scale, crop and otherwise manipulate images is essential. One way of providing this capability is with a URL-request-driven image service which can retrieve images, process them and deliver them wherever needed.
But that also means provisioning the service globally to keep latency times low which can be its own administrative headache.
With Fly you can eliminate all that administrative load by letting Fly deploy a service on demand close to your users so they get a rapidly rendered experience. In this guide, we'll show you how to deploy a popular image processing application, Imaginary, into the Fly Global Application Platform to make your own Global Image Service.
## *What’s Imaginary*
Let's start with [h2non/Imaginary](https://github.com/h2non/imaginary), a fantastically useful image processing application written in Go. Once configured, it lets you process images. The images can be posted to it or you can ask it to get an image from a remote server and all of this is controlled through a URL request to the Imaginary server.
You can find all the source in the GitHub repository, but for this guide, we don't need to worry about that. We're going to use the developer-supplied Docker image to deploy it.
## *Deploying Docker Images to Fly*
How do you turn a Docker image into a globally available edge application? With Fly it's remarkably concise process.
One thing we know at Fly is that our users want to deploy more than JavaScript at the edge. Powerful though that is, they also want fully fledged application stacks and backend components to live at the edge too. Now they can, and in this series of guides we'll show you how to use them.
In this first guide, we're going to take an existing application and make it go global with Fly.
## *Creating the Dockerfile*
To use a Docker image, we'll need a Dockerfile. Create a new directory and create a `Dockerfile` in it with these contents.
```dockerfile theme={null}
FROM h2non/imaginary:1.1.1
ENV PORT "8080"
CMD ["-enable-url-source","-http-read-timeout" ,"3"]
```
If you aren't familiar with Docker, the first `FROM` line selects the image of the application and version we want to use. The Imaginary image is configured to run the `imaginary` command.
Imaginary, running in a Docker container, normally opens a port on port 9000, but for this exercise, we want to open its port on 8080, Fly's default port for external communications. The simplest way to do that is to set the PORT environment value to 8080.
The third `CMD` line contains the parameters we pass to the `imaginary` command that are going to configure how Imaginary runs. The `-enable-url-source` tells Imaginary it is allowed to retrieve images from around the internet and, just so it's well-behaved, we set the `-http-read-timeout` to 3 seconds (the default is 60 which is a long time to wait for an image).
## *Testing Locally*
You'll need to have Docker installed locally to test this locally. We won't go into the details of installing Docker - check out the [Docker site and guides](https://docs.docker.com/install/) for how to install. Assuming you've done this we can move on to testing.
#### Build a Docker Instance
To create a container instance using the Dockerfile, run:
```bash theme={null}
docker build -t localimaginary .
```
#### Run the Docker Instance
To start up the Docker container, run:
```bash theme={null}
docker run -p 8080:8080 localimaginary
```
Where does that -p 8080:8080 value come from? Well, it's exposing port 8080, the one we set in the `ENV` line in our docker file.
#### Processing an Image
To test our local image, let's process a file. In this case, we're going to crop an existing image from Imaginary's github repository.
In another terminal session, run:
```bash theme={null}
curl -O "http://localhost:8080/crop?width=500&height=400&url=https://raw.githubusercontent.com/h2non/imaginary/master/testdata/large.webp"
```
This will ask Imaginary to crop, to 100x100, an image stored in the Imaginary GitHub repository. This should result in a file, `large.webp` being written to disc which you can display using your preferred image viewer. MacOS users can simply `open large.webp` at the command line.
Open the cropped image. Take a moment to play with the width and height values in the URL we're querying and you will get different sized cropped images. Once you are happy the local build is working, you can return to the session where you started the docker image and ctrl-C to stop it.
We now have a working docker image. Now to globally deploy it with Fly.
## *Going Global*
For this you'll need flyctl, it's your CLI-powered remote control for your Fly applications.
### Installing `flyctl`
If you've already installed `flyctl`, move on to the next step. If not, hop over to [our installation guide](/flyctl/install).
### Sign In (or Up) for Fly
If you have a Fly account, all you need to do is [sign-in with `flyctl`](#sign-in-or-up-for-fly).
##### New User?
Welcome to Fly. To get your Fly account run:
```bash theme={null}
flyctl auth signup
```
This will take you to the sign-up page where you can either:
* **Sign up with Email**: Enter your name, email and password.
* **Sign up with GitHub**: If you have a GitHub account, you can use that to sign up. Look out for the confirmatory email we will send you which will give you a link to set a password; you'll need a password set so we can actively verify that it is you for some Fly operations.
##### Returning User
Run:
```bash theme={null}
flyctl auth login
```
Your browser will open up with the Fly sign-in screen. Enter your user name and password to sign in. If you signed up with GitHub, use the `Sign in with GitHub` button to sign in.
Whichever route you take you will be signed-up, signed-in and returned to your command line, ready to Fly.
## Preparing to fly
### Creating a `fly.toml` File
The flyctl app helps you handle the entire lifecycle of your Fly Global application. That all starts with creating the `fly.toml` file which will control how your application is configured when deployed. To do so, run:
```bash theme={null}
flyctl apps create
```
This will kick off a short dialogue with you about your new application. First up, it'll ask for an application name:
```output theme={null}
? App Name (leave blank to use an auto-generated name) flyimaginary
```
Your name has to be unique across all Fly users, so unless there's a good reason, go with an auto-generated name.
Next you'll be asked to select an organization. If this is your first time on Fly, there'll only be one organization - your own personal organization, just for your applications. Organizations are all about sharing apps and collaborating. For this guide, we don't need to worry about that, so we can use our personal organization.
```output theme={null}
? Select organization: Demo (demo)
```
Once the organization has been selected, `flyctl` gets to generating the app.
```output theme={null}
New app created
Name = flyimaginary
Owner = demo
Version = 0
Status =
Hostname =
Wrote config file fly.toml
```
So, what's in `fly.toml`? If you look in it, you'll find various configuration parameters. You can skip to Deploying if you want to.
```toml theme={null}
app = "flyimaginary"
[[services]]
internal_port = 8080
protocol = "tcp"
[services.concurrency]
hard_limit = 25
soft_limit = 20
[[services.ports]]
handlers = ["http"]
port = "80"
[[services.ports]]
handlers = ["tls", "http"]
port = "443"
[[services.tcp_checks]]
interval = 10000
timeout = 2000
```
As well as the app name in the file, there's a services section which sets what the internal port number exposed by the app is and how it should be handled. It also says this port should be handled as TCP.
That's followed by `services.concurrency` which sets soft and hard limits to the number of connections the application should take before being transparently scaled horizontally.
There are also two `services.ports` sections. The first says incoming connections on port 80 will be routed to port 8080 and handled as HTTP. The second, more interesting entry, says that incoming connections on port 443 will be handled as TLS terminated HTTP connections and then routed on as HTTP to port 8080.
There's also a `services.tcp_checks` section which is used to define how we do health checks on the application.
We can leave all of this completely untouched for this guide. We're ready to Fly globally.
### Deploying
For deployment, we go back to flyctl.
```bash theme={null}
flyctl deploy
```
Flyctl will find the application name from the `fly.toml` file automatically and start the deployment process. There'll be a fair amount of output as flyctl gets Docker to assemble the image, create a release of the image and then deploy and check it's running. When it is complete you'll see something like:
```output theme={null}
...
==> Creating Release
Release v0 created
==> Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
v0 is being deployed
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
v0 deployed successfully
```
The application is now deployed.
### Viewing Fly Apps
You'll need the URL for the deployed application to connect to it. Flyctl helps out here with the `flyctl info` command:
```bash cmd theme={null}
flyctl info
```
```output output theme={null}
App
Name = flyimaginary
Owner = demo
Version = 0
Status = running
Hostname = flyimaginary.fly.dev
Services
TASK PROTOCOL PORTS
app tcp 443 => 8080 [TLS, HTTP]
80 => 8080 [HTTP]
IP Addresses
TYPE ADDRESS CREATED AT
v4 77.83.140.214 5m36s ago
v6 2a09:8280:1:a93f:aeee:f28f:2c4e:2fb1 5m35s ago
```
### Testing Imaginary on Fly
This tells you all you need to know about your currently deployed application. If you take the `Hostname` you can use that in a URL request to the Fly deployment:
```bash theme={null}
curl -O "http://flyimaginary.fly.dev/crop?width=500&height=400&url=https://raw.githubusercontent.com/h2non/imaginary/master/testdata/large.webp"
```
Notice that we didn't need to specify port 8080 as we did with the local docker build. That's because port 80 on the host routes to port 8080. As an added bonus, we also got HTTPS support as port 443 also routes to port 8080, with Fly managing the TLS termination and the Let's Encrypt certificates. We can do the same query, over a secured connection, with:
```bash theme={null}
curl -O "https://flyimaginary.fly.dev/crop?width=500&height=400&url=https://raw.githubusercontent.com/h2non/imaginary/master/testdata/large.webp"
```
## *Summary*
In this guide we created a Docker image, based on a third-party image, tested it locally then deployed it to the Fly platform using `flyctl`. You now have a Global Image Service. Whenever you request it processes an image, Fly will create an instance of the service closest to you to fulfill your request.
# Run A Private DNS Over HTTPS Service
Source: https://docs.fly.io/app-guides/run-a-private-dns-over-https-service
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
## *Why DNS over HTTPS?*
DNS over HTTPS (or DoH or 🍩) is a protocol that makes browsing more private. Browsers typically resolve domain names with an unencrypted protocol, allowing nosy neighbors and internet providers to snoop on some internet activity. DoH creates an encrypted connection between browsers and the DNS resolver to make it difficult to even see what domains a user is loading.
Firefox now [uses DNS over HTTPS by default](https://blog.mozilla.org/blog/2020/02/25/firefox-continues-push-to-bring-dns-over-https-by-default-for-us-users/) in the US and [DNS over HTTPS can easily be enabled](https://support.mozilla.org/en-US/kb/firefox-dns-over-https) elsewhere in the world. With the release of version 83, Chrome has gained official support for DNS over HTTPS (see this [Chrome Blog post](https://blog.chromium.org/2020/05/a-safer-and-more-private-browsing-DoH.html) for details). And Microsoft is planning to enable [DNS over HTTPS in Windows 10](https://techcommunity.microsoft.com/t5/networking-blog/windows-will-improve-user-privacy-with-dns-over-https/ba-p/1014229).
It's useful to be able to run private services for protocols like DoH. If you pipe all your encrypted DNS requests to one shared provider, you're essentially just swapping one nosy ISP for an even bigger internet provider. Fly is an especially good place to run DNS services because we deploy apps globally and the internet is faster when DNS is close by. And if you want to, you can even [manage your own TLS](#handle-your-own-tls).
## *Prerequisites*
### Install Fly CLI
If you're on a Mac, you can install the CLI with Homebrew:
```bash theme={null}
brew install flyctl
```
For other systems, download and run the install script:
```bash theme={null}
curl -L https://fly.io/install.sh | sh
```
### Sign In (or Up) for Fly
If you have a Fly account, all you need to do is [sign-in with `flyctl`](#returning-user).
#### New User?
Welcome to Fly. To get your Fly account run:
```bash theme={null}
flyctl auth signup
```
This will take you to the sign-up page where you can either:
* **Sign up With Email**: Enter your name, email and password.
* **Sign up With GitHub**: If you have a GitHub account, you can use that to sign up. Look out for the confirmatory email we will send you which will give you a link to set a password; you'll need a password set so we can actively verify that it is you for some Fly operations.
#### Returning User
Run:
```bash theme={null}
flyctl auth login
```
Your browser will open up with the Fly sign-in screen. Enter your user name and password to sign in. If you signed up with GitHub, use the `Sign in with GitHub` button to sign in.
Whichever route you take you will be signed-up, signed-in and returned to your command line, ready to Fly.
### Create a Fly App
Pick a name for your new DNS over HTTPS service. Then create a Fly app:
```bash cmd theme={null}
flyctl apps create
```
```output output theme={null}
? App Name (leave blank to use an auto-generated name)
> [your-app-name]
✔ New app created
Name = [your-app-name]
Owner = fly
Version = v0
Status =
Created fly.toml
```
## *Two-minute easy mode*
We've published a [Docker image](https://hub.docker.com/r/flyio/doh-proxy) you can use to get started.
### 1. Deploy `flyio/doh-proxy` from Docker Hub
```bash cmd theme={null}
flyctl deploy -i flyio/doh-proxy:0.1.19
```
```output output theme={null}
==> Validating app configuration
--> done
Services
TCP 80/443 ⇢ 8080
==> Creating Release
Release v0 created
==> Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
v0 is being deployed
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
v0 deployed successfully
```
### 2. Use `https://[your-app-name].fly.dev/dns-query`
Your new Fly app includes dedicated IP addresses and a `[your-app-name].fly.dev` hostname with a valid certificate. You can see these by running `flyctl info`:
```bash cmd theme={null}
flyctl info
```
```output output theme={null}
App
Name = doh-proxy
Owner = fly
Version = 10
Status = pending
Hostname = https://[your-app-name].fly.dev
Services
TASK PROTOCOL PORT INTERNAL PORT HANDLERS
app tcp 80 8080 http
app tcp 443 8080 tls http
IP Addresses
ADDRESS TYPE
77.83.140.25 v4
2a09:8280:1:68a1:90f6:e320:d275:70f7 v6
```
The URL for DNS queries is: `https://[your-app-name].fly.dev/dns-query`, you can use it in any app that supports DNS over HTTPS.
You can also try it out with `curl`:
```bash cmd theme={null}
curl -D - -sS "https://doh-proxy.fly.dev/dns-query?ct&dns=q80BAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE" | strings
```
```output output theme={null}
HTTP/2 200
content-length: 56
content-type: application/dns-message
x-padding: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cache-control: max-age=43200
date: Wed, 11 Sep 2019 15:08:54 GMT
server: Fly/a04ef97 (Wed, 11 Sep 2019 11:18:30 +0000)
example
```
## *Hard mode (maybe 10 minutes)*
### 1. Write a Custom `Dockerfile`
We use Docker to build and package applications. For custom builds, you'll need a `Dockerfile` in your working directory — here's a good one for `doh-proxy`:
```dockerfile theme={null}
# ------------------------------------------------------------------------------
# Cargo Build Stage
# ------------------------------------------------------------------------------
FROM rust:latest as cargo-build
ARG VERSION=0.1.19
RUN apt-get update
RUN apt-get install musl-tools -y
RUN rustup target add x86_64-unknown-linux-musl
RUN RUSTFLAGS=-Clinker=musl-gcc cargo install doh-proxy --version $VERSION --root /usr/local/ --target=x86_64-unknown-linux-musl
# ------------------------------------------------------------------------------
# Final Stage
# ------------------------------------------------------------------------------
FROM alpine:3.10
RUN apk add --no-cache libgcc runit shadow curl
COPY --from=cargo-build /usr/local/bin /usr/local/bin
RUN useradd doh-proxy
USER doh-proxy
ENV LISTEN=0.0.0.0:8080
ENV RESOLVER=9.9.9.9:53
RUN /usr/local/bin/doh-proxy --version
CMD ["sh", "-c", "/usr/local/bin/doh-proxy --listen-address $LISTEN --server-address $RESOLVER"]
```
### 2. Deploy working directory with `flyctl`
The Fly CLI will build + deploy a Docker image in one step. Just run `flyctl deploy` in your working directory. If Docker is running locally, we'll do a local build. If you don't have docker installed, we copy the contents of your working directory and perform a build on our servers.
```bash cmd theme={null}
flyctl deploy
```
```output output theme={null}
==> Validating app configuration
--> done
Services
TCP 80/443 ⇢ 8080
Deploy source directory '.'
Docker daemon available, performing local build...
Using Dockerfile from working directory: Dockerfile
Step 1/15 : FROM rust:latest as cargo-build
...
Step 6/15 : RUN RUSTFLAGS=-Clinker=musl-gcc cargo install doh-proxy --version $VERSION --root /usr/local/ --target=x86_64-unknown-linux-musl
...
Step 7/15 : FROM alpine:3.10
...
Step 9/15 : COPY --from=cargo-build /usr/local/bin /usr/local/bin
...
Step 15/15 : CMD ["sh", "-c", "/usr/local/bin/doh-proxy --listen-address $LISTEN --server-address $RESOLVER"]
---> Running in 427a770ed8a5
---> ee696da27723
Successfully built ee696da27723
Successfully tagged registry.fly.io/srcproxy:deployment-1580379849
Image size: 18 MB
==> Pushing image
The push refers to repository [registry.fly.io/srcproxy]
75b521405184: Pushed
ec42405b9867: Pushed
fecdd1368b2a: Pushed
531743b7098c: Pushed
deployment-1580379849: digest: sha256:220099d5aae94190a206bfdda219480ffbf8da988690361ae6d1d280732d8a68 size: 1158
--> done
==> Creating Release
Release v0 created
==> Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
v0 is being deployed
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
v0 deployed successfully
```
This could take a while on the first one. Subsequent runs will reuse image layers and finish quite a bit faster.
## *Optional configuration*
### Use a Different Resolver
The `doh-proxy` project defaults to using the [Quad9\* DNS resolver](https://www.quad9.net) at `9.9.9.9:53`. To change this, you can add an environment variable:
```bash cmd theme={null}
flyctl secrets set RESOLVER=8.8.8.8:53
```
```output output theme={null}
VERSION REASON DESCRIPTION USER DATE
v10 Secrets updated flydev@fly.local 0s ago
```
### Change Hostnames
Fly apps support an unlimited number of hostnames and SSL certificates. To set one up:
1. Add a certificate to your fly app:
```bash cmd theme={null}
flyctl certs create example.com -a doh-proxy
```
```output output theme={null}
Hostname = example.com
Configured = false
Certificate Authority = lets_encrypt
DNS Provider = nsamesystem
DNS Validation Target = example.com.q8o1.flydns.net
Source = fly
```
2. Create a validation CNAME entry:
* Name: `_acme-challenge.example.com.`
* Target: `example.com.q8o1.flydns.net`
3. Make a DNS entry pointing example.com to your Fly app IP
### Handle your own TLS
Fly can take care of TLS + HTTPS for you, but it doesn't have to. If you'd prefer to terminate your own TLS, you can generate your own certificates, store them as encrypted Fly secrets, and tweak [the Dockerfile](#1-write-a-custom-dockerfile) to pull them from your app environment. We'll happily forward encrypted TCP to your app and let you worry about the rest.
# Smokescreen - Sanitize outbound web requests
Source: https://docs.fly.io/app-guides/smokescreen
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# Smokescreen
Managing your outgoing web connections from Fly with Smokescreen
## Rationale
This example runs [Stripe's Smokescreen proxy](https://github.com/stripe/smokescreen) on Fly, with appropriate modifications to add basic password control.
It is always worthwhile to control the outgoing traffic from your other applications. If your apps call other systems with user-entered URLs, say for triggering webhooks or reading responses from an API, then there is a possibility that that feature could be abused. A bad actor could enter URLs designed to access resources inside your application's private network, giving it the names of known machines or well-known private IP addresses. Depending on how your app responds may give that bad actor a clue on how your application works and from that, possibly stage an attack.
To handle this, Stripe created (and open-sourced), Smokescreen, an outbound proxy that makes sure that requests to the outside world from your applications aren't trying to probe your internal network. Out of the box, Smokescreen will check any request isn't destined for 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16" or fc00::/7. There's a lot more that Smokescreen can do around roles, ACLs, and TLS certificates, but that is outside the scope of this example.
## The Dockerfile
One thing with Smokescreen is that there is no binary executable distributed for it. That means it has to be built to deploy it. This example comes with a Dockerfile that does a multi-stage build - first compiling the code, then moving the executable to a clean image to run.
## Deploying on Fly
The quickest way to initialize the app is to import the `fly.source.toml` file supplied:
```
fly init smokescreen-example --import fly.source.toml
```
Replace `smokescreen-example` with your preferred app name (`yourappname`) or omit it to have Fly generate a name for you. You may be prompted for which organization you want the app to run in. Ensure that it is running in the same organization as the apps you wish to manage their external calls.
Smokescreen runs on port 4750 and applications wishing to connect to it from within your Fly organization should send their requests to `yourappname.internal`. To connect to Smokescreen from outside the Fly environment, use [Fly's WireGuard](/blueprints/connect-private-network-wireguard) to create a VPN into your Fly organization.
This Smokescreen example includes support for a proxy password and an access control list.
Before you deploy Smokescreen, you'll need to set a Fly secret for the proxy password; if it is not set, Smokescreen will exit immediately.
```
fly secrets set PROXY_PASSWORD=yourpassword
```
To bring Smokescreen online, run:
```
fly deploy
```
## Testing
To test Smokescreen is running correctly, you can use `curl`. The -x option on curl tells it to use the following address and port as a proxy.
As Smokescreen has a password set, we also have to use the -U option to send our password. Therefore the command:
```bash theme={null}
curl -U anyname:yourpassword -x yourappname.internal:4750 https://fly.io
```
Would attempt to use the proxy to contact the secure version of the fly.io site. It'll echo back the contents of the front page.
Remember for this to work, you'll need to configure a Wireguard VPN into your Fly organization. If you don't, the `.internal` host name resolution will not work.
If an attempt was made to connect to `localhost`, as a network mapper may do, this would happen:
```bash theme={null}
curl -U anyname:yourpassword -x yourappname.internal:4750 http://localhost/
Egress proxying is denied to host 'localhost': The destination address (127.0.0.1) was denied by rule 'Deny: Not Global Unicast'. destination address was denied by rule, see error.
```
## Notes
* The configuration shown is an example, with a simple password lock on the authentication. As it currently stands, if the connection is "authed" (that is, has a valid password), then the ACL will let the connection take place and note it in the logs:
```yaml theme={null}
---
version: v1
services:
- name: authed
project: users
action: report
default:
project: other
action: enforce
```
The `enforce` action only allows connections to sites in the `allowed_domains` list (which as there aren't any, means it blocks all unauthorized connections). Read more about ACLs in the [Smokescreen README](https://github.com/stripe/smokescreen#acls).
The `authed` users requests have a `report` action which allows the connection to be made and logs information about that connection. Running `fly logs` will show entries like this:
```log theme={null}
2021-01-13T11:35:13.843Z be1e33dd lhr [info] time="2021-01-13T11:35:13Z" level=info msg=CANONICAL-PROXY-DECISION allow=true content_length=0 decision_reason="rule has allow and report policy" dst_ip="2a09:8280:1:a270:b8f5:b9f7:8891:f3" dst_port=443 enforce_would_deny=true id=bvvdlsd11c3s4uipn73g project=users proxy_type=connect requested_host="fly.io:443" role=authed source_addr="[fdaa:0:4:a7b:dc6:0:a:2]:53137" src_host="fdaa:0:4:a7b:dc6:0:a:2" src_host_common_name=unknown src_host_organization_unit=unknown src_port=53137 start_time="2021-01-13 11:35:13.865538649 +0000 UTC" trace_id=
2021-01-13T11:35:14.022Z be1e33dd lhr [info] time="2021-01-13T11:35:14Z" level=info msg=CANONICAL-PROXY-CN-CLOSE bytes_in=0xc000137310 bytes_out=0xc000137318 dst_ip="2a09:8280:1:a270:b8f5:b9f7:8891:f3" dst_port=443 duration=0.170710048 end_time="2021-01-13 11:35:14.062381945 +0000 UTC" error= id=bvvdlsd11c3s4uipn73g last_activity="2021-01-13 11:35:14.062118835 +0000 UTC" proxy_type=connect requested_host="fly.io:443" role=authed source_addr="[fdaa:0:4:a7b:dc6:0:a:2]:53137" start_time="2021-01-13 11:35:13.865538649 +0000 UTC" trace_id=
```
The first entry records the connection being allowed, the second the completion of that connection.
## Discuss
* Discuss the Smokescreen example on its [dedicated community.fly.io topic](https://community.fly.io/t/new-smokescreen-example/466)
# VS Code - Remote Server over SSH
Source: https://docs.fly.io/app-guides/vscode-remote
> [!WARNING]
>
> **Warning: This document is old! It is likely wrong in some important way.**
# VSCode-Remote
Deploy a remote development system for VS Code's Remote SSH on Fly.
## Rationale
When you get the need to develop but there's no machine to hand that's quite powerful enough, your next step may be to roll yourself a cloud instance and develop on there. With tools like VS Code's Remote SSH mode, remotely working with a development machine in the cloud can be a joy.
But then you have to go through the entire provisioning process every time you want a fresh machine to work with. Which is why using Fly can make it simpler. This example is all about making that process completely automated. At the end of deploying it, you'll have a Fly instance with attached disk storage, running Ubuntu, accessible by you and using certificate-based authentication, with Docker and whatever packages you select.
## Using
Get the code by git cloning the repository:
```
git clone https://github.com/fly-examples/vscode-remote.git
```
Then cd into the `vscode-remote` directory and run the configure script.
```bash cmd theme={null}
./configure.sh
```
```out out theme={null}
Enter desired app name or press return to have a name generated:
Enter desired organization name or press return to use your personal org:
Enter disk size in GB or press return for default 10GB:
Use Docker on remote machine (y/n):y
Any extra packages:clang htop make
```
We'll pause here to review the only interactive part of the process.
* First, there's a request for an app-name which is used for the instance's hostname. If you don't need a particular name, just hit return and Fly will generate an appname.
* That's followed by a request for an organization. Just hit return to put it into your personal organization.
* Then there's a request for the number of gigabytes of disk space you want attached to the instance. This will be your working space. Hit return to get the minimum (and default) of 10GB.
* If you want Docker's daemon and tools pre-installed, answer Y at the next question.
* You can add other Ubuntu packages to be pre-installed at this point. Hit return for none, or enter their names in a space-separated list.
Why is this pre-installation important you may ask. Remember that this is the image that we will start with and the image that Fly will reload when an instance is restarted or moved. Not pre-installing it means that, although you could install something, it would be completely transitory and disappear when the instance was restarted. Unless you made provision to install things on the persistent disk space (not simple for system packages). So thats why we ask about Docker and extra packages.
From this point on, the deployment process happens without intervention.
#### Creating the configuration
The script starts work by creating a template (`import.toml`) and using that to create a new Fly app. This template is custom created and includes copies of the public keys of the user which are then used to authenticate SSH logins to the app later. This then automatically configures the app to work with SSH on port 10022 externally and port 22 internally.
```
Importing configuration from import.toml
Importing port 22
New app created
Name = crimson-bush-6560
Organization = personal
Version = 0
Status =
Hostname =
App will initially deploy to lhr (London, United Kingdom) region
Wrote config file fly.toml
```
`The created`fly.toml\` file already has an entry in it for a persistent volume called data. That now needs to be created.
#### Creating a persistent volume
The script gets the region the app was created in (here `lhr`) and creates the volume in it.
```
ID: p0Kg0L0JRGjBluq1PD
Name: data
Region: lhr
Size GB: 10
Created at: 02 Dec 20 11:04 UTC
```
It is now able to go ahead and build the image for deployment.
#### Building the image
At this point, the script runs `fly deploy` which itself does two things; build the app's image and deploy the results.
The building is done by processing the `Dockerfile`.
```
Deploying crimson-bush-6560
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 10022 ⇢ 22
Deploy source directory '/Users/dj/temp/vscode-remote'
Docker daemon available, performing local build...
==> Building with Dockerfile
Using Dockerfile Builder: /Users/dj/temp/vscode-remote/Dockerfile
Step 1/17 : FROM ubuntu:bionic
---> 2c047404e52d
...
```
There's a lot more output that comes here with the creating and installation of all the packages required, along with any other packages requested. We've skipped it here, but rest assured we end up here next:
```
Step 17/17 : CMD ["/usr/sbin/sshd", "-D"]
---> Running in 765dba7b0c10
---> b63be4ff51c8
Successfully built b63be4ff51c8
Successfully tagged registry.fly.io/crimson-bush-6560:deployment-1606907053
--> Building with Dockerfile done
```
The next part sees the new image first pushed to the fly.io repository, and then after being processed and optimized, deployed to the Fly infrastructure.
```
Image: registry.fly.io/crimson-bush-6560:deployment-1606907053
Image size: 488 MB
==> Pushing Image
The push refers to repository [registry.fly.io/crimson-bush-6560]
...
deployment-1606907053: digest: sha256:99806ec0460487aaaa8d4899f02471e10c39e9d98a424efda58a9e63e7e2251d size: 2400
--> Done Pushing Image
==> Optimizing Image
--> Done Optimizing Image
==> Creating Release
Release v0 created
Deploying to : crimson-bush-6560.fly.dev
Monitoring Deployment
You can detach the terminal anytime without stopping the deployment
1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v0 deployed successfully
To use in VS Code, tell the remote-ssh package to connect to dj@crimson-bush-6560.fly.dev:10022
```
The very last thing we are told is the address we need to give to VS Code when we do "Remote-SSH -> Connect to Host". Open VS Code, bring up the command line (Command/Control-Shift-P), enter `remote-ssh` and select `Connect to Host`. You'll be prompted to `Select configured SSH host or enter user@host`. We'll do the latter, and enter (as per our example) `dj@crimson-bush-6560.fly.dev:10022`. You'll be prompted to confirm the fingerprint of the machine you are connecting to (hit return) and then a new VS Code window will appear, with everything - file system, language server, tools and terminals running on the remote machine.
## Inside the process
Now, at this point, if you're happy launching your own VS Code SSH instances on Fly and don't need to know more you can go. If you are curious on how this example is built, then read on.
### The Dockerfile
The heart of many Fly apps is the Dockerfile, which creates the image that will be sent up to Fly to run. In this case, it uses Ubuntu Bionic as a base then:
* installs a number of essential packages (SSH server) and anything selected by the user as an extra package.
* creates a user with a selected name.
* gives that user `sudo` privileges.
* installs Docker components if they were requested
* copies over SSH server configuration and keys
* sets up the image to run
The important part here is that variables, `USER`, `USE_DOCKER` and `EXTRA_PKGS` are used to configure some of that build behavior.
### The Script
The `configure.sh` script is responsible for getting the values of those variables. It asks the user running it for settings for the various values. It also asks for an appname (as `$appname`), an organization (as `$orgname`), and a disk size for the attached volume. Finally, it gathers the user's SSH public keys into a single string called AUTHORIZED\_KEYS. With all the values acquired, the script can create the app.
### Creating with a Template
To create the app, the script writes out a file called `import.toml`. It's a template for what we would like in our application when we create it. It sets up SSH's port 22 to appear as port 10022, adds a mount for a disk volume, and passes SSH AUTHORIZED\_KEYS as an environment variable.
Then it runs:
```
fly init $appname --import import.toml --org $orgname --overwrite
```
This creates an app with the user's requested app name and organization and then imports the `import.toml` file to configure it. It places the app in the user's `personal` organization. Finally, an `--overwrite` flag means this command will overwrite existing `fly.toml` files without querying the user.
### Finding the Region/Creating a Volume
Now, this script needs to create a volume in the same region as the app to provide storage for it. While humans can easily read the the output saying which region the app has been created in, it's currently a little harder for scripts. That's where this snippet comes in:
```
REGION=`fly regions list --json | awk '/"Code"/ { sub(/\ *"Code\": \"/ , "")
sub(/",/,"")
print }'`
```
It runs the `fly regions list` command but with the `--json` flag which makes the output somewhat more processable. It then uses `awk` (a fine Unix command) to extract out the region code from the line which reads `"Code": "lhr",`. It's that value it saves into the REGION variable ready for when it runs:
```
fly volumes create data --region $REGION $disksize
```
Which creates the storage volume.
### Deploying with Docker Arguments
At this point in the script we are ready to deploy with `fly deploy` but we still have our `USER`, `USE_DOCKER` and `EXTRA_PKGS` arguments to pass. The command line looks like this:
```
fly deploy --build-arg USER=$(whoami) --build-arg EXTRA_PKGS="$extrapackages" $usedocker
```
The `--build-arg` flag takes a name=value pairing as its parameter. The first build-arg sets `USER` to the value of the current user's name. The second build-arg sets `EXTRA_PKGS` to the value of the \$extrapackages variable; notice that it is surrounded in quotes are the list of extra packages will likely contain spaces which the shell would process.
Finally, we have `$usedocker` on its own, but that's because, if you scan to the top of the script, you'll see that it the user answers "Yes" to the question, the value of \$usedocker is set to "--build-arg USE\_DOCKER=y" (if they answer no, then \$usedocker is just an empty string).
As you can see, there's a lot of ways to build up your build arguments. Consult your shell and Dockerfile references for more.
Once run, the command will build, using the Dockerfile and build-args, an image which will then be deployed onto the Fly infrastructure.
The only thing left is the tell the user how to connect and what address to use.
```
echo "To use in VS Code, tell the remote-ssh package to connect to $(whoami)@$(fly info --host):10022"
```
The `fly info --host` is an easy way to get the host name of your deployment for connection strings.
## Discuss
* You can discuss this example on the [vscode-remote example](https://community.fly.io/t/a-vscode-example-for-fly/460) topic on community.fly.io.
# App Availability and Resiliency
Source: https://docs.fly.io/apps/app-availability
Fly.io resiliency features, from two Machines per process group and standby Machines to health check routing, and what your app must handle itself.
A resilient app is resistant to events like hardware failures or outages. Redundancy, supported by monitoring and scaling, is one of the ways to make your app more resilient, and we provide some built-in features to make this easier.
## What your app needs to do
First, it's important to understand what your app needs to take care of.
Since Fly Proxy does load balancing by default, you need to consider how your app handles:
* session storage, so that users stay logged in when their session moves to a different server
* persistent file storage and replication, as discussed in the following paragraphs
> [!NOTE]
>
> **A note about Fly Proxy**:
> Fly Proxy does a lot of work in the background, applying handlers, managing traffic and load balancing, and managing connections to services. Fly Proxy also automatically starts and stops Machines based on traffic to your app. The proxy works for [process groups with services defined](/networking/app-services#services-routed-with-fly-proxy).
For databases, you'll need to follow the specific recommendations for the database you're using, including how to set up clusters for replication and failover.
If you're deploying Fly Postgres, our un-managed database app, then you need to [manage backups, recovery, replication, and scaling](/unmanaged-postgres) yourself. For information about high availability with Fly Postgres, refer to [High Availability & Global Replication](/unmanaged-postgres/advanced-guides/high-availability-and-global-replication).
When you use [Fly Volumes](/volumes) with your app, your app needs to implement replication and backups. Volumes are block devices that live on a single disk array and get mounted directly by your Machine. There's no magic. And there's no built-in replication between volumes. If your app provides a clustering mechanism with data replication, like most databases do, we recommend you take advantage of that and run multiple instances with attached volumes. When possible, we place your app's volumes in different hardware zones within a region to mitigate hardware failures.
This brings us to disaster recovery and being prepared. You should have backups of your data and understand how to recover that data if needed. Fly.io takes daily snapshots of volumes and stores them for 5 days by default. You can [restore a volume's data](/volumes/snapshots#restore-a-volume-from-a-snapshot) from one of these daily snapshots. If your use case requires more frequent backups, then you'll need to set up tooling and processes to backup your volumes on the required schedule.
## Fly.io features for app resiliency
The Fly Platform has features to help make your app more resilient in case of outages or hardware failures.
The features for redundancy and availability are:
* [Redundancy by default on first deploy](#redundancy-by-default-on-first-deploy)
* [Automatically start and stop Machines](#automatically-start-and-stop-machines)
* [Health check-based routing](#health-check-based-routing)
> [!NOTE]
>
> These features require or depend on your app configuration in the `fly.toml` file, including process groups. Every Fly App has at least one process group. An app with no extra process groups defined in `fly.toml` still uses the `app` process group by default. Learn more about [app configuration](/reference/configuration).
## Redundancy by default on first deploy
You get redundancy by default on first deploy and when you deploy after scaling down to zero Machines. This feature helps you deploy a recommended setup without having to think about it. If the defaults in the following sections aren't what you're looking for, then you can still configure your app and processes the way you want. Learn more about [resilient apps and multiple Machines](/blueprints/resilient-apps-multiple-machines), [scaling Machine CPU and RAM](/launch/scale-machine), and [scaling the number of Machines](/launch/scale-count).
Here's what Fly Launch (the `fly launch` or `fly deploy` command) does, based on your app configuration:
* creates and starts [two Machines](#two-machines-for-process-groups-with-services) for process groups with services, sets automatic start and stop to true and minimum machines running to zero
* creates and starts one Machine and creates one stopped [standby Machine](#standby-machines-for-process-groups-without-services) for process groups without services
* creates and starts only one Machine if the process group has volumes mounted, because there's no built-in replication between volumes
### Two Machines for process groups with services
The most basic way to improve resiliency is to create more than one Machine per process group, and the `fly launch` and `fly deploy` commands do this by default for process groups with services. A process group with services has mappings to the external internet or to a private (Flycast) network, so that Fly Proxy can manage its connections and operation.
If a process group has services defined, two Machines are automatically created and started when:
* you deploy an app for the first time using `fly launch` or `fly deploy`,
* you redeploy an app using `fly deploy` after scaling down to zero, or
* you add a new process group with services in the `fly.toml` file and then run `fly deploy`
### Standby Machines for process groups without services
Standby Machines provide low-effort redundancy for process groups with no services defined. A process group with no services has no mappings to the external internet or to a private (Flycast) network, so Fly Proxy can't manage its connections and operation.
A standby Machine is a stopped Machine that's essentially paired to and watching a running Machine. The standby starts only if the paired Machine becomes unavailable.
If a process group doesn't have services defined, then a standby Machine is automatically created, along with a running Machine, when:
* you deploy an app for the first time using `fly launch` or `fly deploy`,
* you redeploy an app using `fly deploy` after scaling down to zero, or
* you add a new process group in fly.toml and then run `fly deploy`
The standby Machine doesn't consume resources or start up until it's needed.
#### Remove a standby Machine
To remove a standby Machine, run `fly status` and copy the ID of the Machine that displays the `†` symbol beside the process group name. For example:
```
PROCESS ID VERSION REGION STATE ROLE CHECKS LAST UPDATED
task† 784e469a443e38 43 yul stopped 2024-08-06T20:48:10Z
```
Then destroy the standby Machine with `fly machine destroy `.
If you destroy a standby Machine, it is not recreated when you [scale up or back down](/launch/scale-count) using `fly scale count`.
If you add services for a process group in `fly.toml` that previously had no services, then the standby designation is removed from the standby Machine on the next deploy.
#### Create a standby Machine
You can recreate the standby Machine if you scale down to 0 and then run `fly deploy`, which will create one Machine and one standby Machine again.
At a lower level, you can also use the `fly machines update` command with the `--standby-for` option to create a standby configuration using two existing Machines.
### Turn off redundancy on deploy
What if you don't want to keep Machines running all the time for an app with a small or variable workload? The [automatic start and stop feature](#automatically-start-and-stop-machines) is enabled by default for Machines and makes it possible to run at least two Machines without wasting resources.
If you still don't want `fly launch` or `fly deploy` to create two Machines, then you can use the `--ha` option to turn off this feature.
Create one Machine for a new app:
```bash theme={null}
fly launch --ha=false
```
Create one Machine on first deploy or when your app is scaled down to zero Machines:
```bash theme={null}
fly deploy --ha=false
```
Scale down an app to one Machine:
```bash theme={null}
fly scale count 1
```
For apps with more than one process, refer to [Scale a process group horizontally](/launch/processes#scale-a-process-group-horizontally).
## Automatically start and stop Machines
Automatic start and stop works for process groups with services defined. This feature enables you to have multiple Machines that only run when needed.
Fly Proxy can start and stop existing Machines based on incoming requests, so that your app can accommodate bursts in demand without keeping extra Machines running constantly. And if your app needs to have one or more Machines always running in your primary region, then you can set a minimum number of machines to keep running. Fly Proxy is also responsible for [load balancing](/reference/load-balancing).
Default settings for new apps created using the `fly launch` command: automatically start and automatically stop Fly Machines, and minimum machines running is zero.
Default settings for some existing apps (or any apps that don't have these settings in `fly.toml`): automatically start but don't automatically stop Fly Machines.
Learn more about [how Fly Proxy autostop/autostart works](/reference/fly-proxy-autostop-autostart) and [how to configure it](/launch/autostop-autostart).
## Health check-based routing
Fly Proxy routes network connections away from Machines that are failing health checks.
Health check-based routing works for process groups with services defined and is based on the configuration of [`services.tcp_checks`](/reference/configuration#services-tcp_checks) or [`services.http_checks`](/reference/configuration#services-http_checks) in your app's `fly.toml` file.
If the configured health checks are failing for a Machine, then the proxy doesn't route network connections to that Machine. The proxy routes connections to a healthy Machine instead. If there aren't any healthy Machines, then the connection will block, waiting for a Machine to become healthy.
> [!NOTE]
>
> Health check-based routing doesn't work for the top-level health checks defined in the `checks` section of fly.toml, because the top-level checks don't apply to specific services.
# App handover guide
Source: https://docs.fly.io/apps/app-handover-guide
Hand an app to a client on Fly.io: build in their org from day one, or run fly apps move and rebuild Postgres, Upstash Redis, and Tigris by hand.
> [!NOTE]
>
> **If you're building an app for someone else and they’re going to run it themselves on Fly.io, you’ve got two good options. You can either start development inside their Fly.io organization, or build it in yours and move it over later. Here’s how both paths work, and what can get messy during the move.**
***
### Option 1: Build in the customer’s org from day one
The cleanest handoff is no handoff at all. Have the customer create their own Fly.io organization before you start building.
1. **They create the organization and add a payment method.** Basic stuff. This is what Fly.io bills against.
2. **They invite your devs to their org.** They can do this in the dashboard under “Team → Invite Member.” You now have access to deploy apps directly into their account.
3. **You build inside their org.** Fly.io treats organizations as isolated namespaces, so there’s no awkward boundary crossing here. It’s just your team shipping code, in their environment.
4. **They remove you when it’s over.** Or not—you can stick around for maintenance if that’s part of the deal.
5. **What can get messy.** Make sure that the invited developer is not the only admin user in your customer's org. Double-check that your customer has at least two admins in their org.
***
### Option 2: Build in your org, then move the app to theirs
Sometimes it’s easier to get going inside your own org—especially if you’ve got infra and tooling already set up. When the app is ready, you move it.
Here’s how that works:
1. **Build as usual in your organization.**
2. **Have the customer invite one of your devs to their org.** This person needs to be in *both* orgs to do the move.
3. **Transfer the app:** Use the CLI command `fly apps move --org `,
or go to the App Settings page in your Dashboard, click Move app, pick the new organization, and wait for the deployment to finish.
1. **Be aware:** the move causes a couple minutes of downtime. It’s a good idea to schedule this when traffic is low, and inform your users of a maintenance window.
2. **What moves automatically:**
* Machines and volumes (data included)
* Secrets and environment variables
* Certificates and domain names
* LiteFS databases (as long as you’re using `$FLY_APP_NAME` for the Consul key)
3. **What doesn’t move automatically:**
* **Postgres** **(unmanaged and managed):** Moving Postgres databases across orgs is not supported. You’ll need to spin up a new Fly Postgres app in the target org and restore from a volume snapshot. Or you can use `pgloader` to migrate the data.
* **Upstash Redis:** This is tied to an org’s private network. You’ll need to provision a new DB in the new org.
* **Tigris buckets:** You’ll have to delete the old Tigris bucket, recreate it in the new org, and copy the data over (try `s3sync`). Don’t forget to reset your app’s secrets.
### Summary
If you know up front that your customer will own the app long-term, starting in their org avoids a bunch of fiddly work later. But if you need to hand over an app from your org to theirs, Fly.io gives you enough tools to pull it off—just be ready to rewire a few things by hand.
### Related reading
* [Move an app between orgs](/apps/move-app-org)
* [Fly.io billing](/about/billing)
# Build Secrets
Source: https://docs.fly.io/apps/build-secrets
Use secrets in a Docker build on Fly.io: mount them in a RUN step with --mount=type=secret, pass values with fly deploy --build-secret, and test locally.
You can set [secrets](/apps/secrets) for your applications, but these are only available at *run-time*. They aren't available when building your Docker image without a little extra work.
To make a secret available at build time, we'll use [Docker secrets](https://docs.docker.com/develop/develop-images/build_enhancements/).
It's a 2-step process:
1. Mount a secret into your `Dockerfile`
2. Provide the value for that secret when running `fly deploy`
## Mounting secrets
Mount a secret into your `Dockerfile` within a `RUN` statement:
```dockerfile theme={null}
# Note: You can mount multiple secrets
RUN --mount=type=secret,id=MY_SUPER_SECRET \
MY_SUPER_SECRET="$(cat /run/secrets/MY_SUPER_SECRET)" some_command \
&& more_commands_maybe
```
This creates a new file when running `docker build`. Secrets are stored in the `/run/secrets` directory. The file name is the `id` you passed when mounting the secret. The content of that file contains the value of the secret.
To mount more than one secret, add a `--mount` line for each:
```dockerfile theme={null}
RUN --mount=type=secret,id=SECRET_ONE \
--mount=type=secret,id=SECRET_TWO \
SECRET_ONE="$(cat /run/secrets/SECRET_ONE)" \
SECRET_TWO="$(cat /run/secrets/SECRET_TWO)" some_command
```
The `--mount` directive is not a shell command, so there's no need to add `&&` after it as you commonly see when chaining commands.
## Secret values
You need to provide the values of the mounted secrets when running `fly deploy`:
```bash theme={null}
fly deploy \
--build-secret MY_SUPER_SECRET=some_value
```
To pass more than one secret, repeat the `--build-secret` flag:
```bash theme={null}
fly deploy \
--build-secret SECRET_ONE=value_one \
--build-secret SECRET_TWO=value_two
```
## Testing build secrets locally
If you want to test your Docker build locally (before deploying to Fly.io), the commands to do so would look something like this:
```bash theme={null}
echo -n "secret_value" > mysecret.txt
docker build --secret id=MY_SUPER_SECRET,src=mysecret.txt .
```
## Automate the inclusion of build secrets using an ephemeral Machine
The above requires you to have access to the values of secrets and to
provide those values on the command line. If this poses a problem, an
alternative may be to create an ephemeral Machine using the `fly console`
command to do the deployment.
An example Dockerfile you can use for that purpose:
```dockerfile theme={null}
# syntax = docker/dockerfile:1
FROM flyio/flyctl:latest as flyio
FROM debian:bullseye-slim
RUN apt-get update; apt-get install -y ca-certificates jq
COPY <<"EOF" /srv/deploy.sh
#!/bin/bash
deploy=(flyctl deploy)
touch /srv/.secrets
while read -r secret; do
echo "export ${secret}=${!secret}" >> /srv/.secrets
deploy+=(--build-secret "${secret}=${!secret}")
done < <(flyctl secrets list --json | jq -r ".[].name")
deploy+=(--build-secret "ALL_SECRETS=$(base64 --wrap=0 /srv/.secrets)")
${deploy[@]}
EOF
RUN chmod +x /srv/deploy.sh
COPY --from=flyio /flyctl /usr/bin
WORKDIR /build
COPY . .
```
The `deploy.sh` script contained within this Dockerfile will provide each secret individually as well as package up a script to set all secrets at once. You can set all build secrets at once in your Dockerfile using:
```dockerfile theme={null}
RUN --mount=type=secret,id=ALL_SECRETS \
eval "$(base64 -d /run/secrets/ALL_SECRETS)" && \
some_command
```
Assuming your builder Dockerfile is named `Dockerfile.builder`, you can launch the ephemeral machine using the following command:
```bash theme={null}
flyctl console --dockerfile Dockerfile.builder -C "/srv/deploy.sh" --env=FLY_API_TOKEN=$(fly auth token)
```
# Guidelines for concurrency settings
Source: https://docs.fly.io/apps/concurrency
Set soft_limit, hard_limit, and connections vs requests concurrency in fly.toml, and see how Fly Proxy uses them for load balancing and autostop.
Concurrency settings are used by Fly Proxy for important things like [load balancing](/reference/load-balancing#load-balancing-strategy) and [autostop/autostart](/launch/autostop-autostart) for Machines.
The following concurrency settings apply per Machine and per service in your app:
`type`: Use `connections` or `requests` as the metric for app load concurrency.
`hard_limit`: At or above this number, no new traffic goes to the Machine.
`soft_limit`: At or above this number, traffic to the Machine is deprioritized. Traffic is only routed to the Machine if all Machines reach the soft limit.
For Fly Launch apps, configure concurrency settings in the [\[http\_service.concurrency\]](/reference/configuration#http_service-concurrency) or [\[services.concurrency\]](/reference/configuration#services-concurrency) sub-sections of your `fly.toml` file. If you're using the Machines API, then concurrency settings are per `service` in [the `config` object](/machines/api/machines-resource#machine-config-object-properties).
## Default settings
The default concurrency settings when not specified are a `soft_limit` of 20 and no `hard_limit`. Many apps can handle a few hundred concurrent connections or more. You can try out different concurrency settings and see how it affects performance and load balancing.
`connections` is the default concurrency type.
## How Fly Proxy handles concurrency
Fly Proxy doesn't consider the exact number of concurrent connections or requests. From the proxy's point of view a Machine is handling between 0 and the soft limit, handling between the soft limit and the hard limit, or is at the hard limit. Along with region, this is how the proxy decides which Machine gets traffic.
The proxy's view of a Machine's load is affected when the gap between the soft limit and the hard limit is small and/or the app's concurrent connections or requests oscillate between them more frequently. In that case, the proxy routes according to the settings, but the thresholds change too quickly and requests or connections end up being re-routed.
General caveats:
* If the limits are too high, then your Machines might not be able to process that much concurrently and your app could crash.
* If the limits are too low, then the proxy is artificially limiting what your app can process.
* If the soft and hard limit are too close, then there might not be enough “time” for the proxy to decide to load balance and the result could be multiple retries.
## Connections or requests
The decision to use `connections` or `requests` for concurrency depends on the type of app and its load.
`requests` are HTTP requests and the recommended concurrency type for web services. Using `requests` for concurrency can prevent too many connections to your app and reduce latency; the proxy can temporarily pool and reuse connections.
`connections` are TCP connections. Multiple requests can be sent over a connection, so you need to consider how your app handles that. If you use `connections` for web services, then the proxy opens a new connection for each HTTP request, which is why `requests` is a better setting for HTTP apps.
## Concurrency limit tuning tips
When tuning concurrency, try setting a relatively high `hard_limit`, or leave it unset to have no hard limit. If you do want to set a `hard_limit` to have more control over load balancing, then you might have to do an initial benchmark to estimate the maximum number of concurrent connections or requests that your app can handle. Then tune the `soft_limit` and [create more Machines](/blueprints/resilient-apps-multiple-machines) to optimize autostop/autostart and load balancing. Once your app is getting real-world traffic, you can continue to monitor your app and adjust the `soft_limit` further to suit your workload.
Want a more detailed guide to concurrency limits? See this blueprint: [Setting Hard and Soft Concurrency Limits on Fly.io](/blueprints/setting-concurrency-limits).
# Delete an app
Source: https://docs.fly.io/apps/delete
Permanently delete a Fly App with fly apps destroy, removing its Machines, volumes, snapshots, IPs, secrets, and images and ending charges.
You can scale an app right down to zero Machines if you like. But if you're done with a Fly App forever, you can delete it using `fly apps destroy `.
Once you destroy an app, you can no longer access any part of it, and you'll no longer be charged for any part of it either. This includes Fly Machines, Fly Volumes, IP addresses, secrets, Docker images, and so on.
> [!WARNING]
>
> Warning: When you destroy an app, you delete its volumes and volume snapshots. If you want the data from an app's persistent storage, then get the data before you destroy the app.
# Tips to fine-tune your app on Fly.io
Source: https://docs.fly.io/apps/fine-tune-apps
Benchmark an app on Fly.io: compare edge and app response metrics, check routing with debug.fly.dev, and factor in concurrency and autostop.
It’s safe to assume Fly.io can support thousands of requests or connections per second and can keep hundreds of thousands of them live at any given moment. This is even truer when the clients connect from multiple geographic locations.
That settled, you can still use the following guidelines to gather data to fine-tune and optimize your app and configuration to run on Fly.io.
## Fly.io metrics
We have various ways to access metrics for Fly Proxy, your app, and Fly Machines. Learn more about [Fly Metrics](/monitoring/metrics).
* Edge HTTP response time is the total time it takes to respond with the first HTTP response headers.
* App HTTP response time is the same, but only considers your app’s response time, not the overhead of routing, load balancing, retries, and so on.
CPU and memory resource usage are interesting, but are rarely the culprit unless you have everything else in your benchmark right. The application logic often is often the issue when it comes to excessive CPU and memory usage.
## Routing
Check which [Fly.io region](/reference/regions) you're reaching.
The region you reach should be the fastest region for you, which may not be the closest geographically. You can use the debug.fly.dev server to check your region:
```
curl debug.fly.dev
```
Use `traceroute` or `mtr` to see the routing to Fly.io.
For example:
```
traceroute debug.fly.dev
```
Check the base latency for the region you’re reaching:
```
ping debug.fly.dev
```
> [!NOTE]
>
> **Important**: Do you have a stable internet connection? `ping` can jitter quite a bit on bad routes.
## Application logic and performance
Some common culprits of excessive CPU and memory usage are:
* Using a single thread
* Blocking
* Various misconfigurations, for example, wrong or missing environment variables
### HTTP response times
Fly.io HTTP response time metrics measure how long it takes for your app to start responding with headers, not the response body.
### Multithreading
If your app has multiple threads, make sure that contention isn't an issue. You can also [scale CPU resources](/launch/scale-machine) as needed.
## Fly.io configuration
Some configuration settings that are more likely to affect app performance and benchmarking.
### Machine sizing
Right-size Machines to use more parallel processing. Learn more about [Machine sizing](/machines/guides-examples/machine-sizing#general-rules).
### Concurrency settings for connections or requests
Refer to our guidelines for [concurrency settings](/apps/concurrency).
### Auto stop and start feature
[Auto stop and start](/launch/autostop-autostart) can be turned off if you want to test the performance of your app without that variable.
If you don’t turn off auto stop and start, then you’ll likely have some pretty high p99 values due to the initial cost of Fly Proxy queueing connections while waiting for the Machine to start.
## Distributed systems
Fly.io has rate limits and hard limits outside of the control of each app; these limits are per edge and per service. If you test from a single location, then you'll hit our rate limits much faster than if you test from multiple locations.
Pay attention to how your benchmarking nodes are routing. Inefficient routing adds latency and will skew your results. If incorrect routing is causing issues with your app testing or your production apps, then post in community or contact support (depending on you plan) and we'll look into it with our network providers.
## Challenges with benchmarking
You probably won’t be able to realistically benchmark your app for a combination of reasons, including:
* Users will have a large variety of ISPs and be routed differently based on ad hoc peering agreements, sometimes in non-optimal ways.
* It’s difficult to remove the Anycast routing factor from benchmarking; you have to live with it.
In some cases, a better option might be to send part of your real traffic to Fly.io and then compare metrics.
# Going to production checklist
Source: https://docs.fly.io/apps/going-to-production
Production checklist for Fly.io apps: SSO, tokens, secrets, Managed Postgres, Machine sizing, multi-region scaling, custom domains, metrics, CI/CD.
Use this checklist to help you set up a production environment on Fly.io.
## Overview
Moving an app from staging to production can expose unexpected failure modes: security holes, performance and scaling issues, or data loss. This checklist is meant to catch common pitfalls for apps on Fly.io, but it’s not a guarantee of production readiness. Not every item here will apply to your app, and you may have additional requirements that aren't listed. Use this as a foundation and adapt it to your needs. Think of this list as a scaffold, not a silver bullet.
> [!NOTE]
>
> **Important:** The checklist is not exhaustive and does not guarantee production-readiness for your app. Apps can have unique requirements for production depending on the framework and type of app. Some items won't be applicable and there may be other considerations not listed here; you'll need to decide which checklist items work for your app.
## Security
> **Set up single sign-on for organizations**
>
> Enable SSO on your organization to take advantage of Google or GitHub authentication security. Learn more about [Single sign-on for organizations](/security/sso).
> **Isolate staging and production environments**
>
> Use organizations to limit access to your production environment. Read this guide: [Staging and production isolation](/blueprints/staging-prod-isolation).
> **Enforce least privilege access**
>
> Use access tokens to allow only the minimum access level required by team members to your organization, apps, and Machines. Understand [access tokens](/security/tokens).
> **Protect sensitive information**
>
> Set secrets to store sensitive data and make them available as environment variables to your app. Read about [Secrets and Fly Apps](/apps/secrets).
> **Make sure private services are not exposed**
>
> Check that your private apps with services don’t have public IP addresses. Run `fly ips list` and use `fly ips release` to release unnecessary public IPs. More detail is available in this `flyctl` reference: [`fly ips` commands](/flyctl/cmd/fly_ips). Assign private apps a [Flycast address](/networking/flycast) instead.
> **Use Arcjet application security for JavaScript apps**
>
> Secure your app with rate limiting, bot protection, email validation, and defense against common attacks through our extension partner Arcjet. Read more about [Application Security by Arcjet](/security/arcjet).
## Databases
> **Use Managed Postgres**
>
> We recommend using [Fly.io’s Managed Postgres](/postgres), our fully-managed database service that handles all aspects of running production PostgreSQL.
> **Practice your disaster recovery plan**
>
> Practice restoring your managed Postgres database from a backup before you actually need to. You can do this anytime from the Managed Postgres dashboard.
## App performance
> **Get Machine sizing right**
>
> Most conventional production web apps require [performance CPUs](/machines/cpu-performance). Also make sure you have enough RAM for your app and/or enable [swapping to disk](/reference/configuration/#swap_size_mb-option) to deal with brief spikes in memory use. Find out more details in our [Machine sizing guide](/machines/guides-examples/machine-sizing).
> **Fine-tune your app**
>
> Learn about optimizing your app on Fly.io. Read these tips to [fine-tune your app on Fly.io](/apps/fine-tune-apps).
## Availability, resiliency, and costs
> **Use multiple Machines for resiliency**
>
> Make your app resilient to single-host failures with multiple Machines that stay stopped until you need them. Learn more in our guide: [Resilient apps use multiple Machines](/blueprints/resilient-apps-multiple-machines).
> **Scale your app into more regions**
>
> Scale your app in multiple regions closest to your app’s users. Find out how to [Scale an app’s regions](/launch/scale-count/#scale-an-apps-regions).
> **Use autostop/autostart to reduce costs**
>
> Autostop/autostart lets you stop or suspend Machines when there’s low traffic, saving on resource usage and costs. You get autostop/autostart by default with a new app, but you can configure it to optimize for your use case. Find out more: [Autostop/autostart Machines](/launch/autostop-autostart).
> **Set up autoscaling by metric to reduce costs**
>
> For apps that aren’t running web services, use the autoscaler app to scale your app’s Machines based on any metric, saving on resource usage and costs. Learn how to [Autoscale based on metrics](/launch/autoscale-by-metric).
## Networking
> **Set up a custom domain**
>
> Configure a certificate for your domain. Learn how to [use a custom domain](/networking/custom-domain).
> **Consider using a dedicated IPv4 address**
>
> Completely eliminate the chance of blacklisted spammers causing problems for your app. There is a small [added cost](/about/pricing/#anycast-ip-addresses) for dedicated IPv4 addresses. Read more about [Dedicated IPv4](/networking/services/#dedicated-ipv4).
> **Set up Flycast for private apps**
>
> If you haven’t already done so, give your private apps a Flycast address to communicate with them entirely on your private network. Find out about [Flycast - Private Fly Proxy services](/networking/flycast).
## Monitoring
> **Monitor your app with fully-managed metrics**
>
> Use managed Prometheus and managed Grafana dashboards to monitor your app. Read about [Metrics on Fly.io](/monitoring/metrics).
> **Use Sentry for error tracking**
>
> Sentry is an application monitoring platform that helps you identify and fix software problems before they impact your users. [Sign up with Sentry](https://sentry.io/signup/), then set your `SENTRY_DSN` as an app secret. Read more about [Application Monitoring by Sentry](/monitoring/sentry).
> **Export your logs**
>
> Set up the Fly Log Shipper to aggregate your app’s logs to a service of your choice. Read more about [Exporting logs](/monitoring/exporting-logs).
## CI/CD
> **Generate review apps with GitHub Actions**
>
> Automatically generate ephemeral review apps on Fly.io for each pull request (PR) using GitHub Actions. Learn more about [Git Branch Preview Environments on GitHub](/blueprints/review-apps-guide).
> **Deploy with GitHub Actions**
>
> Set up your app for continuous deployment to Fly.io from the app’s GitHub repository. Find out how to use [Continuous Deployment with Fly.io and GitHub Actions](/launch/continuous-deployment-with-github-actions).
## Get support
# Apps on Fly.io
Source: https://docs.fly.io/apps/index
Fly Apps docs hub: Fly Launch for deploying and scaling, runtime and build secrets, and production guides on availability, tuning, and concurrency.
An app on Fly.io can be anything from a simple frontend web app to a complex arrangement of processes and Machines all doing their own thing. Check out the [Fly Apps overview](/apps/overview) for details.
## Fly Launch
*If you're looking to deploy an app on Fly.io, you probably want Fly Launch. Use Fly Launch to create your app and then manage the whole lifecycle, from starting to scaling to changing and redeploying.*
* **[Get started](/getting-started):** Launch your own app or try a demo app.
* **[Use Fly Launch](/launch/index):** Learn how to create, configure, deploy, and scale your app.
***
## Secrets
*Store sensitive data, like usernames and passwords, as secrets in your app. Secrets are made available to the app as environment variables.*
* [Set app secrets to be available at runtime](/apps/secrets)
* [Mount secrets in your Dockerfile to be available at buildtime](/apps/build-secrets)
***
## Production apps
*Get your app ready for production on Fly.io.*
* [Going to production checklist](/apps/going-to-production)
* [App availability and resiliency](/apps/app-availability)
* [Fine-tune your app](/apps/fine-tune-apps)
* [Guidelines for concurrency settings](/apps/concurrency)
***
## Related topics
* [Databases & Storage](/database-storage-guides)
* [Networking](/networking)
* [Monitoring](/monitoring)
* [Security](/security)
* [Blueprints: Fly.io patterns for your projects](/blueprints)
# Get information about an app
Source: https://docs.fly.io/apps/info
Inspect a running Fly App with fly status, fly machine status, fly ips list, fly config show, fly ssh console, and fly logs, with sample output.
Once your Fly App is launched, `flyctl` has various tools for getting information about it. You can also find a lot of information on your Fly.io [dashboard](https://fly.io/dashboard).
## Find all your apps
You can see a list of all your Fly Apps:
```bash cmd theme={null}
fly apps list
```
```out out theme={null}
NAME OWNER STATUS PLATFORM LATEST DEPLOY
testrun personal deployed machines 21h17m ago
my-app personal suspended machines 2023-11-15T23:33:07Z
```
## App overviews
If you want a brief app overview, including a list of Machines on that app with their current status, use `fly status`:
```bash cmd theme={null}
fly status -a testrun
```
```out out theme={null}
App
Name = testrun
Owner = personal
Hostname = testrun.fly.dev
Platform = machines
ID STATE REGION HEALTH CHECKS IMAGE CREATED UPDATED
06e82d43ad1587 started yyz 1 total, 1 passing testrun:deployment-01GQ0HKV6TDT7DX0G50MT50FZD 2023-01-17T17:54:04Z 2023-01-17T21:42:33Z
```
As with many flyctl commands, if you leave off the `-a` flag, `fly status` will infer the app name from the `fly.toml` file in the working directory, if there is one.
`fly machine list` yields a different set of information for each Machine, including the Machine's [internal IPv6 address](/networking/private-networking):
```bash cmd theme={null}
fly machine list -a testrun
```
```out out theme={null}
1 machines have been retrieved from app testrun.
View them in the UI here (https://fly.io/apps/testrun/machines/)
testrun
ID NAME STATE REGION IMAGE IP ADDRESS VOLUME CREATED LAST UPDATED APP PLATFORM PROCESS GROUP
178115db494e18 holy-waterfall-9884 started lhr testrun:deployment-01GTYFV11PM7D7B30AWZSH1FZE fdaa:0:3b99:a7b:a98:6c48:67c5:2 2023-03-07T16:52:59Z 2023-03-07T16:53:06Z v2 app
```
## Examine a specific Machine
You can drill down and get an overview of a particular Machine with `fly machine status`. This is where you'll find the Machine's CPU size and RAM settings.
```bash cmd theme={null}
fly machine status e286065f969386
```
```out out theme={null}
Machine ID: e286065f969386
Instance ID: 01GQR4GR5TPQC65T5RMKR0HS87
State: started
Machine
ID = e286065f969386
Instance ID = 01GQR4GR5TPQC65T5RMKR0HS87
State = started
Image = testrun:deployment-01GQR4GN7EA8G12VE1DBRQG4KQ
Name = rough-tree-1360
Private IP = fdaa:0:3b99:a7b:8aeb:fea3:148b:2
Region = iad
Process Group =
Memory = 256
CPUs = 1
Created = 2023-01-25T21:35:32Z
Updated = 2023-01-26T22:52:36Z
Command =
Event Logs
STATE EVENT SOURCE TIMESTAMP INFO
started start flyd 2023-01-26T17:52:36.678-05:00
created launch user 2023-01-26T17:52:33.348-05:00
```
## Services
See the configured services with `fly services list`.
```bash cmd theme={null}
fly services list
```
```out out theme={null}
Services
PROTOCOL PORTS FORCE HTTPS
TCP 80 => 8080 [HTTP] True
443 => 8080 [TLS,HTTP] False
```
## Public IP addresses
Find your app's public Anycast IPs and private Flycast IPs with `fly ips list`.
```bash cmd theme={null}
fly ips list
```
```out out theme={null}
VERSION IP TYPE REGION CREATED AT
v6 2a09:8280:1::2d:678b public (dedicated) global Sep 1 2023 19:47
v6 fdaa:2:45b:0:1::23 private global Mar 16 2024 18:20
v4 66.241.124.63 public (shared) Jan 1 0001 00:00
```
Read more about [Public networking](/networking/services) and [Private networking](/networking/private-networking).
## Check on it from inside
You can use `fly ssh console` to get a prompt on a Machine in your app (as long as the Machine's Docker image includes `sh`). You may want to connect to a specific Machine, and you can do that with `fly ssh console -s`:
```bash cmd theme={null}
fly ssh console -s
```
```out out theme={null}
? Select Machine: iad: e286065f969386 fdaa:0:3b99:a7b:8aeb:fea3:148b:2 rough-tree-1360
Connecting to fdaa:0:3b99:a7b:8aeb:fea3:148b:2... complete
#
```
If you have a particular command in mind, it might be quicker to use the `-C` flag to just run the command; for example to check on free space in the Machine file system, you might run
```bash cmd theme={null}
fly ssh console -C df
```
```out out theme={null}
Connecting to fdaa:0:3b99:a7b:7e:3155:9844:2... complete
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 103068 0 103068 0% /dev
/dev/vda 8191416 172748 7582856 3% /
shm 113224 0 113224 0% /dev/shm
tmpfs 113224 0 113224 0% /sys/fs/cgroup
/dev/vdb 1011672 2564 940500 1% /storage
```
> [!NOTE]
>
> If you are running an interactive command (like a shell, IEx, Django management command, or the Rails console), you may need to use the `--pty` flag. This tells the SSH server to run the command in a pseudo-terminal. (If you're familiar with OpenSSH, this is like the `-t` flag for `ssh`.)
## Inspect the current configuration of a deployed app or Machine
Machines can be configured individually, but Fly Launch applies the app's config on `fly deploy` to all Machines that are administered by the app. Display the app configuration in JSON format with `fly config show`.
### Show the app config
```bash theme={null}
fly config show -a testrun
```
### Show a Machine's configuration
Tack a specific Machine's current configuration onto the output of `fly machine status` by appending the `-d` flag. Machines belonging to a Fly App will *usually* be managed by `fly deploy` and so have the same configuration as their app.
```bash cmd theme={null}
fly m status e784459b655483 -d
```
```out out theme={null}
Machine ID: e784459b655483
Instance ID: 01GQTQV0EMGV8XY4NYJN68247W
State: started
VM
ID = e784459b655483
Instance ID = 01GQTQV0EMGV8XY4NYJN68247W
State = started
Image = testrun:deployment-01GQTQTJB5RCS03FKGX9K2H60B
Name = red-feather-7761
Private IP = fdaa:0:3b99:a7b:92:9336:667f:2
Region = iad
Process Group =
Memory = 256
CPUs = 1
Created = 2023-01-27T23:08:41Z
Updated = 2023-01-27T23:09:10Z
Command =
Event Logs
STATE EVENT SOURCE TIMESTAMP INFO
started start flyd 2023-01-27T18:09:10.855-05:00
created launch user 2023-01-27T18:08:41.361-05:00
Config:
{
"init": {},
"image": "registry.fly.io/testrun:deployment-01GQTQTJB5RCS03FKGX9K2H60B",
"metadata": {
"fly_platform_version": "v2",
"fly_release_id": "QgDeJ3o11V2D7fkXVQGvPBON",
"fly_release_version": "1"
},
"restart": {},
"services": [
{
"protocol": "tcp",
"internal_port": 4999,
"ports": [
{
"port": 80,
"handlers": [
"http"
],
"force_https": true
},
{
"port": 443,
"handlers": [
"tls",
"http"
]
}
],
"checks": [
{
"type": "tcp",
"interval": "15s",
"timeout": "2s"
}
],
"concurrency": {
"type": "connections",
"hard_limit": 25,
"soft_limit": 20
}
}
],
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 256
}
}
```
## Watch an app's logs
Running `fly logs` displays an app's logs as they happen. Logs include the console output of all instances of an application. Here are logs for an app that hangs around doing nothing but has a volume mounted so you can ssh in and create files.
```bash cmd theme={null}
fly logs -a testrun
```
```out out theme={null}
2023-03-07T16:17:48Z runner[5683606c41098e] lhr [info]Pulling container image
2023-03-07T16:17:51Z runner[5683606c41098e] lhr [info]Unpacking image
2023-03-07T16:18:00Z runner[5683606c41098e] lhr [info]Setting up volume 'data'
2023-03-07T16:18:00Z runner[5683606c41098e] lhr [info]Uninitialized volume 'data', initializing...
2023-03-07T16:18:00Z runner[5683606c41098e] lhr [info]Encrypting volume
2023-03-07T16:18:05Z runner[5683606c41098e] lhr [info]Opening encrypted volume
2023-03-07T16:18:07Z runner[5683606c41098e] lhr [info]Formatting volume
2023-03-07T16:18:08Z runner[5683606c41098e] lhr [info]Configuring firecracker
2023-03-07T16:18:08Z app[5683606c41098e] lhr [info]Starting init (commit: 08b4c2b)...
2023-03-07T16:18:08Z app[5683606c41098e] lhr [info]Mounting /dev/vdb at /storage w/ uid: 0, gid: 0 and chmod 0755
2023-03-07T16:18:08Z app[5683606c41098e] lhr [info]Preparing to run: `sleep infinity` as root
2023-03-07T16:18:08Z app[5683606c41098e] lhr [info]2023/03/07 16:18:08 listening on [fdaa:0:3b99:a7b:7e:3155:9844:2]:22 (DNS: [fdaa::3]:53)
```
`fly logs` stays open, watching the logs, until you stop it (ctrl-C ).
Learn more about [logging on Fly.io](/monitoring/logging-overview).
# Move an app between organizations
Source: https://docs.fly.io/apps/move-app-org
Move an app to another Fly.io org with fly apps move or the dashboard. Expect brief downtime and reconfigure Upstash Redis, Tigris, and Postgres.
You might want to move an app to a new org to hand off your completed work to a client, to take advantage of a new pricing model, or to move a production app out of your personal organization.
You can move an entire app with its resources to another organization with the `fly apps move` command in `flyctl`, or using the App Settings page of the Dashboard. You'll need to be a member of both organizations to move an app.
**Move an app from one organization to another via the CLI:**
```bash theme={null}
fly apps move --org
```
> [!NOTE]
>
> **Fly Postgres:** You can't use the `fly apps move` command for Fly Postgres apps. To move a Postgres app to another organization, create a new Postgres app under the target organization, and then [restore the data from your current Postgres app volume snapshot](/unmanaged-postgres/managing/backup-and-restore#restoring-from-a-snapshot).
**Move an app from one organization to another via the App Settings page:**
Go to the App Settings page in your Dashboard, click Move app, pick the new organization, and wait for the deployment to finish. Your app will be up and running once the transfer is complete.
## App downtime
Your app will have up to a few minutes of downtime while the move operation completes.
## Resources moved with the app
The following app resources are transferred to the new org automatically:
* Fly Machines and Fly Volumes (including data)
* environment variables and secrets
* certificates and domain names
* LiteFS databases (when `$FLY_APP_NAME` is used for your Consul key in `litefs.yml`)
## Resources that you need to manually reconfigure
The following extension services need to be reconfigured for the app after the move:
* **[Upstash for Redis](/upstash/redis):** Upstash Redis is only available over an organization's private network. After you move the app, you'll need to provision a new database for the new org.
* **[Tigris object storage](/tigris):** Before moving the app, you'll need to delete the bucket, then recreate it in the new org and set the new secrets. Billing for the new bucket will be tied to the new org.
## Read more
[App handover guide: considerations and approaches](/apps/app-handover-guide)
# Fly Apps overview
Source: https://docs.fly.io/apps/overview
See what makes up a Fly App on Fly.io, a group of Machines with config, IPs, certificates, secrets and volumes, and the ways to create one.
A Fly App is an abstraction for a group of Fly Machines running your code on Fly.io. You can create and manage your app as a whole using [Fly Launch](/launch/index), but you can also have a Fly App with individual Machines running tasks or user code.
For a quick primer on Fly Launch, Fly Machines, and Fly Apps, see [Fly.io essentials](/getting-started/essentials).
## A Fly App can be almost anything
From an admin point of view, a Fly App is just a group of Machines (with optional attached volumes) that belongs to one organization.
From a developer point of view, a Fly App might be:
* a fullstack application (or just part of one)
* a database
* a few Machines running tasks, or a bunch of Machines, all with different configs, doing things you want them to do
* anything you can think of doing with fast-launching Machines
All the apps in your organization can communicate over a [private network](/networking/private-networking), so it’s also possible to have multiple apps working together as one system.
## Components of a Fly App
Fly Apps include:
* app configuration
* provisioned resources
* Anycast IP addresses
* certificates
* custom domains
* secrets
* Fly Volumes (optional)
Fly Apps run on flyd. Learn more about how flyd works and how it came to be in the blog post: [Carving the Scheduler Out of Our Orchestrator](https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/).
## Ways to create an app
There might be an infinite number of potential apps you can run on Fly.io, but there are only a few ways to create one.
### Create and deploy with Fly Launch
Fly Launch is perfect for most apps and databases. Use Fly Launch to create your app and then manage the whole lifecycle, from starting to scaling to changing and redeploying. [Get started](/getting-started) with Fly Launch or learn more about using [Fly Launch](/launch/index) to create, configure, deploy, and scale your app.
### Create an app manually
Most of the time, all you need is [Fly Launch](/launch/index) to create and manage your app. But you can skip all the handy launch scanners and resource provisioning Fly Launch offers and use the `fly apps create` command or the Machines API to create an app and then piece it together.
You can get a [`fly.toml` config file](/reference/configuration) from an example app or hand-craft one yourself. Or if you don't need app-wide config, you can use the Fly App to hold the Machines you plan to spin up individually based on a particular image.
Using the `fly apps create` command is useful when deploying our [autoscaler app](/launch/autoscale-by-metric), or example apps created by others, that already have `fly.toml` and Dockerfiles ready to go.
Creating apps with the Machines API might be the right choice if you want to create apps for your own users with pre-defined or custom `fly.toml` files.
For more information, see the [`fly apps create` docs](/flyctl/cmd/fly_apps_create) and the [create App resource docs](/machines/api/apps-resource#create-a-fly-app).
> [!NOTE]
>
> **Tip:** When you [create a Machine with `fly machine run`](/machines/flyctl/fly-machine-run) without specifying an app to put it in, you automatically get a new app at the same time. This scenario might be useful when you're spinning up individual Machines on-demand for multiple users or tasks.
# Restart apps or Machines
Source: https://docs.fly.io/apps/restart
Restart all Fly Launch Machines in an app with fly apps restart, or reboot specific Machines by ID with fly machine restart to reset the root filesystem.
Running `fly deploy` on an app creates a new release of the app, and updates and restarts the Machines that belong to Fly Launch.
Sometimes you don't want to update anything; you just want to reboot and start the root file system afresh. Restarting wipes the ephemeral file system, just as a `fly deploy` or `fly machine update` will.
## Restart every Machine managed by Fly Launch
`fly apps restart ` restarts all Machines in the app that are managed by Fly Launch. In other words, all the Machines deployed with `fly deploy` and configured with `fly.toml`.
## Restart individual Machines
If one particular Machine needs a kick—or you want to restart an unmanaged Machine—then run `fly machine restart`:
```bash theme={null}
fly machine restart
```
You can provide multiple ``s to restart several Machines in one command.
# Secrets and Fly Apps
Source: https://docs.fly.io/apps/secrets
Set, stage, list and unset app secrets on Fly.io with fly secrets, see how the encrypted vault injects them as env variables, and mount secrets as files.
## Overview
You can specify secrets for your Fly App using the `fly secrets` command. Secrets allow sensitive values, such as credentials, to be passed securely to your Fly App. The secret is encrypted and stored in a vault. An app's secrets are available as environment variables at runtime on every Machine belonging to that Fly App, whether the Machine is managed by Fly Launch or not.
Data stored as secrets doesn't have to be sensitive; secrets are made available to the app as environment variables to use for whatever purpose you like.
If you need secrets to be available when building your Docker image, see [Build secrets](/apps/build-secrets).
## Architecture
Secrets are stored in an encrypted vault. When you set a secret through `flyctl`, it sends the secret value through our API, which writes to the vault for your specific Fly App. The API servers can only encrypt; they cannot decrypt secret values. Secret values are never logged.
When we launch a Machine for your app, we issue a temporary auth token to the host it runs on. The Fly.io agent on the host uses this token to decrypt your app secrets and inject them into your Machine as environment variables at boot time. When you destroy your Machines, the host environment no longer has access to your app secrets.
> [!WARNING]
>
> **Warning:** `flyctl` and our API servers are designed to prevent user secrets from being extracted. However, secrets are available to your application code as environment variables. People with deploy access **can** deploy code that reads secret values and prints them to logs, or writes them to unencrypted data stores.
## Working with Secrets
### Set secrets
The `fly secrets set` command sets one or more app secrets, then updates each Machine belonging to that Fly App. This involves a restart of the Machine and a consequent reset of its ephemeral file system.
The following example sets a secret that's available as the `DATABASE_URL` environment variable within your application processes:
```sh theme={null}
fly secrets set DATABASE_URL=postgres://example.com/mydb
```
To set, or update, a secret in the app's vault, but defer updating the Machines to later, use the `--stage` option. For example:
```bash theme={null}
fly secrets set DATABASE_URL=postgres://example.com/mydb --stage
```
In the preceding example, the staged secret will be available only on Machines that are started or updated after the `fly secrets set` command was run.
The `secrets` command can also take secrets from `stdin`. For commands and options, see the [`fly secrets` docs](/flyctl/cmd/fly_secrets) or run `fly secrets --help`.
> [!NOTE]
>
> **Note:** You can update a machine by triggering a new release with `fly deploy`. Alternatively, the `fly secrets deploy` command will redeploy the current release with the staged secrets. This is helpful if you want to skip rebuilding the image from source code.
### List secrets
List secrets that are set for your app. The list shows only the secret name; the value is not shown as it is a secret.
For example:
```bash cmd theme={null}
fly secrets list
```
```output output theme={null}
NAME | DIGEST | DATE
+--------------+----------------------------------+---------+
MY_SECRET | b9e37b7b239ee4aefc75352fe3fa6dc6 | 17s ago
DATABASE_URL | cdbe3268a82bfe993921b9cae2a526af | 17s ago
```
For security reasons, we do not allow read access to the plain-text values of secrets.
### Remove secrets
Remove one or more secret values from your app by name.
The following example removes two secrets from the app:
```bash theme={null}
fly secrets unset MY_SECRET DATABASE_URL
```
### Mounting Secrets as Files
You can write a secret’s value directly to the machine's filesystem at startup using the `[[files]]` section in `fly.toml`. This is useful for things like private keys your app expects on disk.
Secrets must be base64‑encoded.
Example:
```bash theme={null}
fly secrets set SUPER_SECRET=$(cat filename.txt | base64)
```
`secret_name`: The name of an app secret that’s been set with `fly secrets set`. The value of the secret will be written to the file. The referenced secret’s value must be base64 encoded.
`guest_path`: The full path inside the Machine where the file will be written at boot.
```toml theme={null}
[[files]]
guest_path = "/path/to/secret.txt"
secret_name = "SUPER_SECRET"
```
You can use this with other `[[files]]` entries, check the `fly.toml` [reference](/reference/configuration#the-files-section) for more details.
# Troubleshoot apps when a host is unavailable
Source: https://docs.fly.io/apps/trouble-host-unavailable
Recover apps on Fly.io when a host goes down: scale to zero and back up, redeploy, restore a volume from a snapshot, or rebuild a single-node Postgres app.
Fly.io has multiple hosts (physical hardware) in every supported [region](/reference/regions). From time to time, a hardware failure, connectivity issues, or other factors will make a host unavailable. A single-host issue may cause a sustained or intermittent outage, or generally degraded performance, and this will disproportionately affect apps that run on a single Machine.
A host can be down for an hour, or a day, or sometimes longer.
For production apps, or any apps that require high availability, we strongly recommend redundancy to protect against single-host failure. This means running multiple Machines, and for stateful apps it means having a clustering mechanism (such as those built into LiteFS and Postgres, for example) to replicate data between multiple volumes, because volumes are pinned to specific hosts.
> [!NOTE]
>
> **Important:** The steps for recovery in this document don’t guarantee that you’ll be able to access your app if you have a single Machine on a host that’s affected by an ongoing issue.
## Notification of host issues
You’ll get a notification on the Status page in your [dashboard](https://fly.io/dashboard/) when there’s a host issue that affects one or more of your apps. We don’t post individual host issues to our main status page because a host issue usually affects a small number of users in a single region.
## Apps with one Machine and no volumes
You can usually restore an app with one Machine and no volumes by scaling down and back up, or by deploying a new version of the app.
### Scale the app to get new Machines
For apps without volumes, you can bring up a new Machine by scaling to zero and then back up again. The new Machines will be placed on a healthy host.
1. Scale the app to zero Machines:
```bash theme={null}
fly scale count 0
```
2. Scale back up to 2 or more Machines:
```bash theme={null}
fly scale count 2
```
> [!NOTE]
>
> **Note:** When you scale all your app's Machines to zero, the Machine size resets to shared-cpu-1x with 256MB of RAM. Use one of the `--vm-` options to size the Machines. For example: `fly scale count 2 --vm-size shared-cpu-4x`. See the [fly scale count docs](/flyctl/cmd/fly_scale_count) for options.
### Deploy the app to get new Machines
In most cases, deploying a new version of your app with `fly deploy` should also work for an app without volumes, because the new Machines will be deployed to healthy hosts.
> [!NOTE]
>
> **Important:** Running `fly deploy` during a host outage generally won't destroy the existing Machine. When the impacted host returns, the Machine that was on it may return as well. In the case of deploying during an outage, this can result in one Machine running an older version of your code. In that case you should destroy the outdated Machine if it comes back.
## Apps with one Machine and an attached volume
Volumes are pinned to physical hosts, so when there's a host outage the volume is unreachable. In most cases, you should be able to get your app running again by restoring from a snapshot into a new volume.
> [!WARNING]
>
> **Warning:** We take snapshots once every 24 hours. Any data stored between the time when the snapshot was taken and the time when the restore is made will not be included.
1. Run `fly volumes list` and copy the volume ID.
2. Run `fly volumes snapshots list ` to list the volume’s snapshots and then copy the snapshot ID.
3. Run `fly volumes create --snapshot-id -s ` to create a new volume from a stored snapshot.
4. Run `fly scale count 1` to create a new Machine that will pick up the new volume.
## Fly Postgres database apps
If you're running a high availability Postgres cluster with multiple nodes, a host issue impacting one of your nodes shouldn't cause an issue; by default we run each node on a separate host. If the host your primary node is on goes down, the cluster will fail over to a healthy node.
However, if your database is running on a single Machine, and you don’t have any replicas to fail over to, then you won’t be able to connect during the host outage. Similar to the single volume steps above, you can create a new Postgres app from your most recent volume snapshot using `fly postgres create --snapshot-id --image-ref `. See [Backup, restores, & snapshots](/unmanaged-postgres/managing/backup-and-restore) for details.
## Prevent downtime when there's a single host issue
Fly.io strongly recommends running at least two Machines per app in your primary region and we have features that can help make [app availability and resiliency](/apps/app-availability) more affordable. When possible, we place volumes on different hosts in the primary region. If one host goes down, then your app will still have a Machine and volume on a healthy host.
## Related topics
* [App availability and resiliency](/apps/app-availability)
* [Resilient apps use multiple Machines](/blueprints/resilient-apps-multiple-machines)
* [Troubleshooting your deployment](/getting-started/troubleshooting)
* [Fly.io error codes and troubleshooting](/monitoring/error-codes)
# Autoscale Fly Machines with autostop or fly-autoscaler
Source: https://docs.fly.io/blueprints/autoscale-machines
Pair fly-autoscaler with Fly Proxy autostop/autostart to keep a fixed number of stopped Machines ready to start as traffic rises and falls through the day.
## Overview
You have an app with services that's configured to [automatically start
and stop Machines based on traffic demand](/launch/autostop-autostart). But the traffic to your app changes
significantly during the day and you don't want to keep a lot of stopped
Machines during the period of low traffic.
This guide will take you through the process of configuring the
[`fly-autoscaler` app](/launch/autoscale-by-metric) in conjunction with
[Fly Proxy autostop/autostart](/launch/autostop-autostart) to
always keep a fixed number of Machines ready to be quickly started
by Fly Proxy.
## Configure autostop/autostart
First, if you haven't already done so, configure the app to allow Fly Proxy to automatically start and
stop or suspend Machines based on traffic demand. The autostop/autostart settings apply
per service, so you set them within the `[[services]]` or `[http_service]`
sections of `fly.toml`:
```toml theme={null}
...
[[services]]
...
auto_stop_machines = "stop"
auto_start_machines = true
min_machines_running = 0
...
```
With these settings Fly Proxy will start an additional Machine if all the
running Machines are above their concurrency `soft_limit` and stop running
Machines when the traffic decreases. You can set Machines to `"suspend"` rather than
`"stop"`, for even faster start-up, but with some [limitations on the type of Machine](https://community.fly.io/t/new-feature-in-preview-suspend-resume-for-machines/20672#current-limitations-and-caveats-8).
In the next section you'll configure
and deploy `fly-autoscaler` to ensure that the app always has a spare stopped
Machine for Fly Proxy to start.
## Configuring and deploying fly-autoscaler
`fly-autoscaler` is a metrics-based autoscaler that scales an app’s Machines
based on any metric. You can configure it to ensure that there is always
additional Machine available for Fly Proxy to start if the traffic increases.
First, create a new Fly.io app that will run the autoscaler.
```
$ fly apps create my-autoscaler
```
Create a deploy token so that the autoscaler app has permissions to scale your
target app up and down:
```
$ fly tokens create deploy -a my-target-app
$ fly secrets set -a my-autoscaler --stage FAS_API_TOKEN="FlyV1 ..."
```
Create a read-only token so that the autoscaler app has access to a Prometheus instance:
```
$ fly tokens create readonly
$ fly secrets set -a my-autoscaler --stage FAS_PROMETHEUS_TOKEN="FlyV1 ..."
```
Configure your autoscaler `fly.toml` like this:
```toml theme={null}
app = "my-autoscaler"
[build]
image = "flyio/fly-autoscaler:0.3.1"
[env]
FAS_PROMETHEUS_ADDRESS = "https://api.fly.io/prometheus/my-org"
FAS_PROMETHEUS_METRIC_NAME = "running_machines"
FAS_PROMETHEUS_QUERY = "count(fly_instance_up{app='$APP_NAME'})"
FAS_APP_NAME = "my-target-app"
FAS_CREATED_MACHINE_COUNT = "min(running_machines + 1, 10)"
FAS_INITIAL_MACHINE_STATE = "stopped"
[metrics]
port = 9090
path = "/metrics"
```
With this configuration, the autoscaler will create a new stopped Machine as
soon as all available Machines are running (but never more than 10), and will destroy extra stopped
Machines if more than one Machine is stopped.
Make sure you are using autoscaler version 0.3.1 or newer for
`FAS_INITIAL_MACHINE_STATE` configuration option to work.
And finally, deploy the autoscaler, using the `--ha` option to deploy only one Machine:
```
$ fly deploy --ha=false
```
## Related reading
* [Autoscale based on metrics](/launch/autoscale-by-metric)
* [Autostop/autostart Machines](/launch/autostop-autostart)
* [Autostop/autostart private apps](/blueprints/autostart-internal-apps)
# Autostart and autostop private apps with Flycast
Source: https://docs.fly.io/blueprints/autostart-internal-apps
Let Fly Proxy start and stop Machines for a private app on demand by giving it a Flycast address instead of public IPs, for new apps or existing ones.
## Overview
You have a private, or internal, app that communicates only with other apps on your [private network](/networking/private-networking). This private app might be a database, authentication server, or any other "backend" app that you don't want exposed to the public Internet. You want the app's Machines to stop when they're not serving requests from your other apps, and start again automatically when needed.
To use Fly Proxy autostop/autostart you need to configure services in `fly.toml`, like you would for a public app. But instead of using a public Anycast address, you assign a Flycast address to expose those services only on your private network.
This guide focuses on using autostop/autostart to control Machines based on incoming requests. But when you use Flycast for private apps you also get other [Fly Proxy features](/reference/fly-proxy) like geographically aware load balancing.
Learn more about [Flycast](/networking/flycast).
## Create a new private app with a Flycast address
When you run `fly launch` to create a new app, it automatically assigns your app a public IPv6 address and a shared public IPv4 address. If you know your app won't need to be reachable from the Internet, you can inform Fly Launch to assign a Flycast private IPv6 address instead:
```
fly launch --flycast
```
Next steps: [Configure and deploy a private app](#configure-and-deploy-a-private-app).
## Use Flycast for an existing app
If you already have an app and you want to make it private and use Flycast, it's important to make sure you remove the app's public IP addresses.
### Add a Flycast address
```
fly ips allocate-v6 --private
```
### Remove public IP addresses
List your IPs to check whether your app has public IP addresses:
```
fly ips list
```
Example output:
```
VERSION IP TYPE REGION CREATED AT
v6 2a09:8280:1::2d:1111 public (dedicated) global Sep 1 2023 19:47
v6 fdaa:2:45b:0:1::11 private global Mar 16 2024 18:20
v4 66.241.124.11 public (shared) Jan 1 0001 00:00
```
This example app has public IPv4 and IPv6 addresses. These public addresses are automatically assigned to an app with services on the first deploy.
Copy the public IP addresses and run the `fly ips release` command to remove them from your app:
```
fly ips release ...
```
For example:
```
fly ips release 2a09:8280:1::2d:1111 66.241.124.11
```
Next steps: [Configure and deploy a private app](#configure-and-deploy-a-private-app) below.
## Configure and deploy a private app
Whether you're creating a new app or making an existing app private, there are a few things you'll need to check or configure.
### Add services in your `fly.toml` config file
If your app was private, you might not have configured an `[http_services]` or `[services]` section in `fly.toml` because you didn't want it reachable through the public Internet. Now that you removed the public IPs, you can safely add services to allow access to the app on your private network and enable Fly Proxy to control Machines and load balance traffic.
Here's an example `fly.toml` snippet:
```toml theme={null}
[http_service]
# the port on which your app receives requests over the 6PN
internal_port = 8081
# must be false - Flycast is http-only
force_https = false
# Fly Proxy stops Machines based on traffic
auto_stop_machines = "stop"
# Fly Proxy starts Machines based on traffic
auto_start_machines = true
# Number of Machines to keep running in primary region
min_machines_running = 0
[http_service.concurrency]
type = "requests"
# Fly Proxy uses this limit to determine Machine excess capacity
soft_limit = 250
```
> [!NOTE]
>
> **Important:** Set `force_https = false`; Flycast only works over HTTP. HTTPS isn't necessary because all your private network traffic goes through encrypted WireGuard tunnels.
Learn more about [app configuration](/reference/configuration) and [Fly Proxy autostop/autostart](/launch/autostop-autostart).
### Make sure your app binds to `0.0.0.0:`
To be reachable by Fly Proxy, an app needs to listen on `0.0.0.0` and bind to the `internal_port` defined in the `fly.toml` configuration file.
### Deploy the app
Run `fly deploy` for the configuration changes to take effect.
Other apps in your organization can now reach your private app using the [Flycast](/networking/flycast) IP address or the `.flycast` domain.
# Connect AWS and other clouds with WireGuard
Source: https://docs.fly.io/blueprints/bridge-deployments-wireguard
Join a server on AWS or another cloud to your Fly.io private network as a site-to-site WireGuard peer, so Machines and outside services talk privately.
## Overview
Sometimes you can recreate production on Fly.io without any issues. Other times you need to be able to incrementally move things over. Fly.io [has private networking](/networking/private-networking) by default for apps in the same organization, but you can also easily connect your existing external servers to this private network. This lets you use your private network as a way to incrementally move services over.
Say you have an existing service on AWS or a database with RDS that needs to be accessed over RDS. You can use that AWS instance to pivot traffic from your Fly Machines to those other services. You can also go the other way and access your Fly apps (such as [an instance of Ollama](https://fly.io/blog/scaling-llm-ollama/)) from AWS, your laptop, or any other self-hosted server you have in your arsenal. Or maybe you're making a tool for a support team, and it needs to hit that one database on prem.
Fly.io private networks use [WireGuard](https://www.wireguard.com/) internally, and when you connect other computers to that network, you do connect to them over WireGuard. WireGuard is used by many people for many reasons, but the primary use is a "site-to-site" VPN like this. This won't route all of your traffic through Fly.io's gateway servers, but it will give you internal access to your private network from anywhere.
## Weaving the networks together
Make sure you have [WireGuard](https://www.wireguard.com/install/) installed on your target server. Their [install page](https://www.wireguard.com/install/) has more details, but on an Ubuntu server all you need to do is:
```docker theme={null}
sudo apt -y install wireguard
```
Once that's done, you're off to the races!
On your laptop, make the config for the target server with `fly wireguard create`. You'll need the following information:
* The organization you want to join that server to (such as `personal`, or your company's name).
* The region you want to create the peer in: you should choose [the closest region to the target computer](/reference/regions).
* If you're really not sure which region is the closest, here's a magic command you can try: `curl -Iso /dev/null -w '%header{fly-request-id}' https://fly.io | cut -d- -f2`.
* The name you want to use for that computer on your Fly network, such as the hostname.
* A path to put the generated WireGuard config, such as `~/fly0.conf` to drop it as `fly0.conf` in your home directory.
For example, I'm going to create a peer in my personal organization for my server named `phantoon`. I live in Ottawa, so the closest datacenter to me is `yyz`. The command to create my peer will look like this:
```docker theme={null}
fly wireguard create personal yyz phantoon ~/fly0.conf
```
The `fly0.conf` file contains all the configuration WireGuard needs to set up a tunnel, namely:
* The private key for the node
* The local IPv6 address the node should use
* The DNS server for your private network
* Addresses, IP ranges, and public keys for the WireGuard gateway
Copy `fly0.conf` to `/etc/wireguard` on the target computer:
```docker theme={null}
scp ~/fly0.conf root@phantoon.local:/etc/wireguard/fly0.conf
```
Then connect to the computer (likely over SSH) and enable WireGuard to start on boot:
```docker theme={null}
ssh root@phantoon.local
systemctl enable --now wg-quick@fly0.service
```
This will create an interface named `fly0`:
```docker theme={null}
$ ip addr show fly0
4: fly0: mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet6 fdaa:0:641b:a7b:9285:0:a:1b02/120 scope global
valid_lft forever preferred_lft forever
```
Now try to ping the Machines API server `_api.internal`:
```docker theme={null}
$ ping _api.internal -c 4
PING _api.internal (fdaa:0:641b::3) 56 data bytes
64 bytes from fdaa:0:641b::3: icmp_seq=1 ttl=64 time=10.4 ms
64 bytes from fdaa:0:641b::3: icmp_seq=2 ttl=64 time=39.2 ms
64 bytes from fdaa:0:641b::3: icmp_seq=3 ttl=64 time=83.7 ms
64 bytes from fdaa:0:641b::3: icmp_seq=4 ttl=64 time=123 ms
```
This means that you can successfully query any [`.internal` addresses](/networking/private-networking#fly-io-internal-dns) for apps and Machines in your organization. Congrats, you're in!
## Accessing workloads outside of your private network
To access a workload outside of your private network (such as a secret store), use the DNS name `peername._peer.internal`. For example, to access the node `phantoon` from your Fly apps, connect to `phantoon._peer.internal`:
```docker theme={null}
$ ping phantoon._peer.internal
PING phantoon._peer.internal (fdaa:0:641b:a7b:9285:0:a:1b02) 56 data bytes
64 bytes from phantoon._peer.internal (fdaa:0:641b:a7b:9285:0:a:1b02): icmp_seq=1 ttl=62 time=24.4 ms
64 bytes from phantoon._peer.internal (fdaa:0:641b:a7b:9285:0:a:1b02): icmp_seq=2 ttl=62 time=37.6 ms
64 bytes from phantoon._peer.internal (fdaa:0:641b:a7b:9285:0:a:1b02): icmp_seq=3 ttl=62 time=85.4 ms
64 bytes from phantoon._peer.internal (fdaa:0:641b:a7b:9285:0:a:1b02): icmp_seq=4 ttl=62 time=197 ms
```
## Homework
Getting ping working should be good enough to get you started, but here's some optional homework in case you want to see what you can really do with this:
* Connect your laptop/workstation to your private network with the same flow (you may need to install the GUI WireGuard app if you use Windows or macOS).
* Expose Prometheus metrics on port 9195 of your app and then grab the current metrics with `curl yourapp.internal:9195/metrics` on your laptop.
* Install Postgres on your laptop somehow. Configure your app to connect to that Postgres database on your laptop over WireGuard.
* Connect a few other computers to your private network. Install a Minecraft server on one of them and play together.
Hint: you may need to allow private network addresses through your firewall. Check the documentation of your firewall tool of choice and allow traffic from `fly0` through as if it's an "internal" interface.
## Related Reading
* [Jack into your private network with WireGuard](/blueprints/connect-private-network-wireguard) A guide showing how to connect a laptop or external server to your Fly.io private network via WireGuard.
* [Private Networking](/networking/private-networking) Overview of Fly’s internal IPv6 mesh (6PN), how apps talk to each other privately, and how WireGuard‑based tunnels integrate.
* [Custom Private Networks](/networking/custom-private-networks) How to isolate tenants, apps, or services by creating separate private networks inside your organization — relevant when bridging external infrastructure.
# Cell-based architecture
Source: https://docs.fly.io/blueprints/cell-based
Give each user of your app a dedicated Fly Machine that stops when idle and restarts on access, routed with fly-replay and fly-force-instance-id.
Previously know as
[shared-nothing-architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture),
the concept was rediscovered and rebranded as
[cell-based](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html)
by Amazon.
This blueprint shows you how to implement a scenario where each user of your
application is assigned a new Machine. These Machines, by default, will
automatically stop when not in use and be restarted when accessed, so the
primary expense will be for volumes and actual usage.
> [!WARNING]
>
> **Warning**: if you follow this example, applications will be deployed without
> their databases being replicated. This exposes your application to volume and
> Machine failures. Backups become your responsibility. Some approaches to
> addressing this requirement are listed at the [bottom of this
> page](#backups).
The example given here will be based on Rails, but the approach applies to all
frameworks.
## Start With a Single Machine Application
If you don't have one, the following will do:
```
rails new blog --css tailwind
cd blog
bin/rails generate scaffold Post title:string body:text
fly launch --name blog-$USER-$RANDOM
```
Feel free to substitute a different name for your application.
## Configure Your Routes
Modify `config/routes.rb`. Wrap your routes with:
```ruby theme={null}
scope ENV.fetch("FLY_MACHINE_ID", "") do
...
end
```
If present, move `get "up"` line outside of this block.
Uncomment the `root` line.
Your final result will look something like this:
```ruby theme={null}
Rails.application.routes.draw do
scope ENV.fetch("FLY_MACHINE_ID", "") do
resources :posts
# Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html
# Defines the root path route ("/")
root "posts#index"
end
# Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500.
# Can be used by load balancers and uptime monitors to verify that the app is live.
get "up" => "rails/health#show", as: :rails_health_check
end
```
What this will do is place your entire application, minus the health check
endpoint, into a namespace when deployed to fly.io. Access to your application
will need to be prefixed by your Machine id. When run in environments where
the `FLY_MACHINE_ID` environment variable is not set, no prefix is required.
You can get a list of Machine ids using:
```
fly machines list -q
```
## Starting a Second Machine
In order to prepare for a multiple Machine deployment, we are going to need to
ensure that each request is routed to the correct Machine. We can accomplish
this via middleware that makes use of the
[fly-replay](/networking/dynamic-request-routing#how-fly-replay-works)
response header.
Place the following into `config/initializers/fly_router.rb`:
```ruby theme={null}
if ENV["FLY_MACHINE_ID"]
class FlyInstanceRouter
def initialize(app)
@app = app
end
def call(env)
segments = env["PATH_INFO"].split('/')
instance = segments[1]
if instance =~ /^[0-9a-f]{14}$/ and instance != ENV["FLY_MACHINE_ID"]
return [409, {"Fly-Replay" => "instance=#{instance}"}, [""]]
end
@app.call(env)
end
end
Rails.application.config.middleware.use(FlyInstanceRouter)
end
```
Deploy this change using `fly deploy`, then use the following command to create
a second Machine:
```
fly machine clone
```
At this point, you have two instances of the blog application, each on its own
Machine, with its own volume and sqlite3 database.
Note that [fly machine clone](/flyctl/cmd/fly_machine_clone) has a
number of options that can be used to specify such things as the region, vm
sizes, and whether the volume requires a unique zone.
Regions, in particular, can enable you to put both applications *and* data
close to users.
## Optimizing your routes
This section is optional, and depends on a unique feature of Rails'
[Hotwire](https://hotwired.dev/) implementation.
At this point, many requests will end up making two hops: first to a nearby
Machine then to the desired Machine. This increases latency of responses,
particularly when it is necessary to wake two Machines to process the first
request.
Not much can be done to avoid this for the first request, but subsequent
requests can route to the correct Machine using the `fly-force-instance-id`
request header.
Place the following into `app/javascript/controllers/fly_router_controller.js`:
```js theme={null}
import { Controller } from "@hotwired/stimulus"
// Connects to data-controller="fly-router"
export default class extends Controller {
connect() {
console.log('fly router connected')
this.instance = this.element.dataset.instance;
document.documentElement.addEventListener(
'turbo:before-fetch-request',
this.beforeFetchRequest
)
}
disconnect() {
document.documentElement.removeEventListener(
'turbo:before-fetch-request',
this.beforeFetchRequest
)
}
beforeFetchRequest = event => {
console.log('injecting ' + this.instance);
event.detail.fetchOptions.headers['fly-force-instance-id'] = this.instance;
}
}
```
Finally, modify the `` line in `app/views/layouts/application.html.erb`:
```erb theme={null}
data-instance="<%= ENV['FLY_MACHINE_ID'] %>" data-controller="fly-router"<% end %>>>
```
Deploy this change using `fly deploy`.
## Next steps
The above merely amounts to a proof of concept. A production application will
likely make use of one or more of the following:
* While putting the Machine id into the URL path is effective,
it isn't very ergonomic. A better solution would be have a registry of
paths and their associated Machines.
* Implementing routing in middleware gets the job done, but
separating this out to a separate process using an application like
[`nginx`](https://nginx.org/en/) can have a number of advantages:
* `fly-replay` is limited to request payloads of one megabyte or less,
which would affect features like file uploads. A [reverse
proxy](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/)
might be a better solution.
* support for custom subdomains as an alternative to path namespaces and
scopes.
* authentication and serving static assets can be performed outside of your
application.
* Add-ons like [Phusion Passenger](https://www.phusionpassenger.com/) can
enable you to run multiple instances of your application in one Machine.
* Monitoring becomes more crucial when you have hundreds or even dozens of
independent Machines. Fly.io will combine your logs and while this
addresses many problems, it doesn't help deal with logs that are missing due
to application or network problems. Consider having applications emit a
heartbeat, and write monitoring software that looks for missing heartbeats
and reports them as issues.
* While applying updates when all of the services for a single Machine are
self contained is an easier problem then upgrading potentially
interdependent services running live in production, it does come at a cost:
a blue/green deployment is not possible so a focus on reducing startup times
is necessary.
* Consider building an administrative web based interface for your
application. User registration and Machine assignments are likely coupled
in your workflow. While Fly.io provides a [Machines API](/machines/api),
you can go a long way with old-fashioned scripting using the [flyctl](/flyctl) command.
See [Showcase
Architecture](https://github.com/rubys/showcase/blob/main/ARCHITECTURE.md) for
an example of an application running in production that implements many of
these ideas.
## Backups
As mentioned above, backups are crucial. Items to explore:
* [LiteFS](/litefs) - Distributed SQLite. By itself, it
supports failover situations naturally. Additional [disaster
recovery](/litefs/backup) options are available.
* Sqlite3 databases are just files. Build a separate application to host
backups and have your application periodically POST copies there.
* [Rsync](https://rsync.samba.org/) is a utility available with Linux
distributions that can be used to efficiently copy changes between Machines.
* Run multiple Machines per cell so that you get the full benefits of
traditional high availability configurations. If you have implemented an
admin web UI, you can automate the deployment of new clusters easily.
# Connect to your private network with WireGuard
Source: https://docs.fly.io/blueprints/connect-private-network-wireguard/index
Create a WireGuard peer with fly wireguard create and connect your laptop to your Fly.io organization's private network to reach apps and Flycast services.
[YouTube video player](https://www.youtube-nocookie.com/embed/4NcvlIlIlso?si=DPbDPwzlRTQFx0hB)
Every [Fly.io](http://Fly.io) organization comes with a [private network](/networking/private-networking) that lets all your apps connect to each other. This is super convenient when you need to have microservices call each other’s endpoints or use [Flycast](/networking/flycast) to let your private apps turn off when you’re not using them. However, this isn’t just limited to your apps. You can jack into this network with WireGuard.
This blueprint shows you how to create a WireGuard peer to your private network and connect to it so that you can access it from anywhere.
## Prerequisites
To get started, you need to have the following:
* A [fly.io](http://fly.io) account
* `flyctl` installed ([https://fly.io/docs/hands-on/install-flyctl/](https://fly.io/docs/hands-on/install-flyctl/))
* The [WireGuard client](https://www.wireguard.com/install/) installed
## Steps
To create a WireGuard peer, you need the following information:
* The organization you want to create the peer in, such as your personal organization.
* The [Fly.io region](/reference/regions) that’s closest to you.
* The name of the peer, such as your computer’s hostname.
* A filename to save the configuration to.
```
$ fly wg create --help
Add a WireGuard peer connection to an organization
Usage:
fly wireguard create [org] [region] [name] [file] [flags]
```
You can figure out your list of organizations with `fly orgs list`:
```
$ fly orgs list
Name Slug Type
---- ---- ----
Xe Iaso personal PERSONAL
devrel-demos devrel-demos SHARED
```
You can figure out which region is nearest you with `fly platform regions` or by checking the [regions page](/reference/regions) in the documentation:
```
$ fly platform regions
```
With all this in mind, let’s assemble the command. Each of these steps is going to build up the command, but don't hit enter until it's all done.
Start with the base command `fly wireguard create`:
```
$ fly wireguard create
```
I want to create this in my personal organization, so I’ll enter in personal for the organization name.
```
$ fly wireguard create personal
```
I’m in Ottawa, so I’m using the Montreal region `yul`.
```
$ fly wireguard create personal yul
```
My computer’s hostname is Camellia, so I’ll use that as the peer name.
```
$ fly wireguard create personal yul camellia
```
Finally I want to save this as camellia.conf so that WireGuard can load it.
```
$ fly wireguard create personal yul camellia camellia.conf
```
Then run the command (you can hit enter now), and once it’s done open up the WireGuard app.
Import the tunnel from the configuration file and then turn it on. macOS may prompt if you want the WireGuard app to manage VPN connections. If it does, hit accept, otherwise you won’t be able to get into your network.
The above video shows how you import a WireGuard config on macOS. Here's a summary of the steps:
* Click on the "Import tunnel(s) from file" button.
* Select the configuration file you saved earlier.
* Click on the "Activate" button.
* macOS will ask you if you want to allow the WireGuard app to manage VPN connections. Click "Allow" if you trust it.
On a Linux system you can use [wg-quick](https://www.man7.org/linux/man-pages/man8/wg-quick.8.html):
```sh theme={null}
wg-quick up camellia.conf
```
To test the connection, ping `_api.internal` (on macOS you need to run `ping6 _api.internal` because it's an IPv6 address):
```
$ ping6 _api.internal -c4
PING6(56=40+8+8 bytes) fdaa:3:9018:a7b:9285:0:a:602 --> fdaa:3:9018::3
16 bytes from fdaa:3:9018::3, icmp_seq=0 hlim=64 time=9.741 ms
16 bytes from fdaa:3:9018::3, icmp_seq=1 hlim=64 time=49.103 ms
16 bytes from fdaa:3:9018::3, icmp_seq=2 hlim=64 time=97.667 ms
16 bytes from fdaa:3:9018::3, icmp_seq=3 hlim=64 time=14.726 ms
--- _api.internal ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 9.741/42.809/97.667/35.111 ms
```
To test the connection with an instance of [Ollama](https://ollama.com), you can fire it up with the following commands in an empty folder (don't forget to delete it after!):
```
fly launch --from https://github.com/fly-apps/ollama-demo --no-deploy
fly ips allocate-v6 --private
fly deploy
```
Then you can set the `OLLAMA_HOSTNAME` environment variable to the hostname of the instance you just created (it will be something like `xe-ollama.flycast`), and then run the following commands:
```
export OLLAMA_HOST=http://xe-ollama.flycast
ollama run llama3 "Why is the sky blue? Explain in a single sentence."
```
And then the model will reply with something like this:
> The sky appears blue because of a phenomenon called Rayleigh scattering, where shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by tiny molecules of gases like nitrogen and oxygen in the Earth's atmosphere.
And there you go! You're in your private network and can access all your apps and Machines like they were right next to you.
# Route requests to per-user Fly Machines
Source: https://docs.fly.io/blueprints/connecting-to-user-machines
Route HTTP requests to per-user Fly Machines through a router app with fly-replay, including port matching, authenticating replayed requests, and antipatterns.
## Overview
When running machines for end users, a common challenge is efficiently managing and routing requests to these machines. This guide outlines the recommended approach for connecting to user machines on Fly.io.
## Typical Stack
* **Coordinator**: Your app that manages machines on behalf of end users. This can also function as the Router.
* **Router**: An app that relays requests to user machines. This can be the same as the Coordinator.
* **User App**: Apps assigned to specific users on isolated networks.
* **User Machine**: Machine(s) in those apps.
## HTTP Services
A typical setup includes two services:
* A **management** HTTP service (`port 9090`) for tasks such as:
* Sending LLM generated code changes
* Health checks
* Metrics requests
* A **public** HTTP service (`port 4443`) for:
* Live Previews
* LLM \<-> MCP Server Connectivity
* Deployed User Apps
### HTTP Request Processing
This diagram illustrates how HTTP requests are processed:
* Public HTTP requests arrive at your router app, which issues a fly-replay to a specific user app. The Fly Proxy then replays the request to the target app, matching the service based on the inbound request port.
* Private HTTP requests arrive at an internal management gateway, which is not accessible from the internet. The same replay process gets that request to the user app on the specified port, matching the service based on the inbound request port.
## Using fly-replay
The recommended mechanism for connecting to user machines over HTTP is the `fly-replay` response header. This works by instructing our global proxy to reroute the request to a specific machine. If the machine is not running, the proxy will start it. This approach works for any kind of HTTP request, including WebSockets. Since our proxy manages the entire request, your router app is no longer a point of failure after it sends the `fly-replay` header.
### Port Matching
When using fly-replay, the proxy matches the incoming request's port to the target service's public port. For example:
* A request to port 80 will be routed to a service configured with `port: 80`
* A request to port 9090 will be routed to a service configured with `port: 9090`
This means your router app needs to receive requests on the same ports that your target services are configured to use. For example, to connect to a management API on port 9090, the router must receive the request on port 9090.
### Private Network Router Example
The [private-network-router](https://github.com/fly-apps/private-network-router) example shows how to build a router that can handle multiple services. It uses flycast to keep management APIs private while still allowing routing to them. The router:
1. Receives requests on any port
2. Determines the target app and machine based on the request
3. Sends a `fly-replay` response header with parameters like `app` and `instance` to route the request to the correct app and machine. For more details on the parameters, see the [fly-replay documentation](/networking/dynamic-request-routing#how-fly-replay-works).
This pattern allows you to:
* Keep management APIs private by running them with flycast
* Route requests to multiple services from a single router
* Maintain security by not exposing management ports directly to the internet
## Authentication
Validation of replayed requests should happen within the user machine. We recommend using the `state` mechanism in fly-replay for this purpose:
1. **Create a preshared key for each user app** and set it as an app secret.
2. When issuing a replay, set the header: `fly-replay: app=,state=`.
3. On the user machine, parse the `fly-replay-src` header. This header contains fields such as `instance`, `region`, `t`, and `state` (see [fly-replay documentation](/networking/dynamic-request-routing#how-fly-replay-works)).
4. Check the `state` value in `fly-replay-src` against the preshared key for the app. If it matches, the request is validated.
See the [TypeScript example below](#example-authenticating-a-replayed-request-in-typescript) for a practical implementation of this validation.
## Antipatterns
* **flycast, \.fly.dev, and Dedicated IPs**: These are meant for permanent apps that need to communicate with each other. They and involve more moving parts than simply routing with `fly-replay`. More moving parts mean more potential for issues.
* **TCP or UDP**: Avoid using raw TCP or UDP when communicating with end user machines. If necessary, consider running Tailscale containers alongside your code.
## Example Service Configuration
Here's an example service configuration for a machine that includes both a management service and a standard public HTTP service:
```json theme={null}
{
"services": [
{
"name": "management",
"internal_port": 9090,
"ports": [
{
"port": 9090,
"handlers": ["http"]
}
]
},
{
"name": "public",
"internal_port": 8080,
"ports": [
{
"port": 80,
"handlers": ["http"]
}
]
}
]
}
```
This configuration sets up:
* A management service listening internally on port 9090 and exposed on port 9090
* A public service listening internally on port 8080 and exposed on port 80
Both services will automatically start when needed and stop when idle.
### Example: Authenticating a Replayed Request in TypeScript
Here's a simple example of how you might authenticate a replayed request in a TypeScript HTTP handler using the `fly-replay-src` header and a preshared key:
```typescript theme={null}
// Example preshared key for the app (in practice, load from env or secret store)
const PRESHARED_KEY = process.env.PRESHARED_KEY;
function parseFlyReplaySrc(header: string | undefined) {
if (!header) return {};
return Object.fromEntries(
header.split(',').map(pair => {
const [key, value] = pair.split('=');
return [key.trim(), value?.trim()];
})
);
}
async function authenticateReplay(request: Request): Promise {
const flyReplaySrc = request.headers.get('fly-replay-src');
const params = parseFlyReplaySrc(flyReplaySrc);
return params.state === PRESHARED_KEY;
}
// Example usage in a fetch handler
addEventListener('fetch', (event) => {
event.respondWith(handleRequest(event.request));
});
async function handleRequest(request: Request): Promise {
if (!(await authenticateReplay(request))) {
return new Response('Unauthorized', { status: 401 });
}
// ...handle the authenticated request...
return new Response('OK');
}
```
This code checks the `fly-replay-src` header, parses out the `state` value, and compares it to the preshared key. If it matches, the request is authenticated.
## Related reading
* [Per‑User Dev Environments with Fly Machines](/blueprints/per-user-dev-environments) Find out how to run isolated, on‑demand environments for each user.
* [Observability for User Apps](/blueprints/observability-for-user-apps) Learn about how to gather logs and telemetry from per‑user apps/machines once you’ve routed traffic to them.
* [Dynamic Request Routing with fly‑replay](/networking/dynamic-request-routing) Learn about the `fly‑replay` header and dynamic request routing.
# Custom deploy workflows: staggered and selective deploys
Source: https://docs.fly.io/blueprints/custom-deploy-workflows
Roll out fly deploy to stopped Machines, one region, or one process group at a time, to deploy gradually without restarting Machines running long jobs.
## Overview
> [!NOTE]
>
> **This guide shows how to take control of your deployment flow, whether you're trying to avoid restarting machines, roll out changes gradually, or update specific parts of your app without touching others.**
Most people think of `fly deploy` as an all-or-nothing operation: you push a new image, and all your machines restart to run it. That behavior makes sense for many apps, but your app might need a more granular approach. Maybe you want to avoid interrupting machines processing long-running jobs (like video encoding or LLM chat sessions), or you want to gradually roll out changes to certain regions first.
The `fly deploy` command supports several flags for targeting specific machines by ID, region, or process group, giving you the control to tailor your rollout strategy to your app's needs.
## Why you might want this
These strategies are useful when you want to roll out code without disrupting active work. Maybe you're:
* Handling long-running tasks that shouldn't be interrupted mid-process
* Running workloads with unpredictable activity spikes where idle machines should be reused
* Trying to deploy more cautiously in production by segmenting your rollout
* Building your own logic to manage machine lifecycle instead of relying on a central deploy command
They're especially useful for apps that:
* Encode media, process jobs, or run background workers
* Exit cleanly after finishing a task
* Benefit from reusing stopped machines instead of spinning up new ones constantly
## Staggered and selective deploys
You don’t have to update your whole fleet all at once. Fly gives you several ways to target machines for more careful rollouts.
### By status: stopped machines
If your machines exit when they finish work, you can update only the stopped ones. This is useful for long-running tasks, background workers, or apps using `auto_stop_machines`.
```bash theme={null}
# Build the new image and push it to the registry, but don't deploy it yet
fly deploy --build-only --push
# List all machines, filter for those in a "stopped" state, and save their IDs
fly machines list --json \
| jq -r '.[] | select(.state == "stopped") | .id' \
> stopped-machines.txt
# Deploy the new image only to the stopped machines
fly deploy --image THE_IMAGE \
--only-machines $(cat stopped-machines.txt | paste -sd, -)
```
Repeat this process until all machines have rotated. This works best when:
* You're using `auto_stop_machines`
* Your app exits cleanly when done (`System.exit(0)` or equivalent)
* You want machines to manage their own lifetimes
### By region
To contain the blast radius or do a phased rollout, target machines by region:
```bash theme={null}
# Deploy the current app image only to machines in the "dfw" region
fly deploy --regions dfw
```
Or combine with `--only-machines` to target specific IDs within a region:
```bash theme={null}
# List all machines, filter for those in the "dfw" region, and save their IDs
fly machines list --json \
| jq -r '.[] | select(.region == "dfw") | .id' \
> dfw-machines.txt
# Deploy to just the first two machines from that list
head -n 2 dfw-machines.txt > batch-1.txt
fly deploy --image THE_IMAGE \
--only-machines $(paste -sd, - < batch-1.txt)
# Deploy to the next three machines that weren't in batch 1
head -n 5 dfw-machines.txt | grep -vxFf batch-1.txt > batch-2.txt
fly deploy --image THE_IMAGE \
--only-machines $(paste -sd, - < batch-2.txt)
```
Extend this rollout gradually by increasing the number of machines targeted in subsequent deploys. If you want to go further, you can script this logic to apply region by region, or even automate it based on health checks or metrics from your app.
### By process group
In multi-process apps, you might want to update only certain roles while leaving others untouched. For example, consider a cluster with `zk`, `supervisor`, and `conductor` process groups. If the `zk` machines need to stay up during an upgrade, you can deploy only to the other groups:
```bash theme={null}
# Deploy the new image only to the "supervisor" and "conductor" process groups
fly deploy --image THE_IMAGE --process-groups supervisor,conductor
```
This allows the `zk` group to keep running on the previous image until you're ready to update them. This pattern is helpful for maintaining quorum or minimizing downtime in clustered environments.
## Handling partial rollout failures
Sometimes an image rolls out cleanly to some machines but fails on others. You should:
* Watch for startup errors or unhealthy states
* Retry failed updates individually
* Pause rollouts if failure rates climb
Eventually, you might want to implement an automated rollback based on health checks or deploy output. For now, just keeping track of which machines succeeded and which didn’t is a good start.
## CI integration
If you're using GitHub Actions or another CI, you can bake this deployment pattern in. For instance:
* `fly deploy --build-only --push` on main
* Update stopped machines using the image tag as part of a scheduled job
* Optionally notify Slack or GitHub comments when a full rollout completes
This lets you deploy continuously without interrupting running tasks.
## Dynamic config updates
Not every change needs a new image. For apps that load config from volumes or environment variables, you can:
* Update secrets or config maps with `fly secrets set`
* Restart idle machines to pick up the change
* Let active machines shut down naturally and restart with updated config
This is especially useful when your app supports live reloading or watches config files.
## Things to watch out for
### Forgetting to update all machines
If your script misses some stopped machines, or if others stop later, they might keep running an old image. Make sure your update loop is thorough and runs long enough to catch all of them.
### Long-lived machines never get updated
If your deployment strategy only updates stopped machines but your machines never exit on their own, they'll never upgrade. The most robust solution is to ensure you're using `auto_stop_machines` and that your app exits cleanly when it's done. If that’s not possible, you might configure your app to shut itself down after a fixed amount of time (a kind of time-to-live or TTL). Alternatively, use health checks to detect stale or idle machines, or forcibly stop them after a deadline using automation.
### Mixed-image behavior
When machines are on different image versions, your app needs to handle it. Schema changes, protocol mismatches, or config divergence can cause issues if you don’t plan for this.
### Hidden dependencies on deploy
Even if you're not restarting machines, some changes can still cause issues: environment variables, shared storage, or secrets might behave differently or get out of sync.
## Related reading
Want to go deeper on the ideas in this guide? Check out these Fly.io docs:
* [Machines overview](/machines): Learn how Machines work and what makes them different.
* [fly deploy CLI reference](/flyctl/cmd/fly_deploy): Full list of deploy options including `--only-machines`, `--regions`, and `--process-groups`.
* [Dynamic request routing](/networking/dynamic-request-routing): Use `fly-replay` to shift traffic between machines on the fly.
# Deno KV with LiteFS Cloud
Source: https://docs.fly.io/blueprints/deno-kv-litefs
Back Deno KV with a SQLite database on LiteFS to run a Deno Deploy app on Fly.io. LiteFS Cloud has been sunset, so this guide is kept for reference.
> [!WARNING]
>
> **The documentation in this section is deprecated and refers to a product that was retired on October 15, 2024.** [LiteFS](https://github.com/superfly/litefs) itself is not affected by the retirement of the LiteFS Cloud service. Learn more about how to disconnect from LiteFS Cloud and other options for disaster recovery in our community post: [Sunsetting LiteFS Cloud](https://community.fly.io/t/sunsetting-litefs-cloud/20829).
> [!NOTE]
>
> **Important:** LiteFS is outside our [scope of support](/about/support#scope-of-support). We are not able to provide support or guidance for it. Use with caution.
> [!WARNING]
>
> **Don't use LiteFS with autostop/autostart on Fly Machines.**
>
> The Fly Proxy can stop and restart Machines without knowing which Machine holds the LiteFS lease or has the latest data. This can allow a Machine with stale data to acquire the lease, causing LiteFS to discard newer changes and LTX files. The result can be a database rollback and permanent data loss.
## Problem
If you're porting an app from Deno Deploy to Fly.io, you may hit a bit of a snag: if it was built with [Deno KV](https://deno.com/blog/kv), a *traditionally* FoundationDB backed K/V store made for Deno Deploy, there's no obvious way to easily run Deno KV at scale on Fly.io. You *could* use [denoland/denokv](https://github.com/denoland/denokv) on Fly.io, but due to an alignment of the stars there happens to be an easier way.
Deno KV can actually be backed by an SQLite DB stored on disk in a cache folder, and you can specify the path of this DB with the parameter to `Deno.openKv( )`.
This means you can do something like this to get Deno KV to use an arbitrary SQLite DB:
```typescript theme={null}
const kv = await Deno.openKv("/any/path/i/want.db");
```
Now, Fly.io has some special features for SQLite users, namely [LiteFS Cloud](/litefs/speedrun), a distributed file system that transparently replicates and backs up SQLite databases across all your instances. The magic here is that you can just treat it like a local on-disk SQLite database but behind the curtain it’s doing all the work to replicate your DB.
## Setting up LiteFS
Firstly, you need to add a `litefs.yml` file to your project, and make sure it gets included in the Dockerfile.
Here’s an example:
```yml theme={null}
fuse:
dir: "/litefs"
data:
dir: "/var/lib/litefs"
exit-on-error: false
proxy:
addr: ":8080"
target: "localhost:8081"
db: "db"
passthrough:
- "*.ico"
- "*.png"
exec:
- cmd: "deno run -A main.ts"
lease:
type: "consul"
advertise-url: "http://${HOSTNAME}.vm.${FLY_APP_NAME}.internal:20202"
candidate: ${FLY_REGION == PRIMARY_REGION}
promote: true
consul:
url: "${FLY_CONSUL_URL}"
key: "litefs/${FLY_APP_NAME}"\
```
This is 99% the [sample config](https://github.com/superfly/litefs-example/blob/main/fly-io-config/etc/litefs.yml), only changing the exec command to a Deno one.
> [!NOTE]
>
> It's not immediately clear based on the example config, but your app needs to listen on "target" (:8081) and your service/http\_service in fly.toml needs to listen on "addr" (:8080). This is because part of LiteFS acts as a proxy, so you need the fly-proxy to send requests to LiteFS which then forwards them on to your app.
>
> If you’re not able to change your application’s port, make sure `target` is set up correctly then change `addr` and `fly.toml` to something other than `:8080`.
Once you've added the config, you just need to make your app look for the SQLite DB in the right location. For Deno KV, that looks like this:
```typescript theme={null}
const kv = await Deno.openKv(Deno.env.get("DB_LOCATION"));
```
Where `DB_LOCATION` is set to `/litefs/my.db` in `fly.toml` under `[env]`.
Now, if you’re deploying to Fly.io, you’re almost ready to go. [Here’s where to look if you’re not running on Fly.io](/litefs/getting-started-docker).
## Creating a LiteFS Cloud cluster
For a more in-depth article, check out [Getting Started with LiteFS on Fly.io](/litefs/getting-started-fly).
Head over to the [LiteFS section](https://fly.io/dashboard/personal/litefs) of the dashboard and create a cluster, **make a note of the auth token** (you’ll need it later).
### Dockerfile
You need to add the dependencies for LiteFS to your Dockerfile:
```docker theme={null}
# for alpine-based images
RUN apk add ca-certificates fuse3 sqlite
# or for debian/ubuntu-based images
RUN apt-get update -y && apt-get install -y ca-certificates fuse3 sqlite3
```
And copy in the LiteFS binary:
```docker theme={null}
COPY --from=flyio/litefs:0.5 /usr/local/bin/litefs /usr/local/bin/litefs
```
And update the `ENTRYPOINT`/`CMD`:
```docker theme={null}
ENTRYPOINT litefs mount
```
Now you're ready to launch your Fly app!
```bash theme={null}
# Create a volume
$ fly volumes create litefs --size 10
# Create your app without deploying
$ fly launch --no-deploy
# Attach the Fly.io provided Consul server to your app.
$ fly consul attach
# Set the secret for LiteFS cloud, replace the example value with your token from earlier
$ fly secrets set LITEFS_CLOUD_TOKEN=yoursecrettoken
```
Update your `fly.toml` to mount the LiteFS volume:
```
[mounts]
source = "litefs"
destination = "/var/lib/litefs"
```
Aaannnndddd, Deploy!
```
fly deploy
```
Your reward for running all those commands should be a running app with LiteFS! Let us know if you run into friction using LiteFS with Deno KV or any other client, you can reach out on [the Fly Community](https://community.fly.io/).
### Scaling
Deploying just one node using LiteFS doesn’t actually *do much*, so try scaling outwards! You can add a few Machines in regions close to your users like this:
```
fly scale count 3 -r yyz,ewr,waw
```
This will automatically add volumes and Machines to [scale your app](/launch/scale-count#scale-an-apps-regions) out and around the world!
# Deploy OpenClaw on Fly.io
Source: https://docs.fly.io/blueprints/deploy-openclaw
Deploy the OpenClaw AI agent gateway on Fly.io with a download-and-run script that creates the app, volume and secrets, then connect Discord, Telegram or Slack.
To deploy [OpenClaw](https://github.com/openclaw/openclaw) to Fly.io, download the deploy package and run the script. It handles everything — app creation, volumes, secrets, and deployment.
[Download Deploy Package](https://github.com/fly-apps/Openclaw-Fly-Deploy/archive/refs/heads/main.zip)
```bash theme={null}
unzip Openclaw-Fly-Deploy-main.zip
cd Openclaw-Fly-Deploy-main
bash deploy.sh
```
You'll need **[flyctl](/flyctl/install)** installed, a **Fly.io account** ([free trial](/about/free-trial) works), and an **LLM API key** (Anthropic, OpenAI, Google Gemini, OpenRouter, Moonshot AI, or MiniMax).
## What is OpenClaw?
OpenClaw is an open-source AI agent gateway that lets you run a persistent AI assistant reachable from anywhere — Discord, Telegram, Slack, or your local CLI. It manages conversations, tool use, and connections across multiple channels through a single gateway process.
## How it works
The deploy package sets up a wrapper server that manages the OpenClaw gateway and provides a browser-based setup wizard:
```
Internet → Fly.io proxy → Wrapper server (:3000) → OpenClaw gateway (:18789)
├── /setup → Setup wizard (password-protected)
├── /healthz → Health check (no auth)
└── /* → Proxied to gateway
```
All state lives on a persistent volume mounted at `/data`, so your configuration, conversation history, and installed tools survive restarts and redeployments.
The script will prompt you for:
1. **App name** — defaults to `openclaw-XXXX` (random suffix). This becomes your URL: `https://your-app-name.fly.dev`
2. **Region** — where to run your Machine (defaults to `iad` / Virginia). Pick one close to you for lower latency. See [Fly.io regions](/reference/regions) for the full list.
3. **Setup password** — protects the `/setup` wizard from the internet. Pick something strong.
4. **LLM provider** — choose from Anthropic, OpenAI, Google Gemini, OpenRouter, Moonshot AI, or MiniMax
5. **LLM API key** — the key for your chosen provider
6. **Channel tokens** (optional) — Discord bot token, Telegram bot token, or Slack bot + app tokens if you want to connect chat channels
The script then creates a Fly app, provisions a persistent volume, sets your credentials as encrypted [Fly secrets](/apps/secrets), and builds and deploys the Docker image on Fly's remote builders. The first deploy takes a few minutes. Your credentials never leave your machine — they go directly to Fly.io via `flyctl`.
## Post-deploy setup
Once deployment completes, the script prints your app details:
```
=== Deployment Complete ===
App URL: https://your-app-name.fly.dev
Setup wizard: https://your-app-name.fly.dev/setup
Gateway URL: wss://your-app-name.fly.dev
Gateway token:
```
### Setup wizard
Visit `https://your-app-name.fly.dev/setup` in your browser. Log in with any username and the setup password you chose. From the wizard you can:
* Change your LLM provider and API key
* Add or update Discord, Telegram, and Slack channel connections
* Edit the raw OpenClaw config file
* Run debug commands against the gateway
* Export and import configuration backups
* Approve device pairing requests
### Connect your local CLI
If you have OpenClaw installed locally, point it at your remote gateway:
```bash theme={null}
openclaw config set gateway.mode remote
openclaw config set gateway.remote.url wss://your-app-name.fly.dev
openclaw config set gateway.remote.token
openclaw health # verify the connection
```
This lets you use the `openclaw` CLI on your laptop while the gateway runs on Fly.io, keeping conversations and state persistent even when your laptop is closed.
## Configuration
### Secrets
All sensitive values are stored as [Fly secrets](/apps/secrets), encrypted at rest and injected as environment variables at boot. They are never visible in logs, config files, or the Fly dashboard.
| Secret | Required | Description |
| - | - | - |
| `SETUP_PASSWORD` | Yes | Protects the `/setup` wizard with HTTP Basic Auth |
| `OPENCLAW_GATEWAY_TOKEN` | Yes | Auth token for gateway connections (auto-generated) |
| `OPENCLAW_API_KEY` | Yes | Your LLM provider API key |
| `OPENCLAW_AUTH_CHOICE` | Yes | Provider identifier (set by deploy script) |
| `OPENCLAW_API_KEY_FLAG` | Yes | CLI flag for the provider (set by deploy script) |
| `OPENCLAW_DISCORD_TOKEN` | No | Discord bot token |
| `OPENCLAW_TELEGRAM_TOKEN` | No | Telegram bot token |
| `OPENCLAW_SLACK_BOT_TOKEN` | No | Slack bot token (`xoxb-...`) |
| `OPENCLAW_SLACK_APP_TOKEN` | No | Slack app token (`xapp-...`, required if bot token is set) |
To update a secret after deployment:
```bash theme={null}
fly secrets set OPENCLAW_API_KEY=sk-new-key-here -a your-app-name
```
The Machine restarts automatically when secrets change.
### VM sizing
The default configuration uses a `shared-cpu-2x` Machine with 4 GB RAM, which costs roughly \$20–25/month when running continuously. With auto-stop enabled (the default), you only pay for time the Machine is actually running.
To adjust resources:
```bash theme={null}
fly scale memory 4096 -a your-app-name
fly scale vm shared-cpu-4x -a your-app-name
```
### Persistent storage
OpenClaw stores all state on a [Fly Volume](/volumes/overview) mounted at `/data`. This includes:
* `openclaw.json` — gateway configuration
* Conversation history and context
* Installed tools and plugins
* npm/pnpm caches for user-installed packages
The default volume size is 1 GB. To extend it:
```bash theme={null}
fly volumes extend -s 3 -a your-app-name
```
## Useful commands
| Command | Description |
| - | - |
| `fly logs -a your-app-name` | Stream live logs |
| `fly ssh console -a your-app-name` | SSH into the Machine |
| `fly apps restart your-app-name` | Restart after config changes |
| `fly scale memory 4096 -a your-app-name` | Increase memory |
| `fly status -a your-app-name` | Check Machine status |
| `fly volumes list -a your-app-name` | List attached volumes |
## Troubleshooting
**"SETUP\_PASSWORD is not set"**
The setup password secret is missing. Set it:
```bash theme={null}
fly secrets set SETUP_PASSWORD=your-password -a your-app-name
```
**Out of memory / crashes**
Increase the Machine's memory:
```bash theme={null}
fly scale memory 4096 -a your-app-name
```
**Gateway won't start**
Visit `/setup` in your browser and check the debug console for errors. Common causes: invalid API key, missing config file, or a corrupted state directory.
**Lock file errors**
If the gateway didn't shut down cleanly, stale lock files can prevent it from starting:
```bash theme={null}
fly ssh console -a your-app-name
rm -f /data/gateway.*.lock
exit
fly apps restart your-app-name
```
**Need to start fresh**
Use the "Reset" button in the setup wizard to clear the config and re-run onboarding, or SSH in and remove the config:
```bash theme={null}
fly ssh console -a your-app-name
rm /data/openclaw.json
exit
fly apps restart your-app-name
```
## Supported LLM providers
| Provider | What you need |
| - | - |
| Anthropic | API key from [console.anthropic.com](https://console.anthropic.com/) |
| OpenAI | API key from [platform.openai.com](https://platform.openai.com/) |
| Google Gemini | API key from [aistudio.google.com](https://aistudio.google.com/) |
| OpenRouter | API key from [openrouter.ai](https://openrouter.ai/) |
| Moonshot AI | API key from Moonshot's developer portal |
| MiniMax | API key from MiniMax's developer portal |
You can switch providers at any time through the setup wizard — no redeployment needed.
# Run HIPAA-compliant healthcare apps
Source: https://docs.fly.io/blueprints/going-to-production-with-healthcare-apps
Evaluate Fly.io for HIPAA: encryption in transit and at rest, network isolation, and access control. Then launch a pilot, sign a BAA, and deploy to production.
## Overview
Fly.io was built by security researchers from the ground up to be both productive and secure, making it a great home for HIPAA-compliant production healthcare applications for productive teams that ship often.
This guide runs a developer or operations engineer through the process of evaluating Fly.io's security for HIPAA healthcare apps, launching a pilot application, signing a Business Associate Agreement (BAA), and deploying to production.
## HIPAA and Fly.io Primer
The Health Insurance Portability and Accountability Act (HIPAA) sets the standard for protecting sensitive patient data. Any company that processes protected health information (PHI) must ensure that all the required physical, network, and process security measures are in place and followed.
Fly.io takes a "principle of least privilege" approach to security. Here are the highlights of what healthcare app developers get out of the box on Fly.io:
### Data Protection and Encryption
HIPAA requires that PHI be encrypted both in transit and at rest to prevent unauthorized access:
* **In Transit**: Fly.io uses [WireGuard](https://fly.io/blog/our-user-mode-wireguard-year/) to encrypt data as it moves between networks, ensuring compliance with HIPAA's transmission security requirements.
* **At Rest**: Fly.io ensures data at rest is secured on [encrypted NVMe user volumes](/volumes), aligning with the encryption standards required by HIPAA for storage security.
### Access Control
Access to PHI must be limited to authorized personnel only, which is managed through:
* **Network Isolation**: Each application on Fly.io runs within its own private network by default, reducing the risk of unauthorized access.
* **Secrets Management**: Fly.io provides a [secrets manager](/apps/secrets) for storing and handling API credentials and environment variables securely, ensuring that sensitive information is accessible only to authorized applications.
* **Isolated Application Environments**: Fly.io allows organizations to create separate organizations for different environments, ensuring that production environments are isolated from test and staging environments.
* **Workload Isolation:** [Docker workloads on Fly.io are isolated at the kernel hardware level through the Firecracker VM runtime](https://fly.io/blog/fly-machines/). These workloads are hosted in secure data centers on Fly.io's dedicated hardware, ensuring HIPAA compliance by minimizing the risk of unauthorized access and maintaining strict control over PHI data.
### Audit Controls
HIPAA demands that you implement hardware, software, and procedural mechanisms that record and examine activity in systems containing PHI:
* **Logging and Monitoring**: Fly.io allows organizations to monitor and log access and usage, helping comply with HIPAA’s information system activity review requirements.
* **Version-Controlled Configuration**: Fly.io uses a Dockerfile and fly.toml configuration file for managing application deployments. These files can be checked into version control systems, like Github, allowing for comprehensive audit trails of infrastructure changes. This capability supports HIPAA's mandates for traceability and accountability in the modification and management of systems handling PHI.
### Data Integrity
To comply with HIPAA, entities must ensure that PHI is not altered or destroyed in an unauthorized manner:
* **Data Storage and Backup**: Fly.io’s infrastructure provides robust data integrity controls, including [regular snapshots](/volumes/volume-manage) of [encrypted volumes](/volumes/overview#volume-encryption) to prevent data loss.
* **Reproducible Deployments**: Fly.io's use of Docker ensures that application deployments are reproducible and can be locked down to a mostly read-only state. This reduces the likelihood of unauthorized modifications to application environments, meeting HIPAA's requirements for protecting PHI from unauthorized alteration or destruction.
### Physical Security
Physical access to data centers where PHI is stored must be controlled:
* **Data Centers**: Fly.io hosts services in secure data centers with restricted access, surveillance, and environmental controls, ensuring the physical security of the hardware that stores and processes PHI.
* **Regional Data Control**: Fly.io allows users to control the [physical region](/reference/regions) where data is stored and processed using simple commands, facilitating compliance with data residency requirements of HIPAA by ensuring PHI is stored within designated secure regions, such as the United States.
There's more detail at [https://fly.io/docs/about/healthcare](/about/healthcare) and [https://fly.io/security](https://fly.io/security). If you have additional questions about Fly.io's security infrastructure & practices, please [reach out](mailto:sales@fly.io).
## Launch pilot application
The fastest way to deploy a pilot application is to run through the [Fly.io Speedrun](https://fly.io/speedrun/).
If your free demo account doesn't have enough resources for your pilot application, you may [upgrade to a paid plan](/about/pricing) or [contact us](mailto:sales@fly.io) to make other arrangements.
### Specify deployment regions
Fly automatically detects which of its 30+ regions around the globe is closest to you as your default data region.
If the default region is not where you want to host your application, run `fly platform regions` to see the most recent list of regions.
```bash theme={null}
fly platform regions
NAME CODE GATEWAY CAPACITY LAUNCH PLAN+
North America
Ashburn, Virginia (US) iad ✓ 164
Chicago, Illinois (US) ord ✓ 385
Dallas, Texas (US) dfw ✓ 426
Los Angeles, California (US) lax ✓ 635
San Jose, California (US) sjc ✓ 2399
Secaucus, NJ (US) ewr 233
Toronto, Canada yyz ✓ 70
# ... Lots more regions...
```
Then select your desired region:
```
$ fly regions add ord
```
And remove undesired regions as well:
```bash theme={null}
$ fly regions remove sin
```
Your application will automatically be deployed to the regions specified and communications between regions happen over a private, encrypted WireGuard network.
### Add API keys and credentials to the Secrets Manager
Fly.io's [secret manager stores](/apps/secrets) your application's API credentials and secrets in a secure vault. The contents remain encrypted and are only available from running Machine instances as environment variables.
### Talk to our solution architects if you have questions
If at any time during your evaluation of Fly you have questions about the infrastructure, security, or service terms you can ask in the community forums or [email Fly.io with your questions](mailto:sales@fly.io).
## Deploy to production
Once you've evaluated Fly.io and have a BAA, it's time to go to production. The remainder of this guide will walk you through how to setup an isolated production environment, control access, and deploy your application.
### Sign a Business Associate Agreement (BAA)
When you're ready to start deploying HIPAA apps, you'll need to do some paperwork (don't worry, we use digital signatures) to make sure everything is compliant.
* [Choose the Compliance Package](https://fly.io/compliance) that includes HIPAA/BAA documents.
* Sign in to the dashboard and request a signed BAA at [https://fly.io/dashboard/personal/compliance](https://fly.io/dashboard/personal/compliance) or [contact us](mailto:sales@fly.io) and we'll help.
### Provision an isolated production environment
Ensure the production environment to be separate from the test and staging environments used to test this application.
Fly uses "organizations" to control access to "applications". Since production environments require their own access control, we'll create a new organization that will contain all of our production applications.
```bash theme={null}
# Replace $MYORG with the name of your organization
$ fly orgs create $MYORG-production
```
You'll walk through the steps of creating a new organization within Fly.io. You'll also have to add a new payment method and select the appropriate plan for the new organization. If you need help during this process or have questions about how a plan spans multiple organizations, [please reach out](mailto:sales@fly.io).
### Invite team members to production environment
Grant access to team members who may have access to the production environment:
```bash theme={null}
$ fly orgs invite somebody@$MYORG.com --org $MYORG-production
```
### Launch application
Next, ensure you have a `fly.toml` file for each environment:
```bash theme={null}
$ cp fly.toml fly.staging.toml
$ mv fly.toml fly.production.toml
```
Run `fly launch` to provision the production environment.
```bash theme={null}
# Replace $MYAPPNAME with the name of your app
$ fly launch --name $MYAPPNAME-production --path fly.production.yml --org $MYORG-production
An existing fly.toml file was found for app $MYAPPNAME
? Would you like to copy its configuration to the new app? (y/N) y
```
You'll be asked to copy the configuration file. Select `y` assuming you want to use the same settings from the `fly.toml` file that was formerly staging. You'll want to commit these `fly.toml` files to your repo.
If you run into problems during or after deploy you can run `fly logs --path fly.production.yml` to see errors your application may be logging.
### Create SSL certificate and custom domain
Next, point a domain name to the new production environment via `fly certs`.
```bash theme={null}
$ fly certs add myappname.com --path fly.production.yml
```
The output of the command includes the information you need to update DNS records to point to the application on Fly.io
### Deploy application
Finally run `fly deploy --path fly.production.yml` to deploy your application, then open your application in your browser to ensure that it's up and running.
## Wrap-up
Fly.io is a secure, productive platform for deploying HIPAA-compliant healthcare applications. The platform provides the necessary security controls and features to ensure that PHI is protected in accordance with HIPAA requirements while giving development teams the flexibility to ship often and scale quickly.
## Related reading
* [Healthcare apps on Fly.io](/about/healthcare) Overview of how Fly.io supports HIPAA‑compliant apps, including encryption, network isolation, and audit‑controls.
* [Going to production checklist](/apps/going-to-production) A general production‑readiness checklist for apps on Fly.io.
# Run Hermes Agent on Fly.io
Source: https://docs.fly.io/blueprints/hermes-agent-on-fly-io
Run the Hermes AI agent from Nous Research on a Fly.io Machine with a volume for its state, configure an LLM key, and reach its web dashboard through fly proxy.
[Hermes](https://github.com/NousResearch/hermes-agent) is an AI agent from Nous Research with a built-in learning loop: it watches its own output, notices when it had to improvise to finish a task, and writes that experience back as a reusable skill it can call next time. The skill library grows with use, so Hermes works best when it runs continuously on a persistent host rather than a short-lived sandbox, which is exactly what a Fly Machine with an attached volume gives you.
This guide walks you through running Hermes on a Fly Machine, configuring it, and reaching its web dashboard from your laptop.
You'll need **[flyctl](/flyctl/install)** installed, a **Fly.io account** ([free trial](/about/free-trial) works), and an **LLM API key** (Anthropic, OpenAI, Google Gemini, or [OpenRouter](https://openrouter.ai/) for access to 200+ models).
You'll use Nous Research's **official prebuilt image** (`nousresearch/hermes-agent:latest`) so there's no Dockerfile to maintain and no remote builder to wait on. Fly pulls the image straight from Docker Hub.
## Create the app and volume
Hermes keeps all its state in `/opt/data` inside the container. That includes config, API keys, sessions, skills, and memories. You'll back that with a Fly volume so it persists across deploys and restarts.
Pick an app name (must be globally unique on Fly) and a [region](/reference/regions) close to you, then:
```bash theme={null}
fly apps create
fly volumes create data -a --region --size 3
```
3 GB is comfortable headroom for sessions and the bundled skills directory.
## Write fly.toml
Create a directory for the deployment config and drop a `fly.toml` in it:
```toml theme={null}
app = ""
primary_region = ""
machine_config = "machine_config.json"
[build]
image = "nousresearch/hermes-agent:latest"
[[mounts]]
source = "data"
destination = "/opt/data"
[[vm]]
memory = "4gb"
cpus = 2
```
The `machine_config` key points to a JSON Machine config file that defines the containers for a [multi-container Machine](/machines/guides-examples/multi-container-machines). Each entry in its `containers` array follows the [`ContainerConfig`](/api/machines/models#fly-containerconfig) schema from the Machines API.
Create that file next to `fly.toml` (called `machine_config.json` in our example):
```json theme={null}
{
"containers": [
{
"name": "hermes",
"image": "nousresearch/hermes-agent:latest",
"cmd": ["gateway", "run"]
}
]
}
```
A few notes:
* **No `[build.dockerfile]`.** Fly pulls the image directly. Deploys take seconds, not minutes.
* **Multi-container Machine for `s6-overlay`.** Hermes images published from `v2026.5.28` onward use `s6-overlay` as the in-container supervisor. s6-overlay's `/init` calls `s6-overlay-suexec`, which checks `getpid() == 1` before doing anything else and aborts otherwise. Running Hermes as a container in a multi-container Machine (via `machine_config.json`) gives it its own PID namespace, so `/init` runs as PID 1.
* **No `[[services]]` block.** The gateway talks *outbound* to chat platforms, so you don't need a public port. The dashboard exposes API keys and shouldn't be public; you'll reach it through a Fly proxy tunnel below.
* **4 GB / 2 CPU** is the recommended size when browser tools (Playwright/Chromium) are active. If you don't use browser tools you can drop to `shared-cpu-1x` and 1–2 GB.
### Optional: pass API keys as Fly secrets
This guide enters API keys through `hermes setup` after the first deploy. If you'd rather manage keys with [Fly secrets](/apps/secrets), set the secrets before you deploy:
```bash theme={null}
fly secrets set ANTHROPIC_API_KEY=sk-... -a
```
Then list each one under the container's `secrets` array in `machine_config.json`. Each entry is `{ "env_var": "" }`, where `` must exactly match a secret that already exists on the app:
```json theme={null}
{
"containers": [
{
"name": "hermes",
"image": "nousresearch/hermes-agent:latest",
"cmd": ["gateway", "run"],
"secrets": [
{ "env_var": "ANTHROPIC_API_KEY" }
]
}
]
}
```
If a listed secret doesn't exist, the Machine won't start. The deploy times out without any error in `fly logs`. Run `fly secrets list -a ` to check. See [`MachineSecret`](/api/machines/models#fly-machinesecret) for the full entry format.
## Deploy
```bash theme={null}
fly deploy -a --ha=false
```
`--ha=false` keeps it to a single Machine. By default, `fly deploy` creates two Machines for availability. Since Fly volumes can’t be shared between Machines, the second Machine would get an empty volume and run a separate Hermes instance with its own config, sessions, and skills.
When the deploy finishes, the machine boots, the entrypoint bootstraps `/opt/data` (creating `.env`, `config.yaml`, `SOUL.md`, `sessions/`, `skills/`, etc.), and `hermes gateway run` starts. It'll keep running but it has no API key yet, so it can't talk to a model.
Confirm it's alive:
```bash theme={null}
fly logs -a
```
You should see the bundled skills sync, then the gateway starting up.
## Configure Hermes
SSH into the machine. The `hermes` binary lives at `/opt/hermes/.venv/bin/hermes` inside the image, but `fly ssh console` opens a login shell that resets PATH and won't find it there. Add a symlink into `/usr/local/bin` (which is always on PATH) so `hermes` works as a bare command:
```bash theme={null}
fly ssh console -a -C \
"ln -sf /opt/hermes/.venv/bin/hermes /usr/local/bin/hermes"
```
Then open a shell on the machine and run the setup wizard:
```bash theme={null}
fly ssh console -a
hermes setup
```
The wizard walks you through model selection, tool configuration, and connecting your messaging platforms. When it's done, exit the SSH session.
Restart the machine so the gateway picks up the new config:
```bash theme={null}
fly machine restart -a
```
Get `` from `fly machine list -a `.
## Web dashboard
Hermes has a web dashboard on port 9119 for managing sessions, skills, and config. The dashboard reads your API keys, so the upstream guidance is to never expose it on a public port. Tunnel to it instead.
The Fly proxy reaches the Machine over its private IPv6 address, which requires the dashboard to listen on all interfaces, not only localhost. When Hermes binds to anything other than a loopback address, it [requires dashboard auth](https://hermes-agent.nousresearch.com/docs/user-guide/features/web-dashboard#when-the-gate-engages). Set up a username and password first.
Generate a session secret on your laptop:
```bash theme={null}
openssl rand -base64 32
```
Then open a shell on the machine and add the credentials to Hermes's `.env` on the volume, pasting the generated value as the secret:
```bash theme={null}
fly ssh console -a
cat >> /opt/data/.env <<'ENV'
HERMES_DASHBOARD_BASIC_AUTH_USERNAME=admin
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=
HERMES_DASHBOARD_BASIC_AUTH_SECRET=
ENV
```
Because they're on the volume, the credentials survive deploys and restarts.
In the same shell (or a new `fly ssh console --pty` session), start the dashboard, bound to the IPv6 wildcard address. Pass `::` without square brackets:
```bash theme={null}
hermes dashboard --no-open --host ::
```
In a second terminal, open a Fly proxy from your laptop:
```bash theme={null}
fly proxy 9119:9119 -a
```
Now visit `http://localhost:9119` in your browser and sign in with the username and password you set. Traffic goes over your authenticated WireGuard tunnel; the dashboard isn't published to the public internet, though it is reachable from other Machines on your organization's [private network](/networking/private-networking)
When you're done, `Ctrl+C` both commands. The gateway keeps running on the machine.
## Upgrading
The image is stateless; your data lives on the volume. To pull the latest Hermes:
```bash theme={null}
fly deploy -a
fly ssh console -a -C \
"ln -sf /opt/hermes/.venv/bin/hermes /usr/local/bin/hermes"
```
The deploy pulls `nousresearch/hermes-agent:latest` again. The second command re-creates the `/usr/local/bin/hermes` symlink; it lives on the container's filesystem, not the data volume, so each new container starts without it.
## VM sizing
If you're running heavy tool use or multiple concurrent sessions, scale up:
```bash theme={null}
fly scale memory 8192 -a
fly scale vm shared-cpu-4x -a
```
## Useful commands
| Command | Description |
| - | - |
| `fly logs -a ` | Stream live logs |
| `fly ssh console -a ` | SSH into the machine |
| `fly ssh console -a -C "hermes doctor"` | Health check |
| `fly machine restart -a ` | Restart after config changes |
| `fly status -a ` | Check machine status |
| `fly volumes list -a ` | List attached volumes |
| `fly ssh console -a -C "hermes skills list"` | List learned skills |
## Troubleshooting
**Gateway won't start**
Check `hermes doctor` for missing API keys or other diagnostics:
```bash theme={null}
fly ssh console -a -C "hermes doctor"
```
**Deploy times out with "timeout reached waiting for machine's state to change"**
If `fly logs` stops after "Successfully prepared image" and the Machine never starts, check that every secret in `machine_config.json`'s `secrets` array exists:
```bash theme={null}
fly secrets list -a
```
A missing secret stops the Machine before Hermes runs, so Hermes itself logs nothing.
**Out of memory**
Increase RAM:
```bash theme={null}
fly scale memory 8192 -a
```
**Need to start fresh**
Wipe the config files (skills, sessions, and memories survive):
```bash theme={null}
fly ssh console -a -C \
"sh -c 'rm -f /opt/data/config.yaml /opt/data/.env'"
fly machine restart -a
fly ssh console -a
hermes setup
```
To wipe everything including conversations, destroy and recreate the volume:
```bash theme={null}
fly machine stop -a
fly volumes destroy -a
fly volumes create data -a --region --size 3
fly machine start -a
```
**Skills behaving unexpectedly**
List, view, and delete:
```bash theme={null}
fly ssh console -a
hermes skills list
hermes skills view
hermes skills delete
```
# Guides and Blueprints
Source: https://docs.fly.io/blueprints/index
Architecture patterns and how-to guides for Fly.io: per-user Machines, AI agents, MCP servers, deployments, private networking, scaling, and background jobs.
A growing library of guides for running, designing, and deploying all kinds of apps on Fly.io. Whether you're just getting started or looking for new ideas, these are real-world patterns you can learn from and build on.
## Architecture Patterns
Guides for the structure your app on Fly.io. Layouts, tradeoffs, moving parts.
* [Deploy OpenClaw on Fly.io](/blueprints/deploy-openclaw)
* [Run Hermes Agent on Fly.io](/blueprints/hermes-agent-on-fly-io) NEW!!
* [Warm pools of user Machines](/blueprints/warm-pool-user-machines) NEW!!
* [Deploying Remote MCP Servers](/blueprints/remote-mcp-servers)
* [High availability: run multiple Machines](/blueprints/resilient-apps-multiple-machines)
* [N-tier architecture](/blueprints/n-tier-architecture)
* [Shared Nothing Architecture](/blueprints/shared-nothing)
* [Sticky sessions (session affinity)](/blueprints/sticky-sessions)
* [Multi-region databases with read replicas and fly-replay](/blueprints/multi-region-fly-replay)
## Deployment & Developer Workflow
Stuff you set up once and adjust when you ship code. Includes previews, base images, staging, and Docker wrangling.
* [Working with Docker on Fly.io](/blueprints/working-with-docker)
* [Custom deploy workflows: staggered and selective deploys](/blueprints/custom-deploy-workflows)
* [Zero-downtime deployments](/blueprints/seamless-deployments)
* [Roll back a deploy](/blueprints/rollback-guide)
* [Git Branch Preview Environments on Github](/blueprints/review-apps-guide)
* [Separate staging and production environments](/blueprints/staging-prod-isolation)
* [Per-User Dev Environments with Fly Machines](/blueprints/per-user-dev-environments)
* [Use Docker base images for faster deploys](/blueprints/using-base-images-for-faster-deployments)
* [Managing Docker Images with Fly.io's Private Registry](/blueprints/using-the-fly-docker-registry)
## Networking & Connectivity
Connecting things. Public apps, private services, bridging deployments, SSH, and WireGuard.
* [Run private apps with Flycast](/blueprints/private-applications-flycast)
* [Connect to your private network with WireGuard](/blueprints/connect-private-network-wireguard)
* [Connect AWS and other clouds with WireGuard](/blueprints/bridge-deployments-wireguard)
* [Route requests to per-user Fly Machines](/blueprints/connecting-to-user-machines)
* [Run an SSH server on a Fly Machine](/blueprints/opensshd)
## Scaling, Performance & Observability
Make it fast. Make it reliable. Monitor what's happening.
* [Stream logs from user apps with NATS](/blueprints/observability-for-user-apps)
* [Autoscale Fly Machines with autostop or fly-autoscaler](/blueprints/autoscale-machines)
* [Autostart and autostop private apps with Flycast](/blueprints/autostart-internal-apps)
* [Setting Hard and Soft Concurrency Limits](/blueprints/setting-concurrency-limits)
* [Using Fly Volume forks for faster startup times](/blueprints/volume-forking)
* [Run HIPAA-compliant healthcare apps](/blueprints/going-to-production-with-healthcare-apps)
## Background Jobs & Automation
How to run periodic tasks, long-running jobs, infrastructure automation, and the things that run when you’re asleep.
* [Long-running tasks and machine lifecycle](/blueprints/long-running-tasks) NEW!!
* [Building Infrastructure Automation without Terraform](/blueprints/infra-automation-without-terraform)
* [Background job queues with Celery, BullMQ, and Fly Machines](/blueprints/work-queues)
* [Task scheduling guide with Cron Manager and friends](/blueprints/task-scheduling)
* [Crontab with Supercronic](/blueprints/supercronic)
# Building Infrastructure Automation without Terraform
Source: https://docs.fly.io/blueprints/infra-automation-without-terraform
Fly.io has no Terraform provider. Get repeatable deploys with flyctl and GitHub Actions, or call the Machines API directly for custom orchestration.
## Overview
We don’t have a Terraform provider anymore, and that’s intentional. It’s not a great fit for how our platform works. This guide is about how to get a Terraform-like experience using the tools that do fit.
If you're used to defining infrastructure declaratively, using templated variables, repeatable workflows, a single source of truth, you can absolutely have that on Fly.io. But you’ll get there by thinking a little differently: using tools built around our actual primitives instead of trying to bend them to fit a plan/apply model. If you're looking for a good path forward, you've got options.
This guide walks through two options for automating Fly.io deployments without Terraform.
The first uses `flyctl` and GitHub Actions. It’s what we recommend for most users: push to `main`, your app gets deployed. No infra state to manage, no VMs to track. Just code and Git.
The second uses the Machines API directly. It’s more work, but gives you full control over each virtual machine. You’ll need this if you’re doing custom orchestration like rolling deploys across regions, temporary environments, or using fine-tuned scaling logic.
We’ll show you how both approaches work and help you figure out which one makes sense for your setup.
***
## The easy way: `flyctl` + GitHub Actions
This is the path most Fly.io apps take, and it’s what we recommend. You write code, you push to `main`, your app gets deployed. It’s quick to set up, and it abstracts away the VM lifecycle stuff unless and until you want it.
Here’s the gist:
1. Get your app running on Fly.io
2. Push the code to a GitHub repo
3. Wire up a GitHub Actions workflow that calls `flyctl deploy`
That’s it. Here’s what setting that up looks like.
### Step 1: Generate a deploy token
```
fly tokens create deploy --app
```
Treat that token like a password. Copy it somewhere safe—you’ll need it for GitHub.
### Step 2: Add it to your GitHub repo
In GitHub:
* Go to your repo > Settings > Secrets and variables > Actions
* Add a new secret named `FLY_API_TOKEN`
* Paste in the deploy token
### Step 3: Write the workflow file
Drop this into `.github/workflows/fly.yml` in your repo:
```
name: Fly Deploy
on:
push:
branches:
- main # or your default branch
jobs:
deploy:
name: Deploy app
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: superfly/flyctl-actions/setup-flyctl@master
- run: flyctl deploy --remote-only
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
```
Push that file, and your next `git push` to `main` will kick off a deployment. You can watch it happen in the Actions tab.
The `--remote-only` flag tells `flyctl` to build your image on Fly.io’s remote builders, so your CI environment doesn’t need Docker.
From here on out, you’ve got continuous deployment. Push to `main`, app goes live.
A realistic example: you’re running a Rails app backed by Postgres and Redis, deployed to `iad` and `lhr`. It lives in a monorepo, and you want each push to `main` to trigger a deployment. The `flyctl` + GitHub Actions setup gets you continuous deployment. You'll handle the supporting infrastructure imperatively (think databases, IP addresses, scaling): `fly pg create`, `fly redis create`, `fly scale count`, `fly ips allocate`, and so on. The platform keeps track of what you’ve configured, so you don’t need to re-declare it every time.
> [!NOTE]
>
> **Tip:** If you want a repeatable setup, you can script your commands into a bash file. Just be aware: that script will be linear and imperative, not declarative or idempotent like a Terraform plan.
***
## The harder way: The Machines API
This is the route for teams who were doing more than `terraform apply`. If you had logic layered into your infra automation like canary deploys, ephemeral environments, controlling exactly which VM runs where, you’ll want to look at the Machines API. It’s lower-level, but more powerful. Think of it as Fly.io’s bare-metal interface. No `plan`, no `state`, no `graph`, just HTTP.
You can use it to:
* Spin up new Machines in specific regions
* Roll out changes a Machine at a time
* Build custom scaling logic
* Tear down and rebuild environments on demand
This is what Fly.io’s own orchestration tools use under the hood. You can spin up Machines, wire them together, and orchestrate them across regions, just like we do. You've got all the same knobs and levers we use internally, and you can build exactly the kind of workflow your setup needs.
### What you're working with
The Machines API is a REST interface. It speaks JSON over HTTPS. You authenticate using a [Fly.io API token](/security/tokens), ideally a deploy token scoped to your app or organization. Pass it as a `Bearer` token in the `Authorization` header. If you've used GitHub Actions to deploy with `flyctl`, it's the same kind of token.
Each Fly App is a namespace. You can create as many Machines (VMs) as you want in an App. Each Machine runs a Docker image, has a region, some resource settings, and a lifecycle: create → start → stop → destroy.
You don’t need special tooling to use it. `curl` works. So does `fetch`, `requests`, `httpie`, or whatever HTTP client you like in Go or Rust or Python. Everything you might script with `flyctl`, like creating databases, setting scale counts, can also be done via the Machines API. It’s more verbose, but gives you full control and makes it easier to integrate infrastructure setup into a larger orchestration flow.
### Example: spin up a Machine with curl
```
export FLY_APP_NAME="your-app-name"
export FLY_API_TOKEN="your-fly-token"
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
"https://api.machines.dev/v1/apps/${FLY_APP_NAME}/machines" \
-d '{
"region": "sjc",
"config": {
"image": "nginx",
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 256
}
}
}'
```
The response includes the ID of the new Machine. You can then start, stop, or destroy it by hitting `/machines/{id}` with the appropriate verb.
### When to reach for the API
Most apps don’t need this. But if you’re building something dynamic, like spinning up short-lived browser-based dev environments or launching ephemeral multiplayer game servers, then you’ll want this kind of control.
Some of our favorite real-world uses:
* A “run this one function on a new machine” pattern, demoed in [this JS repo](https://github.com/fly-apps/fly-run-this-function-on-another-machine). It boots a new Machine in-region, runs a job, and tears it down. Works like serverless, but with a full VM and no cold starts.
* A minimal Go client for the Machines API, [`sosedoff/fly-machines`](https://github.com/sosedoff/fly-machines), has helpers to create, update, and destroy Machines programmatically. If you’re replacing Terraform logic with your own Go-based orchestration, this is a good lightweight example. For something more comprehensive, check out our official fly Go SDK, [`superfly/fly-go`](https://github.com/superfly/fly-go), for tighter integration with the platform.
The Machines API doesn’t lock you into a flow, we just give you the primitives.
**Still not sure? Here's the comparison.**
| Feature | `flyctl` + GitHub Actions | Machines API |
| - | - | - |
| Best for | Standard apps, CI/CD | Custom infra, dynamic environments |
| Setup time | Fast | Slower |
| Complexity | Low | High |
| Control | Abstracted (`deploy`, `scale`) | Full (`create`, `start`, `destroy`) |
| Official support | Fully supported & recommended | Fully supported & documented |
## One last thing
We know it sucks when a tool you relied on gets deprecated. Terraform made a lot of things feel clean and declarative. But when it comes to the Machines API specifically, Terraform was always working at a bit of a mismatch since it tries to express an inherently imperative lifecycle in a declarative model. The Machines API gives you fine-grained control and flexibility that didn’t map cleanly to Terraform's abstractions. With a little effort, you can build exactly what you need, and avoid being boxed in by someone else’s idea of what an app should look like.
If you're rebuilding that tooling now, let us know what you're working on. We might be able to help, or at least learn something from it.
## Related Reading
* [Continuous Deployment with flyctl and GitHub Actions](/launch/continuous-deployment-with-github-actions) Walk‑through on wiring CI/CD with Fly.io + GitHub Actions.
* [Access Tokens for Fly.io](/security/tokens) Deep dive into token scopes, permissions, and best practices for automation workflows.
* [Working with the Machines API](/machines/api/working-with-machines-api) Reference guide for automating at the VM/“machine” level (the lower‑level alternative to flyctl).
* [App Configuration (`fly.toml`)](/reference/configuration) Details on how your app config file works, which is often part of your automation setup.
# Long-running tasks and machine lifecycle
Source: https://docs.fly.io/blueprints/long-running-tasks
Why auto_stop_machines can stop a Machine that's still running background work, how Fly Proxy decides a Machine is idle, and two patterns that keep tasks alive.
This page covers what happens when your machine is busy doing work, but Fly thinks it's idle. Specifically: how `auto_stop_machines` decides what to stop, why a background task is invisible to that decision, and the two patterns that keep work from getting killed.
If you're picking a queue technology or a cron runner, start with the [work queues](/blueprints/work-queues) or [task scheduling](/blueprints/task-scheduling) guides instead. This page is about the machine behavior underneath them.
## The problem
A typical setup: a FastAPI endpoint accepts a request, spawns an async task to generate a report, returns `202 Accepted`, and closes the connection. The proxy sees no active connections. A few minutes later, it stops the machine. The report dies half-finished.
This isn't a bug. It's `auto_stop_machines` working exactly as documented. The proxy looks at inbound traffic. It does not look inside the container. From the proxy's point of view, a machine running a 20-minute job and a machine doing nothing look identical.
There are two ways to fix it. Pick one based on whether your work is bursty or steady, request-triggered or queue-driven.
## How autostop actually decides
The Fly proxy evaluates machines every few minutes. The exact rule depends on how many machines you have:
**Multiple machines.** The proxy uses your `soft_limit` [concurrency setting](/blueprints/setting-concurrency-limits) to compute excess capacity:
```
excess = num_machines − (num_machines_over_soft_limit + 1)
```
If `excess ≥ 1`, the proxy stops one machine. The `+ 1` keeps a buffer of one idle machine for incoming traffic.
**Single machine.** Simpler: if load is zero, the proxy stops the machine.
In both cases, "load" means traffic the proxy can see. Background work running inside the machine, whether that's async workers, cron-style loops, or anything else not driven by an inbound request, doesn't count. There's also no way for your application to tell the proxy, "I'm busy, leave me alone."
This is the central fact for the rest of the guide. Everything below is a way to work around it.
### Stop vs. suspend
`auto_stop_machines` takes three values: `"off"`, `"stop"`, and `"suspend"`.
* **stop** shuts the machine down cold. A restart takes seconds (about 2s for a Rails app, less for a small binary).
* **suspend** dumps the entire VM state (memory, CPU, network) to disk. Resume takes a few hundred milliseconds.
**Stop** is the simpler default: the machine shuts down when it's idle and cold-starts when it's needed again. For most apps, that's the right tradeoff.
**Suspend** is the right choice when cold start is too painful (slow framework boot, heavy initialization, large in-memory state) and you'd still like to idle when inactive. The tradeoff: suspend is rougher on the underlying platform and has more constraints:
* Machines must have 4 GB of RAM or less.
* Swap and schedules are not supported.
* Machines updated before June 20, 2024 cannot be suspended.
* Suspend is not durable. Fly does not guarantee that a suspended machine will resume. Host migration, maintenance, or capacity pressure can turn what would have been a resume into a cold start. Treat suspend as a faster version of stop, not a guaranteed warm restart.
* A few log lines may be lost across a suspend/resume cycle, and the system clock can take a second or two to re-synchronize after resume. See "[Suspend vs. Stop](/getting-started/troubleshooting#suspend-vs-stop)" for details on clock skew.
Billing is the same for both: you pay for stopped machines like you pay for suspended ones.
For the rest of this guide, "stop" and "suspend" are interchangeable. The patterns work the same way for both.
## Pattern A: disable autostop, manage shutdown in the app
**Use this when** your app has long-lived workers, in-process job runners, or any background work that the application itself can track.
Turn autostop off in `fly.toml`:
```toml theme={null}
[http_service]
internal_port = 8080
auto_stop_machines = "off"
auto_start_machines = true
```
With autostop off, the proxy never stops your machines for being idle. They stay up until something else stops them (a deploy, `fly machine stop`, or a host migration). You're paying for every machine 24/7, in every region you've scaled into, so make sure that's the right tradeoff before adopting this pattern.
When deploys, manual stops, or host migrations *do* stop the machine, your app gets `SIGTERM` and has `kill_timeout` seconds to clean up. The default of 5 seconds is almost certainly too short. Bump it. These are top-level keys in `fly.toml`:
```toml theme={null}
kill_signal = "SIGTERM"
kill_timeout = "30s"
```
The maximum is 300 seconds. `kill_timeout` is a drain window, not a "let the job finish" knob. If your jobs run longer than 5 minutes, either checkpoint them so they can resume, or stop accepting new work and let in-flight jobs drain before the timeout. Don't wait for everything to finish.
A minimal shutdown pattern in Node:
```javascript theme={null}
let activeJobs = 0
let shuttingDown = false
async function runJob(payload) {
if (shuttingDown) throw new Error("shutting down")
activeJobs++
try {
await doWork(payload)
} finally {
activeJobs--
}
}
process.on("SIGTERM", () => {
shuttingDown = true
const start = Date.now()
const deadline = 25_000 // 5s under kill_timeout
const tick = setInterval(() => {
if (activeJobs === 0 || Date.now() - start > deadline) {
clearInterval(tick)
process.exit(0)
}
}, 200)
})
```
In Python with asyncio:
```python theme={null}
import asyncio, signal
active = 0
shutting_down = asyncio.Event()
async def run_job(payload):
global active
if shutting_down.is_set():
raise RuntimeError("shutting down")
active += 1
try:
await do_work(payload)
finally:
active -= 1
async def shutdown():
shutting_down.set()
try:
await asyncio.wait_for(_drain(), timeout=25)
except asyncio.TimeoutError:
pass
async def _drain():
while active > 0:
await asyncio.sleep(0.2)
loop = asyncio.get_event_loop()
loop.add_signal_handler(signal.SIGTERM, lambda: asyncio.create_task(shutdown()))
```
Both patterns refuse new work as soon as `SIGTERM` arrives, then wait for in-flight jobs up to a deadline a few seconds shorter than `kill_timeout`. The safety margin matters, if you wait the full 30s, Fly's `SIGKILL` arrives before your `exit(0)` runs.
## Pattern B: split web and worker into separate process groups
**Use this when** web traffic is bursty (a good candidate for autostop) but background work is steady or long-running (a bad candidate for autostop).
Split with `processes` in `fly.toml`:
```toml theme={null}
[processes]
web = "bundle exec puma"
worker = "bundle exec sidekiq"
[http_service]
internal_port = 8080
auto_stop_machines = "suspend"
auto_start_machines = true
processes = ["web"]
```
The worker process group has no `[http_service]` attached, so the proxy never touches its machines. Autostop applies only to the web tier.
Scale them independently:
```bash theme={null}
fly scale count web=2 worker=1
```
This is the pattern Sidekiq, Celery, and BullMQ workers actually want. The web tier scales to zero off-hours; the worker tier runs whenever there's work in the queue.
Tradeoff: you're paying for at least one worker machine continuously. If your work is batchy enough that on-demand workers make sense, use the [work queues guide's on-demand worker pattern](/blueprints/work-queues) instead, as that spins up a fresh machine per job and lets it stop when done.
## Graceful shutdown: what Fly sends
When something stops your machine, whether that's `auto_stop_machines`, `fly machine stop`, a deploy, or a host migration, Fly sends `kill_signal` (default: `SIGTERM`) to PID 1. After waiting `kill_timeout` seconds, it sends `SIGKILL`.
The defaults are conservative:
| Option | Default | Max | Notes |
| - | - | - | - |
| `kill_signal` | `SIGTERM` | n/a | Also accepts `SIGQUIT`, `SIGUSR1`, `SIGUSR2`, `SIGKILL`, `SIGSTOP` |
| `kill_timeout` | `5s` | `300s` | The drain window before `SIGKILL` |
Five seconds is enough for an HTTP server to close keepalives. It is not enough for a long-running job to finish. If you have any background work, set `kill_timeout` to a value that allows your typical job to complete. You'll need to determine this on your side. Both keys are top-level in `fly.toml`:
```toml theme={null}
kill_signal = "SIGTERM"
kill_timeout = "30s"
```
PID 1 receives the signal. In a Docker container running your app directly, that's your process. In a container running a shell wrapper (`CMD ["sh", "-c", "..."]`), the shell is PID 1 and `SIGTERM` doesn't propagate. Use the exec form: `CMD ["myapp"]`, or `exec myapp` inside the wrapper.
`kill_timeout` is not a "finish your work" timer. It's a drain window. Inside it, you should:
1. Stop accepting new work
2. Let in-flight work finish, or checkpoint it
3. Exit cleanly
If your jobs take longer than 5 minutes, you can't drain them inside `kill_timeout`. You need either Pattern A with checkpoint/resume, or Pattern B with a worker tier that's never autostopped.
Run `fly config validate --strict` before relying on any of this. By default, `fly config validate` silently accepts unrecognized sections and keys. A typo or outdated section name can pass validation and then do nothing at runtime. Strict mode catches those errors.
## Picking a pattern
| Situation | Pattern |
| - | - |
| Jobs are short (\< 30 seconds) | Increase `kill_timeout`; everything else can stay as default |
| Long-running jobs, steady web traffic | A: disable autostop, in-app drain |
| Long-running jobs, bursty web traffic | B: split web/worker processes |
| Cron-style scheduled jobs | See [task scheduling](/blueprints/task-scheduling) |
| Queue-driven workers | B: combine with [work queues](/blueprints/work-queues) |
| One-off jobs (fire and forget per request) | On-demand workers; see [work queues](/blueprints/work-queues) |
| Can't restructure right now | A: accept the continuous machine cost |
## Common problems
**My `SIGTERM` handler runs but the job still gets killed.** `kill_timeout` is shorter than your handler needs. Bump it (max 300s) and set your handler's deadline a few seconds under that.
**The machine stops mid-job even with `auto_stop_machines = "off"`.** Autostop is only one of several things that stop machines. Deploys, `fly machine stop`, scale-down, and host migrations all do too. Check `fly logs` for the `instance refused` or `host migration` events. Pattern A still applies. The only difference is that autostop is no longer the trigger.
**Why doesn't a self-ping keep my machine alive?** It won't. The [autostop reference](/reference/fly-proxy-autostop-autostart) defines idle as "a load of 0" but doesn't specify what counts as load. Empirically, sending a successful HTTP request every 60 seconds from a machine to its own `.fly.dev` hostname does not prevent autostop. The proxy still stops the machine after 5 to 10 minutes. To keep a machine running through idle traffic, turn off autostop (Pattern A) or move the work into a process group without `[http_service]` (Pattern B).
**Worker machines won't stop when I deploy.** A process group with no `[http_service]`, such as the worker tier in Pattern B, is invisible to the proxy. Deploys still update those machines because `flyctl` talks to them directly, but the proxy does not manage their lifecycle and cannot autostop them. To stop them gracefully, send a signal with `fly machine stop` or let `fly deploy` replace them during a deployment.
**Suspend resumes are slower than the docs say.** Suspend isn't durable. If Fly can't restore the snapshot (host migration, capacity pressure), you get a cold start. There's no flag to tell you which happened; check the first-request latency. If cold starts matter, run with `min_machines_running = 1`.
## Where to go next
* [Work queues guide](/blueprints/work-queues): Picking a queue technology
* [Task scheduling guide](/blueprints/task-scheduling): cron-style triggers and scheduled machines
* [Autostart and autostop reference](/reference/fly-proxy-autostop-autostart): The proxy's full decision logic
* [Configuration reference](/reference/configuration): `kill_signal`, `kill_timeout`, `processes`, `auto_stop_machines`
* [Machine states](/machines/machine-states): what `stopping`, `stopped`, and `suspended` actually mean
# Multi-region databases with read replicas and fly-replay
Source: https://docs.fly.io/blueprints/multi-region-fly-replay
Run your app with a primary database and regional read replicas on Fly.io, and use the fly-replay header to send write requests to the primary region.
## Overview
You want to run your app and database in multiple [regions](/reference/regions) close to your users, and deploying database read replicas will give you better performance on read requests. This is a good solution for apps with read-heavy database workloads. But you also want writes to be efficient, despite needing to write to a primary database on the other side of the world.
This guide will help you understand the concepts to implement an app, with a primary database and read replicas in multiple regions, that uses the `fly-replay` response header to re-route write requests to the primary region. Consider using this guide when:
* Your app's users are in more than one geographic region and you want low latency for reads in all regions.
* Your app has a request distribution of less than 50% writes.
* You don't want the architecture complications and potential for conflicts that come with running a multi-primary database cluster.
## Fly.io was made for multi-region apps
Fly.io already routes requests to the closest region where your app is deployed. Using regional database read replicas can further reduce latency since the Fly Machine (our fast-launching VM) receiving the request can connect to a database instance in the same region. Read replicas also reduce the overall load on the primary database. This multi-reader single-writer setup is great for read-heavy apps.
When your app receives write requests, you can use the `fly-replay` response header to forward the whole HTTP request to the primary region, which is often faster than connecting directly to the primary database from other regions. This is especially true when a request to an endpoint requires multiple database queries to complete; there would be extra latency on each query from a secondary region compared to a local connection in the primary region.
## How it works
The [`fly-replay` response header](/networking/dynamic-request-routing) instructs Fly proxy to redeliver (replay) the original request to another region or Machine in your app, or even another app in your organization. In this case, you’ll be replaying write requests to the Machine in the primary region. Using `fly-replay` to replay write requests is a general pattern that can be applied in most languages and frameworks for databases with one primary and multiple read replicas.
In the following diagram, the app is running one Machine in each of three regions. The primary region is Chicago, and this is where the read/write primary database resides. There are Machines in two other regions, Rio de Janeiro and Amsterdam, each of which has a read replica. This example uses three regions for simplicity, but you could deploy in more than three regions and have more than one Machine per region connecting to the same read replica.
> [!NOTE]
>
> **Note:** To illustrate the `fly-replay` concept in our diagram, we show the replayed HTTP request going directly from Machines in Rio de Janeiro and Amsterdam to the Machine in Chicago. In real life, Fly Proxy routes the request back through an edge node first. The cost of this routing is small, but if extreme efficiency is important for your use case, you can run your app in more regions to mitigate that.
Your app is running on Fly.io. The database can also run on Fly.io—in which case the app and database regions will match—or on another provider where you can pick the region closest to the Fly.io region of your Machines.
## How to make it work for your app
Your app needs logic to:
1. **Read from the corresponding regional replica (or primary!) when a Machine receives a read request.**
2. **Always replay write requests to the primary region rather than connecting to the primary database from another region.**
### Read from the corresponding regional replica
Your app has a configurable primary region and you'll also host your primary database there or, in the case of a separate database provider, as close as possible to it. The primary region is exposed in the `PRIMARY_REGION` environment variable on every Machine in your app.
Each Machine exposes the region in which it's running in the `FLY_REGION` environment variable.
Your app can check the `FLY_REGION` against the `PRIMARY_REGION`, and modify the `DATABASE_URL`, `DB_HOST`, or other database variables when the region codes don't match. In some cases this just means changing a port ([as for Postgres](/unmanaged-postgres/advanced-guides/high-availability-and-global-replication#connecting-to-read-replicas)) and in other cases it will be more complex if the database requires different variables for each read replica (like in this [Laravel example](https://fly.io/laravel-bytes/multi-region-laravel-with-planetscale/)).
### Replay write requests to the primary region
Your app can detect write requests and send a response with the `fly-replay` header that tells Fly Proxy to replay the whole request to the Fly Machine in the primary region.
#### Detect write requests
First, you'll need a way to detect write requests so that you can forward them to the primary database. How you detect write requests will depend on your app and database. With Postgres databases, probably the simplest way to detect write requests is by [catching “read only transaction” errors](/unmanaged-postgres/advanced-guides/high-availability-and-global-replication#detecting-write-requests) when you attempt to write to a read replica.
You can also set up routes in middleware if your language provides support for splitting read and write connections. Or your app can handle GET, POST, DELETE, PUT, and PATCH requests differently, again likely in your app's middleware. For example, assuming HTTP requests using GET are read requests and everything else is a write request will work well most of the time.
#### Send a response with the fly-replay header
When your app detects a write request, it should send a response that contains only the `fly-replay` header with the [region code](/reference/regions) of your primary region like so: `fly-replay: region=`. Fly Proxy will replay the whole request to the specified region. The code would look something like this:
```yaml theme={null}
# This code should be in the handler for non-GET requests to the endpoint you
# want to send to the primary region for write to the primary database.
if os.environ("PRIMARY_REGION") != os.environ("FLY_REGION"):
# This machine is not in the primary region
response.headers = {"fly-replay": f"region={os.environ("PRIMARY_REGION")}"}
return response
else:
# This machine *is* in the primary region
# Handle request normally, write to database.
```
## Challenges and limitations
### Write-heavy apps shouldn't use read replicas
Don't use regional read replicas if your app's requests are more than 50% write requests. In that case, you'd want to look into a multi-region primary or similar solution.
### Reading your own writes
There will be short periods of time when data is not consistent between replicas and the primary. If reading your own writes immediately is a requirement for your app, then you'll need to consider how to handle replication.
You can set up health checks to monitor replication status and fail when replication is not complete.
Another useful pattern is to poll the database until the data is replicated. The Fly Postgres [LSN module](https://github.com/superfly/fly_postgres_elixir/tree/main/lib/lsn) has an example of how to poll with a [Postgres stored procedure](https://github.com/superfly/fly_postgres_elixir/blob/main/lib/migrations/v01.ex). You can reuse this pattern from any database library.
A less efficient method would be for your app to catch the errors thrown when records don't exist and then retry failed requests.
## Implementation resources
Example implementations of this multi-region database with `fly-replay` pattern can give you a head start:
* [Fly Ruby gem](https://github.com/superfly/fly-ruby) for running database replicas alongside your app instances in multiple regions.
* [Fly Postgres Elixir library](https://github.com/superfly/fly_postgres_elixir) for geographically distributed Elixir applications using Ecto and PostgreSQL in a primary/replica configuration on Fly.io.
## Related reading
We've covered multi-region databases with `fly-replay` in some past blog posts and in our docs:
* [Globally Distributed Postgres](https://fly.io/blog/globally-distributed-postgres/)
* [Multi-Region Laravel with PlanetScale](https://fly.io/laravel-bytes/multi-region-laravel-with-planetscale/)
* [Run Ordinary Rails Apps Globally](https://fly.io/ruby-dispatch/run-ordinary-rails-apps-globally/)
* [Dynamic Request Routing](/networking/dynamic-request-routing)
# N-tier architecture
Source: https://docs.fly.io/blueprints/n-tier-architecture
What n-tier architecture is, how presentation, web, and data tiers map to Fly Machines, private networking, and Postgres, and how to deploy a multi-tier app.
## Introduction: What is n-tier architecture?
When people talk about “**n-tier architecture**,” they’re describing a way of splitting an app into layers (or “tiers”) that each have a specific job. The “**n**” just means there could be two, three, or more tiers depending on how you slice things.
***
## Defining tiers
* **Presentation tier**: the UI/frontend of your app — HTML/CSS/JS in a browser, or a mobile app interface — that the user interacts with
* **Web tier**: the app servers/backend that [handle requests](/networking/dynamic-request-routing) and run business logic
* **Data tier**: stores, manages and provides access to application data
***
## N-Tier Examples
### Two tiers
**web app + database**. This is the most common case on Fly.io: your app servers run on Fly Machines and connect to a shared Postgres cluster.
### Three tiers
**frontend + backend + database**. You’ll see this if you have a separate frontend (like a React SPA served from a CDN, or a mobile app) that talks to your Fly backend.
### Mapping to Fly.io
* **Web tier** → your app running on Fly Machines
* **Data tier** → [Fly Managed Postgres](/postgres) (our hosted Postgres that handles the messy parts for you)
* **Optional presentation tier** → a separate frontend app (web or mobile) that talks to your Fly backend
The idea is simple but powerful: the web tier doesn’t keep state. All shared state lives in the data tier, so you can add or remove servers at will without breaking anything. It’s the architecture you want if you care about **scalability** (handling more traffic by adding machines) and **reliability** (requests keep flowing even if one machine goes down).
***
## Why use n-tier on Fly.io?
* **Scale with one command**: Add more Web/App machines whenever traffic spikes.
* **Keep requests flowing**: Fly’s proxy automatically balances load across your Machines.
* **Let the database do its job**: With [Managed Postgres](/postgres), you don’t need to worry about setup, backups, or failover.
You could use Redis, Tigris, or other data stores here too, but if you’re not sure, start with Managed Postgres. It’s the default choice for good reason.
***
## Quickstart: Deploying a n-Tier App
Let’s spin up a simple n-tier setup.
### 1. Launch your app
First, deploy your app to Fly:
```bash theme={null}
fly launch
fly deploy
```
This gives you the **web tier**.
***
### 2. Add a database (Managed Postgres)
Now create your **data tier** with [Fly Managed Postgres](/postgres):
```bash theme={null}
fly mpg create
```
This will:
* Provision a new Postgres cluster in the same region as your app to keep latency to a minimum (though you can choose another region).
* Configure monitoring, daily backups, and automatic failover.
To connect your app, you’ll need to add the database credentials as secrets. You can do this from the Fly.io Dashboard, or with `flyctl`:
```bash theme={null}
fly secrets set DATABASE_URL=postgres://:@/
```
Most frameworks (Rails, Django, Node, etc.) will automatically read `DATABASE_URL` from the environment. Once set, your app is ready to talk to Postgres.
More details on creating and connecting a managed Postgres database can be found [here](/postgres/create-and-connect).
***
### 3. Scale out your web tier
If traffic grows, add more Machines:
```bash theme={null}
fly scale count 3
```
[Fly’s proxy](/reference/fly-proxy) takes care of spreading requests across the machines (load balancing).
**Note:** Scaling works best when your web tier is **stateless**. Any shared data like file uploads, user sessions/cookies, or cached state should live in your data tier (usually Managed Postgres, sometimes object storage or Redis). If your framework defaults to local disk or memory for these, configure it to use a shared store instead.
***
## Finishing up
At this point you’ve got:
* **Optional presentation tier**: a separate frontend app, if you use one (delivered to browsers or devices)
* **Web tier**: Fly Machines running your app (stateless and load balanced)
* **Data tier**: [Fly Managed Postgres](/postgres)
Every request can hit any Machine, and all Machines share the same Postgres. That’s an n-tier architecture in action, and it’s the foundation for most reliable web apps.
***
## Related reading
* Explore the features of [Fly Managed Postgres](/postgres)
* Find out how to [scale apps on Fly](/launch/scale-count)
* Read about [Fly networking](/networking)
* Learn more about [n-tier architecture](https://en.wikipedia.org/wiki/Multitier_architecture#Three-tier_architecture)
# Stream logs from user apps with NATS
Source: https://docs.fly.io/blueprints/observability-for-user-apps
Monitor apps your users run on Fly.io: deploy the fly-telemetry forwarder, subscribe to the NATS log stream, and route each user app's logs back to that user.
## Overview
When you run a platform for users to execute their own code on Fly.io—[think per-user development environments](/blueprints/per-user-dev-environments) or AI/LLM apps—it's a good idea to have real-time observability for those user apps. You might *also* want to stream logs from those apps back to the original developer (your end-user), so they can see what their code is doing.
Fly.io offers built-in telemetry based on [**NATS**](https://nats.io/) that you can tap into to achieve this. At a high level, Fly.io captures everything your app writes to stdout and funnels it into an internal NATS log stream. A proxy in front of this NATS cluster ensures that each organization can only access its own applications' logs. Both the `flyctl logs` command and the our web dashboard's "Live Logs" use this pipeline under the hood. We can use the same system! By connecting a NATS client within your Fly organization's [private network](/networking/private-networking), we can subscribe to log subjects and stream logs out to wherever we need, in real-time.
## Deploying the Fly Telemetry Forwarder
If you want a bird's‑eye view for your own operations team, Fly provides an open source reference app called [**fly-telemetry**](https://github.com/superfly/fly-telemetry) that you can deploy to quickly get an observability stack running in your Fly organization. This app acts as a telemetry forwarder: it subscribes to the platform's NATS logs (and metrics) streams and stores the data for you. It includes a lightweight time-series database (VictoriaMetrics for metrics, VictoriaLogs for logs) and a Grafana instance with pre-configured dashboards, so you can explore your apps' logs and metrics visually.
In an empty directory, run:
```sh theme={null}
fly launch --from https://github.com/superfly/fly-telemetry \
--yes \
--copy-config \
--org "$ORG" \
--env ORG="$ORG" \
--secret ACCESS_TOKEN="$(fly tokens create readonly "$ORG")" \
--flycast
```
After the deployment finishes, browse to the app's private URL [through WireGuard](/blueprints/connect-private-network-wireguard) or `fly proxy` and you have instant Grafana dashboards.
This is handy for SREs, but not necessary for your end‑users.
***
## Streaming Fly app logs to your end users
Streaming live logs straight to your users gives them insight into what their code is doing—developers see exactly what the runtime sees, in real time, without leaving your product or juggling extra tooling.
### Connecting to the NATS Log Stream
Every Fly organization has access to **NATS** which carries the logs for its apps. The server is available at the well-known address `[fdaa::3]:4223`. To connect to this endpoint, you must do so from within the Fly network—for example, from code running in a Fly app (like your telemetry forwarder or router) or from your local machine over a WireGuard connection.
Using the NATS JavaScript client, you could connect as follows:
```javascript theme={null}
import { connect, StringCodec } from "nats";
const nc = await connect({
servers: "[fdaa::3]:4223",
user: "sandwich-expert", // your Fly org slug (example)
pass: process.env.ACCESS_TOKEN // your Fly access token (read-only)
});
// Now nc is a NATS connection that can subscribe to log subjects.
```
### Authentication
When connecting from a Fly app in your org, you'll need to supply an access token. To generate one manually, run `fly tokens create readonly ` and store it as an app secrets. The NATS proxy will accept the token as the password and your org name as the username.
### Log Subjects and Filtering
Logs in the NATS stream are organized by **subject**. The subject format for Fly logs is:
```sh theme={null}
logs...
```
For example, if you have an application named `panini-786` and it's running in region `dfw`, you might see subjects like `logs.panini-786.dfw.d896d7c609dd68` (where `d896d7c609dd68` is the Machine ID). You can subscribe at varying levels of granularity using NATS wildcards:
| Subject Pattern | Description |
| - | - |
| `logs.>` | Subscribe to **all logs** from all apps in your org (every region, every Machine) |
| `logs..*.>` | Subscribe to all logs from a specific app (across all regions and Machine) |
| `logs.*..>` | Subscribe to all logs from a specific region |
| `logs...` | Subscribe to a single Machine (full exact subject) |
In practice, to stream all logs for a given user's app, you'll subscribe to the subject pattern `logs..>` – this will capture any log line from any Machine of that app, in any region.
### A Real-World Example
[**NATstream**](https://natstream.fly.dev/) is a reference app that streams logs from your Fly apps right back to the browser. It connects to NATS and turns those events into a live feed at a simple HTTP `/logs` endpoint using Server-Sent Events (SSE). Developers could log in to your product and watch their app's logs flow in real time, without needing separate logging tools or the Fly CLI. The app keeps things high-level and lightweight—there's no deep framework complexity here, just a straightforward example of hooking into Fly's log pipeline and broadcasting events to the browser.
You can fork it as a foundation for your own developer-facing live log streaming service, or rip out the relevant parts and build this logic into your existing app, using the NATstream implementation as a guide. More robustly, you could run a separate log "router" app that handles authentications and streaming on a per user, per app basis. This would be especially useful if streaming logs to a CLI.
***
## Building a Central Log Router Service
Create a Fly "log router" app that sits between Fly's NATS telemetry stream, your product and your developers. Because each customer already has their own Fly app, every user's logs appear on a unique subject (`logs..>`), so multi-tenancy is solved by design—subscribe only to the subject that matches the requesting developer's app and you'll never mix data between users.
> [!NOTE]
>
> **Note**: Your router service should enforce that a user can only request the logs for the app(s) they own. This means your routing logic needs an access control check (e.g. based on the authenticated user's ID matching the app name or an internal mapping). Fortunately, if you've structured your platform as one Fly app per user, it's straightforward to maintain a mapping from user account to Fly app name and use that for authorization.
On startup the router opens one NATS connection with an org-scoped read-only token. When a developer signs in through your dashboard or CLI, the router authenticates that user, determines their Fly app name, and, if not already listening, subscribes to that app's subject. As log entries arrive, the router forwards them over a live channel—typically a websocket or Server-Sent Events stream—directly to the developer's browser or CLI.
The router itself is a Fly app that does the following:
* On startup, connect to the NATS log stream using the org-wide credentials (as shown earlier).
* Wait for developers to request or connect for log streaming (e.g., a developer opens a web UI to view their app's live logs, or runs a command to view logs).
* When a request for a specific app's logs is received, subscribe to that app's log subject (if not already subscribed).
* Stream the incoming log messages to the requesting developer in real-time.
* Ensure that each developer only receives their own app's logs. The router should use the mapping of app ← → user to publish the right data to the correct output channel (and not mix different users' data).
This approach uses one NATS subscription *per user app*. The messages are forwarded verbatim to the user's websocket in this case (as JSON strings), but you could transform them (for example, format them nicely or filter out certain internal logs) before sending. On the client side, you would simply read from the websocket or SSE stream and append new log lines to a console view.
## Related reading
* [Logging overview](/monitoring/logging-overview) How logs flow from your Fly apps into the platform’s NATS pipeline.
* [Logs API options](/monitoring/logs-api-options) Guide with approaches for tapping directly into structured logs programmatically.
* [fly-telemetry](https://github.com/superfly/fly-telemetry) Lightweight reference implementation for quick, out-of-the-box observability for deployments on Fly.io.
# Run an SSH server on a Fly Machine
Source: https://docs.fly.io/blueprints/opensshd
Install and run OpenSSH in your Fly.io app so rsync, scp, sshfs, VS Code Remote, Ansible, or Kamal can connect over SSH, with key setup and port 22 mapping.
## Overview
A number of tools allow you to interact with your server over SSH. These tools are useful for tasks such as
copying files ([rsync](https://rsync.samba.org/), [scp](https://en.wikipedia.org/wiki/Secure_copy_protocol), [sshfs](https://github.com/libfuse/sshfs#sshfs)),
editing ([emacs](https://www.gnu.org/software/emacs/manual/html_node/emacs/Remote-Files.html), [vim](https://www.vim.org/scripts/script.php?script_id=1075), [vscode](https://code.visualstudio.com/docs/remote/ssh)), and
deployment ([ansible](https://www.ansible.com/), [github actions](https://github.com/marketplace/actions/ssh-deploy), and [kamal](https://kamal-deploy.org/)).
One way to use these tools is to [set up a wireguard VPN](/networking/private-networking#1-install-your-wireguard-app) and
[issue a new SSH credential](/flyctl/cmd/fly_ssh_issue). This may be impractical for some use cases (example: github actions).
As an alternative, you can configure and deploy an SSH server on your machine(s). This guide will walk you through the process.
Before proceeding, a **caution**: unless you are **certain** that all of the clients will access this service through IPv6, you will need a [dedicated IPv4](/networking/services#dedicated-ipv4) address.
## Install and configure opensshd
Most Docker images are ultimately based on Debian, so the following will work. Adjust as necessary for other operating systems (example: [Alpine](https://www.alpinelinux.org/)).
```Dockerfile theme={null}
RUN apt-get update \
&& apt-get install -y openssh-server \
&& cp /etc/ssh/sshd_config /etc/ssh/sshd_config-original \
&& sed -i 's/^#\s*Port.*/Port 2222/' /etc/ssh/sshd_config \
&& sed -i 's/^#\s*PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config \
&& mkdir -p /root/.ssh \
&& chmod 700 /root/.ssh \
&& mkdir /var/run/sshd \
&& chmod 755 /var/run/sshd \
&& rm -rf /var/lib/apt/lists /var/cache/apt/archives
```
Notes:
* This runs `sshd` internally on port 2222. This is because by default `sshd` listens on all network interfaces,
but fly.io machines already are listening on port 22 on the private network (ipv6) interface, so sshd will
either complain or fail to start. Additionally, `fly deploy` will attempt to verify that your application is
listening on the interfaces your application exports, and will incorrectly treat fly's listening on port 22 as
evidence that your application is up and running.
* Password Authentication is disabled. This avoids unnecessary prompts. We will be using SSH keys instead.
* The above assumes `root` user. If your application is running under a different user, replace `/root` with the
home directory of that user (example: `/home/rails`).
## Map internal port 2222 to external port 22
Add the following to `fly.toml`:
```
[[services]]
internal_port = 2222
protocol = "tcp"
auto_stop_machines = true
auto_start_machines = true
[[services.ports]]
port = 22
```
This section is needed even if the internal and external ports are the same.
Notes:
* `internal_port` needs to match the port you selected in the previous step.
* `port` can be any available port. `22` is the [default port](https://www.ssh.com/academy/ssh/port) for SSH,
and the one that most applications expect to be used.
* Like with your web server port, your server can be configured to spin down when idle and restart when accessed.
Feel free to adjust `auto_stop_machines` and `auto_start_machines` if your needs differ.
## Start the openssh server
There are a [number of ways to run multiple processes](/app-guides/multiple-processes). The most straightforward
way to start sshd before your application. Locate the `ENTRYPOINT` in your Dockerfile. If you don't have
one, create a script in your application directory, name that script as the ENTYRPOINT, and make it
executable.
An example of such a script:
```bash theme={null}
#!/bin/bash -e
/usr/sbin/sshd
exec "$@"
```
Note: the above needs to be run as root. If your Dockerfile specifies another `USER`, you can work around this
by installing and configuring `sudo` and then removing sudo access before running your main process.
For example, if your userid is `rails`, you would add the following to your Dockerfile:
```Dockerfile theme={null}
RUN apt-get install -y sudo && \
echo "%rails ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
```
And then update your entrypoint script:
```bash theme={null}
sudo /usr/sbin/sshd
sudo sed -i "/^%rails/d" /etc/sudoers
```
The above commands will start `sshd` as root, then remove sudo access from your unprivileged userid.
## Upload your SSH key
First, locate your SSH key. You can create a new key using [`ssh-keygen`](https://www.ssh.com/academy/ssh/keygen),
Or you can use an existing one: look inside the `.ssh` folder in your home directory for a file with a name like `id_rsa.pub`.
Once located, there are multiple ways to proceed. Following you will find two ways. Depending on the framework your
application uses, you may have other ways to store credentials or environment variables. As this step only deals with
public keys, one need not take extraordinary measures to prevent leakage.
### Alternative 1: Set and use a secret
```bash theme={null}
fly secrets set "AUTHORIZED_KEYS=$(cat ~/.ssh/id_rsa.pub)"
```
Powershell users will want to use the following instead:
```powershell theme={null}
fly secrets set "AUTHORIZED_KEYS=$(Get-Content $HOME\.ssh\id_rsa.pub)"
```
Next, update your entrypoint script to contain the following:
```bash theme={null}
echo $AUTHORIZED_KEYS > /root/.ssh/authorized_keys
```
If you are running with sudo, you would want to do the following instead:
```bash theme={null}
echo $AUTHORIZED_KEYS | sudo tee /root/.ssh/authorized_keys > /dev/null
```
### Alternative 2: Copy from a volume
If you have a volume and if you can arrange to upload the file
there, you can directly copy it as a part of your entrypoint script.
```bash theme={null}
cp /volume/.ssh/authorized_keys /root/.ssh
```
## \[RECOMMENDED] Make SSH host keys stable
The first time you SSH into a server you will be presented with a fingerprint for the server you are accessing.
If you accept that fingerprint, it will be added to your `known_hosts` file in your `.ssh` directory. This key
is generated when you install `openssh-server`.
The issue arises when you redeploy your application. If something changes (or your docker cache expires),
the installation of `openssh-server` may be rerun, and new keys will be generated. To avoid these keys from
being used, you can capture and restore the keys.
The following assumes that you have a [volume](/launch/volume-storage) mounted at `/volume`, and an entrypoint
script.
```bash theme={null}
mkdir -p /volume/.ssh
if [[ "$(ls /volume/.ssh/*_key)" = "" ]]; then
cp /etc/ssh/*_key /volume/.ssh
else
cp /volume/.ssh/*_key /etc/ssh
fi
```
Notes:
* These keys are expected to be private, so if you go with an alternate route, make sure that these values are
not committed to a public repository unencrypted.
* If you are running with `sudo`, you need to add `sudo` before the `mkdir`, the `ls` and both of the `cp` statements.
* If you have multiple machines, you may want all of them to [share the same keys](https://security.stackexchange.com/a/89621).
## \[OPTIONAL] Configure client user and aliases
Your full dnsname may be a mouthful, the user the application runs under may be different than the one you use on your laptop.
the port you expose may be non-standard, or you may have multiple machines and a desire to be able to SSH into a specific one.
If any of these apply to you, you can create or update a file named [`config`](https://www.ssh.com/academy/ssh/config) in your `.ssh` directory.
Following is an example that illustrates addressing a number of the above cases:
```config theme={null}
Host appname
Hostname appname.fly.dev
User rails
Port 2222
Host *.appname
HostName %h.internal
ProxyJump rails@appname.fly.dev:2222
User rails
Port 2222
```
Additionally, if you want to avoid the fingerprint checking, you can add the following to each of the `Host` entries:
```config theme={null}
StrictHostKeyChecking no
```
## Related reading
* [`flyctl` SSH commands](/flyctl/cmd/fly_ssh) Official reference for how to use `fly ssh`/`fly ssh sftp` to access running machines.
* [Multiple processes inside a Fly.io app](/app-guides/multiple-processes) Explains how to run multiple process groups in a single machine/container. Relevant when you’re adding `sshd` alongside your main app.
# Per-User Dev Environments with Fly Machines
Source: https://docs.fly.io/blueprints/per-user-dev-environments
Host isolated dev or code-execution environments for each user on Fly Machines, with one app per user, optional volumes, and a fly-replay router for subdomains.
## Overview
Fly Machines are fast-launching VMs behind [a simple API](/machines/api), enabling you to launch tightly isolated app instances in milliseconds [all over the world](/reference/regions).
One interesting use case: running isolated dev environments for your users (or robots). Fly Machines are a safe execution sandbox for even the sketchiest user-generated (or LLM-generated) code.
This guide explains how to use Fly Machines to securely host ephemeral development and/or execution environments, complete with [dynamic subdomain routing](/networking/dynamic-request-routing) using `fly-replay`.
## What your architecture should include
* **Router app(s)**
* A Fly.io app to handle requests to wildcard subdomains (`*.example.com`). Uses `fly-replay` headers to transparently redirect each request to the correct app and machine. If you have clusters of users (or robots) in different geographic regions, you can spin up a router app in multiple regions. See [Connecting to User Machines](/blueprints/connecting-to-user-machines) for details on how to implement the routing pattern.
* **User apps (pre-created)**
* Dedicated per-user (or per-robot) Fly apps ([more about why you should create a dedicated app per customer/robot](/machines/guides-examples/one-app-per-user-why)), each containing isolated Fly Machines. App and Machine creation is not instantaneous, so we recommend provisioning a pool of these before you need them so you can quickly assign upon request.
* **Fly Machines (with optional volumes)**
* Fast-launching VMs that can be attached to persistent [Fly Volumes](/volumes).
### Example Architecture Diagram
### Router app(s)
Your router app handles all incoming wildcard traffic. Its responsibility is simple:
* Extract subdomains (like `alice.example.com` → `alice-123`).
* Look up the correct app (and optionally machine ID) for that user.
* Issue a `fly-replay` header directing the Fly Proxy to [internally redirect the request](/blueprints/connecting-to-user-machines#using-fly-replay) (this should add no more than \~10 milliseconds of latency if the router app is deployed close to the user).
* When appropriate, use [replay caching](/networking/dynamic-request-routing#replay-caching) to further reduce latency and load on the router app.
* Make sure you've added [a wildcard domain](/networking/custom-domain#add-your-domain-to-your-app) (\*.example.com) to your router app (read more about the [Certificates API reference](/machines/api/certificates-resource)).
### User apps
Creating apps dynamically for each user at request time can be slow. To ensure fast provisioning:
* **Pre-create** a pool of Fly apps and machines ahead of time (using the [Fly Machines API or CLI](/apps/overview)).
* Store app details (e.g., app\_name: `alice-123`) in a datastore accessible to your router app.
* Assign apps to users at provisioning time.
**Fly Machines**
You'll want to spin up at least one Machine per user app (but apps can have as many Machines as needed). If your dev environments need persistent storage (data that should survive Machine restarts):
* Attach Fly Volumes to each machine at creation time.
* Keep in mind that machine restarts clear temporary filesystem state but preserve volume data.
* Learn more about the [Machines API resource](/machines/api/machines-resource) and the [Volumes API resource](/machines/api/volumes-resource).
## Pointers & Footguns
* **Machines & volumes are tied to physical hardware:** hardware failures can destroy machines and attached volumes. **Always persist important user data** (code, config, outputs) to external storage (like [Tigris Data](/tigris) or AWS S3).
* **Your users will break their environments:** pre-create standby machines to handle hardware & runtime failures, or the inevitable user or robot poisoned environment. Pre-create standby machines that you can quickly activate in these scenarios.
* **Machine restarts reset ephemeral filesystem:** the temporary Fly Machine filesystem state resets on Machine restarts, ensuring clean environments. However, volume data remains persistent, making it useful for retaining user progress or state.
* **One app per user:** Putting all user Machines into a single app and routing to them with dynamic routing works, but `auto_stop` won't behave the way you expect once you scale. The Fly Proxy's stop loop is rate-limited: it [stops or suspends at most one Machine per region each pass, and runs every few minutes](/reference/fly-proxy-autostop-autostart#fly-proxy-process-to-stop-or-suspend-machines). That's fine for a normal app; with thousands of Machines the loop can't keep up, and most of your idle Machines stay running. Machines in a single app also share app-level secrets and a flat [private network](/networking/private-networking). A compromised user environment can reach every other Machine in the app. If you do keep all user Machines in one app, implement stop-when-idle behavior in your app or orchestrator (see [apps that shut down when idle](/launch/autostop-autostart#apps-that-shut-down-when-idle)). Don't rely on the Fly Proxy to keep most Machines stopped.
## Related reading
* [Connecting to User Machines](/blueprints/connecting-to-user-machines) How to manage routing and networking when you spin up a fleet of machines dedicated to individual users.
* [Multi‑container Machines](/machines/guides-examples/multi-container-machines) When your per‑user environment needs sidecars (logs, metric exporters, agent processes) alongside the main service.
# Run private apps with Flycast
Source: https://docs.fly.io/blueprints/private-applications-flycast
Deploy a private HTTP service reachable only on your org's network with Flycast on Fly.io, routed by Fly Proxy so stopped Machines wake on demand.
## Overview
A lot of the time your applications are made to be public and shared with the world. Sometimes you need to be more careful. When you deploy your apps on Fly.io, you get a private network for your organization. This lets your applications in other continents contact each other like they were in the same room.
Sometimes you need a middle ground between fully public apps and fully private apps. [Flycast](/networking/flycast) addresses are private but global IPv6 addresses inside your private network that go through the Fly Proxy, so you get all of the load management and Machine waking powers that you get for free with public apps.
This blueprint covers what Flycast is, when and why you’d want to use it, and shows you how to deploy a small private HTTP service that you can connect to over Flycast.
## What is Flycast?
Before we get started, let’s talk about Flycast and when you would want to use it. In general we can split every kind of Fly App into two categories: public apps and private apps.
A public app is what you’d expose to the public Internet for your users. These are usually hardened apps that allow users to do some things, but have access limitations that prevent them from stepping outside their bounds. These are mostly programs that listen over HTTP for browsers to interact with. Your users connect to a public app through the platform router via the .fly.dev domain or whatever other domain you’ve set up.
A private app is something internal, like a database server or a worker queue. These are things that run in the background and help you get things done, but are intentionally designed to NOT be exposed to the public Internet. You wouldn’t want to expose your Postgres or Valkey servers to anyone, would you?
However, with a fully private app, all connections go directly to the Machines via their .internal addresses, so you have to keep them running 24/7 to maintain connectivity. This is fine for services like database engines where you want them to be running all the time, but what about an admin panel? You want your admin panel to be separate from your main app so that users can ever get into it, even by accident, but you also want it to shut down when it’s not in use.
Flycast exists for this middle category of apps. With Flycast, your apps are only visible over your organization’s private network, but any traffic to them goes through the proxy so they can turn on when you need them and turn off when you don’t. This allows your administrative panels to be physically separate so that users can’t access them.
When you want to connect to an app via Flycast, you connect to `appname.flycast`.
### Security note
Just a heads-up. In general, it’s a bad idea to assume that network access barriers like Flycast or NAT are security layers. At best, this is an obfuscation layer that makes it more difficult for attackers to get into private applications. Flycast is not a replacement for authentication in your private applications. With Flycast, you don’t know *who* a request is coming from, but you do know that it’s coming from *something* or *someone* in your private network.
One of the biggest platform features that uses Flycast out of the box is [Managed Postgres](/postgres). Even though Flycast addresses are local to your private network, Managed Postgres still configures usernames and passwords for your database.
## Goal
We'll show Flycast off by deploying a small HTTP service that's only reachable from inside your organization's private network. The classic example is an internal admin panel: a service you and your apps need to reach, but that should never be exposed to the public internet. Flycast also lets the platform turn the service off when nobody's using it, so you're not paying to keep an idle admin panel running.
For this walkthrough we'll use the public [`nginxdemos/hello`](https://hub.docker.com/r/nginxdemos/hello) image as a stand-in for any private HTTP service. It's a tiny container that responds to HTTP requests with the server's hostname and address, which makes it easy to confirm that traffic is reaching the right Machine over Flycast.
## Prerequisites
To get started, you need to do the following:
* [sign up or sign in](/getting-started/sign-up-sign-in) to Fly.io
* [install flyctl](/flyctl/install) (the Fly CLI)
If you want to interact with your Flycast apps from your computer, you’ll need to [jack into your private network with WireGuard](/blueprints/connect-private-network-wireguard).
## Steps
Create a new folder on your computer called `flycast-demo` and open a terminal in it. We don't need any source code for this walkthrough. We'll launch the app directly from a public Docker image with the `--flycast` flag, which tells Fly Launch to allocate a private IPv6 address instead of public ones. We'll also pass `--no-deploy` so we can adjust one setting before the first deploy:
```
fly launch --image nginxdemos/hello --flycast --internal-port 80 --no-deploy
```
The name you choose for your app will be the hostname you use to reach it over Flycast (`.flycast`). When `fly launch` asks if you want to tweak the settings, you can accept the defaults.
The `--internal-port 80` flag tells Fly Launch that our app listens on port 80 (the default for the `nginxdemos/hello` image). Fly Launch defaults to port 8080, so without this flag the Fly Proxy wouldn't be able to reach the app. You could also set `internal_port = 80` in `fly.toml` directly in the next step; the flag is just a shortcut so you don't have to edit the file twice.
Open the generated `fly.toml` in your editor, find the `[http_service]` block, and change `force_https` to `false`:
```toml theme={null}
[http_service]
internal_port = 80
force_https = false
```
Fly Launch enables `force_https` by default, which makes the proxy return a 301 redirect from HTTP to HTTPS. That's a good default for public apps, but Flycast addresses don't have public TLS certificates, so the redirect would just break plain HTTP requests from inside your private network.
Now deploy the app:
```
fly deploy
```
After deploy finishes, you can see the list of IP addresses associated to an app with `fly ips list`:
```
$ fly ips list
VERSION IP TYPE REGION CREATED AT
v6 fdaa:3:9018:0:1::7 private global 23h12m ago
```
Learn more about [Fly.io public, private, shared and dedicated IP addresses](/networking/services#anycast-ip-addresses).
This app only has one IP address: a private Flycast IPv6 address. If it had public IP addresses, it'd look like this:
```
$ fly ips list -a recipeficator
VERSION IP TYPE REGION CREATED AT
v6 2a09:8280:1::37:7312:0 public (dedicated) global May 30 2024 13:51
v4 66.241.124.113 public (shared) Jan 1 0001 00:00
```
Now that we've proven it's private, let’s open an interactive shell Machine to play around with Flycast. Create the shell Machine with `fly machine run`, giving it enough memory to install a few packages with `apt`:
```
$ fly machine run --shell --vm-memory 1024 ubuntu
root@e784127b51e083:/#
```
The Ubuntu image we chose is very minimal, so we need to install a few tools such as ping, curl, and dig:
```
# apt update && apt install -y curl iputils-ping dnsutils
```
Say my app is named `flycast-demo`. Let’s look up its `.flycast` address with `nslookup flycast-demo.flycast`:
```
# nslookup flycast-demo.flycast
Server: fdaa::3
Address: fdaa::3#53
Name: flycast-demo.flycast
Address: fdaa:3:9018:0:1::7
```
It matches that IP address from earlier. Now let’s see what happens when we ping it:
```
# ping flycast-demo.flycast -c2
PING flycast-demo.flycast (fdaa:3:9018:0:1::7) 56 data bytes
64 bytes from fdaa:3:9018:0:1::7: icmp_seq=1 ttl=63 time=0.138 ms
64 bytes from fdaa:3:9018:0:1::7: icmp_seq=2 ttl=63 time=0.223 ms
--- flycast-demo.flycast ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.138/0.180/0.223/0.042 ms
```
Now let’s make a request to the app with curl:
```
# curl http://flycast-demo.flycast
```
You should get back an HTML page with the server's hostname and address. Wait a few moments so your app goes to sleep, then run the `time` command to see how long the first request takes:
```
# time curl -o /dev/null -s http://flycast-demo.flycast
real 0m1.539s
user 0m0.004s
sys 0m0.012s
```
It took a couple of seconds for the platform to wake the Machine up and make sure it was ready for your request. The next request is a lot faster:
```
# time curl -o /dev/null -s http://flycast-demo.flycast
real 0m0.018s
user 0m0.003s
sys 0m0.008s
```
And if you wait a few moments, it’ll spin back down.
And there we go! We’ve covered what Flycast is, why you’d want to use it, and walked through deploying a private HTTP service to show it off. The same pattern works for any internal app: admin panels, internal APIs, dashboards, or anything else you'd rather keep off the public internet.
## Related reading
* [Autostart and autostop private apps](/blueprints/autostart-internal-apps)
* [Flycast: private Fly.io services](/networking/flycast)
* [Private networking](/networking/private-networking)
# Remote MCP Server Hosting: Deploy MCP Servers on Fly.io
Source: https://docs.fly.io/blueprints/remote-mcp-servers
Host a remote MCP server on a Fly Machine that any MCP client can reach over HTTP and that stops when idle, as a multi-tenant app or one app per user.
## Overview
Hosting a remote MCP server on Fly.io means running it on a Fly Machine, a fast-booting isolated VM that any MCP client can reach over HTTP, and that can stop when idle and start again on the next request. The Model Context Protocol (MCP) gives LLMs new powers. It was originally developed by Anthropic and is now supported by most major LLM clients.
The protocol defines a standardized way of connecting tools and providing additional context to LLMs, not dissimilar to the way USB provides a standardized way to connect computers to peripherals and devices. Fly Machines are tightly isolated VMs that are perfect for running MCP servers.
This guide will help you understand, at a very high level, how to build, deploy, and connect remote MCP servers on Fly.io. Keep in mind that this is an nascent, still-emerging protocol – details are subject to change. For specific implementation details, the [Model Context Protocol](https://modelcontextprotocol.io/) site is the authoritative source of truth (complete with [a handy .txt version for LLM prompting](https://modelcontextprotocol.io/llms-full.txt)).
We've also started to integrate some MCP tooling into the `flyctl` that should give you a more concrete sense of how you might deploy MCP servers on Fly.io (read more in [this community forum post](https://community.fly.io/t/running-mcps-on-and-with-fly-io/24588)). Note that our MCP implementation is experimental and may still have sharp edges!
## Remote MCP Servers
MCP to date has been largely local-only, but the protocol also [specs out transports](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports) for remote MCP servers. Remote MCP servers solve a bunch of problems:
* Easier to update a centralized server instead of dispersed, local packages
* You can give MCP clients persistent connections that don't evaporate when someone closes their laptop
* Securely sandbox MCP server activity in case the robots go rogue
## Single Tenant or Multi-tenant
There are broadly two patterns you might want to follow when deploying a remote MCP server to Fly.io:
1. Multi-tenant MCP server (one app, many users)
2. Single-tenant MCP servers (one app per user)
We're partial to the single-tenant pattern – it ensures proper isolation, and also helps with minimize your Fly.io bill: unused Machines can stop and start as needed, so you won't waste resources on idle users (more about why we think one app per customer is the right pattern [here](/machines/guides-examples/one-app-per-user-why)). `fly-replay` makes it easy to route requests to the correct app / Fly Machine (see more detail about this pattern in [Per-user Dev Environments with Fly Machines](/blueprints/per-user-dev-environments)).
## Multi-tenant MCP Servers
You'll need two main components:
1. **MCP-Remote Shim (optional)**: Tiny client-side proxy that connects local MCP clients to your remote servers (only needed if the MCP client doesn't support auth and / or streamable HTTP requests to a remote MCP server). Handles authentication via a secret shared between the shim and the router app (authentication could be a simple API token, username + password, or a full OAuth dance). Securely stores and refreshes tokens as needed. To see an example, try the experimental `fly mcp proxy` command in the `flyctl`. This sets up a local proxy that forwards MCP client requests to a remote URL (more details in [the docs](/flyctl/cmd/fly_mcp) and [the community forum](https://community.fly.io/t/running-mcps-on-and-with-fly-io/24588)).
2. **MCP Server App**: This runs the actual MCP goodness and authenticates requests from the MCP client. Should have a single streamable HTTP endpoint path for MCP client connections, as well as any specific business logic or other integrations. To try our implementation, include the experimental `fly mcp wrap` command in the Dockerfile for your MCP server app. This instantiates a lightweight HTTP server to receive requests forwarded from the local MCP client via `fly mcp proxy` (more details in [the docs](/flyctl/cmd/fly_mcp) and [the community forum](https://community.fly.io/t/running-mcps-on-and-with-fly-io/24588)).
## Single-tenant MCP Servers
There are three main components:
1. **Router App**: Receives requests from the MCP client and handles auth, then routes requests to per-user apps + Fly Machines with `fly-replay`. See [Connecting to User Machines](/blueprints/connecting-to-user-machines) for details on implementing the routing pattern. Optionally handles user management and permissions. You can use the experimental `fly mcp wrap` command in the Dockerfile of your router app to instantiate a lightweight HTTP server to receive requests forwarded from a local MCP client (more details in [the docs](/flyctl/cmd/fly_mcp) and [the community forum](https://community.fly.io/t/running-mcps-on-and-with-fly-io/24588)). Note that `fly mcp wrap` does not handle request routing – you'll have to implement that separately.
2. **MCP Server Apps**: Per-user (or per-team) apps that run the actual MCP goodness. Should have a single streamable HTTP endpoint path for MCP client connections, as well as any specific business logic or other integrations.
3. **MCP-Remote Shim (optional)**: Tiny client-side proxy that connects local MCP clients to your remote servers (only needed if the MCP client doesn't support auth and / or streamable HTTP requests to a remote MCP server). Handles authentication via a secret shared between the shim and the router app (authentication could be a simple API token, username + password, or a full OAuth dance). Securely stores and refreshes tokens as needed. To see an example, try the experimental `fly mcp proxy` command in the `flyctl`, which sets up a local proxy that forwards MCP client requests to a remote URL (more details in [the docs](/flyctl/cmd/fly_mcp) and [the community forum](https://community.fly.io/t/running-mcps-on-and-with-fly-io/24588)).
## How Users Experience It
From your users' perspective, the experience should be delightfully simple:
1. They add a new MCP server to their MCP client with a simple one-liner
2. If needed, the user is walked through an authentication flow (ie a browser window pops open to kick off the OAuth dance or provide an API key)
3. After logging in, the MCP client can now access all the tools you've exposed
4. The connection persists across restarts without re-authentication
## Taking It Further
If you'd like your MCP servers to connect to third party OAuth APIs without having to directly handle API tokens, consider using [ssokenizer](https://github.com/superfly/ssokenizer) (or [tokenizer](https://github.com/superfly/tokenizer) for generic, non-OAuth flows).
## Related reading
* [Model Context Protocol (MCP) Documentation](/model-context-protocol) Learn more in our reference for MCP: what it is, how it works, and how to deploy servers.
* [Connecting to User Machines](/blueprints/connecting-to-user-machines) Find out about patterns for routing from a central router app to per‑user apps/machines via `fly‑replay`, which is useful when you build single‑tenant MCP setups.
* [Launching MCP Servers with flyctl](/flyctl/cmd/fly_mcp_server) Read about the experimental `fly mcp server` command: how to start a local or remote MCP server using `flyctl`, including flags and usage.
# High availability: run multiple Machines
Source: https://docs.fly.io/blueprints/resilient-apps-multiple-machines
A Fly Machine runs on one host and won't move if that host fails. Run at least two Machines per process, using autostop/autostart or standbys to limit cost.
> [!NOTE]
>
> **Fly Machines are fast-launching VMs, the basic compute unit of the Fly.io platform. Each Machine runs on a single physical host. If that host fails, the Machine goes down and does not automatically start again on another host.**
## Overview
To make your app resilient to single-host failure, create at least two Machines per app or process. Fly Proxy autostop/autostart and standby Machines are built-in platform features that you can use to start extra Machines only when needed.
***
## Resiliency: Why and When
Why resiliency? Machines are great until they aren’t. Hosts fail. Regions go dark. Networks get flaky. If your app only runs on one Machine, any of those failures means downtime. Running more than one Machine, or spreading them across regions, is how you keep the lights on when the infrastructure underneath you stumbles.
### When to use it
If you are building something people rely on, you probably want some resiliency. That includes production apps with paying users, customer-facing services where downtime makes people angry, or global apps that need to respond quickly no matter where users are. Regulations can also push you here if you are required to keep systems up in specific geographies.
### When it might not be worth it
For some apps, downtime just does not matter very much. A side project, a staging environment, or an internal tool might run fine on a single Machine. If the stakes are low, adding extra Machines can be more complexity than you need.
### Performance
Machines are really fast to start and stop. If your app has a lot of users, or bursts of high usage, then the Fly Proxy can load balance requests and automatically stop and start Machines based on traffic to your app. Keep one or more Machines running all the time if your app needs even faster first response times.
### Costs
You pay only for `rootfs` when Machines aren't running, not for CPU and RAM. Machine rootfs is [cheap](/about/pricing#stopped-fly-machines), like 18 cents a month for an average Elixir app that uses 1.2 GB of rootfs.
***
## Multiple Machines for apps with services
You can add more Machines for Fly Proxy to start and stop as needed, which is great for apps that have built-in replication or that don't share data.
### You get two Machines on first deploy
When you deploy an app for the first time with the `fly launch` or `fly deploy` command, you automatically get two identical Machines for processes that have HTTP/TCP services configured in `fly.toml`. The Machines have autostop/autostart enabled so that Fly Proxy can start and stop them based on traffic to your app. You'll also get this default redundant Machine when you `fly deploy` after scaling to zero.
> [!NOTE]
>
> **Volumes:** You'll only get one Machine with `fly launch` for processes or apps with volumes mounted. Volumes don't automatically replicate your data for you, so you'll need to set that up before intentionally creating more Machines with volumes.
### Add more Machines yourself
If your app doesn't already have multiple Machines with autostop/autostart configured, then you can set it up yourself. You can create any number of Machines to both meet user demand and provide redundancy against host failures.
#### Set up autostop/autostart
Use [Fly Proxy autostop/autostart](/launch/autostop-autostart) to automatically stop and start Machines based on traffic. Keep one or more Machines running in your primary region if you want to. Example `fly.toml` config:
```toml theme={null}
[http_service]
internal_port = 8080
force_https = true
auto_stop_machines = "stop" # Fly Proxy stops Machines based on traffic
auto_start_machines = true # Fly Proxy starts Machines based on traffic
min_machines_running = 0 # No. of Machines to keep running in primary region
[http_service.concurrency]
type = "requests"
soft_limit = 200 # Used by Fly Proxy to determine Machine excess capacity
```
Fly Proxy uses the concurrency `soft_limit` to determine if Machines have excess capacity. Learn more about [how Fly Proxy autostop/autostart works](/reference/fly-proxy-autostop-autostart).
**Using the Machines API:** To add or change the autostop/autostart settings with the Machines API, use the settings in the `services` object of the [Machine config](/machines/api/machines-resource#machine-config-object-properties) in your `create` or `update` calls.
#### 2. Create more Machines
Create extra Machines, or even just one extra Machine, using flyctl:
```
fly scale count 2
```
Or deploy even more Machines in more than one region:
```
fly scale count 20 --region ams,ewr,gig
```
Learn more about [scaling the number of Machines](/launch/scale-count).
**Using the Machines API:** To scale the number of Machines with the Machines API, you can "clone" Machines. Get and modify a Machine config and use that config to create new Machines in any region. See the[ Machines resource docs](/machines/api/machines-resource).
***
## Standby Machines for apps without services
When apps or processes are running tools like cron that don't require local storage or accept external requests, it's common to run only one Machine. Since these Machines don't have services configured, they can't be automatically started and stopped by the Fly Proxy. To add redundancy against host failures for this kind of Machine, use a standby Machine; it stays stopped and ready to take over in case the original Machine becomes unavailable.
Unlike Fly Proxy autostop/autostart, which starts Machines based on app traffic, a standby Machine watches the Machine it's paired to and starts only if that Machine becomes unavailable. Learn more about [standby Machines](/apps/app-availability#standby-machines-for-process-groups-without-services).
### You get a standby Machine on first deploy
When you deploy an app for the first time with the `fly launch` or `fly deploy` command, you automatically get a standby Machine for processes that don't have services configured in `fly.toml` (and therefore aren't visible to Fly Proxy). That standby Machine is a copy of the original, running Machine. You'll also get this standby Machine when you `fly deploy` after scaling to zero.
### Create a standby Machine yourself
If you don't already have a standby Machine for your worker Machines, then you can create one yourself.
To create a standby Machine that will take over the same tasks when a worker Machine is down, use this one command to clone the worker Machine and make the clone a standby:
```
fly machine clone --standby-for
```
**Using the Machines API:** To create a standby Machine with the Machines API, get and modify your worker Machine config and use that config to create a new standby Machine. Specify the standby in the Machine's config object: [config.standbys](/machines/api/machines-resource#machine-config-object-properties).
### Bonus uses for standby Machines
A standby Machine doesn't need to be a copy of the Machine it's watching. You can configure the standby Machine to run with any image you like when a watched Machine becomes unavailable; you can use it to run a recovery script, trigger alerts, or anything else. You can create a standby Machine for any Machine, including one with services.
To create a standby Machine from a different image:
```
fly machine run --standby-for
```
***
## Scaling to multiple regions
Running multiple Machines in one region protects you from host failures. But entire regions can fail too. Power cuts, network issues, and upstream outages do not happen often, but when they do, every Machine in that region goes down. If your app needs to weather that kind of failure, the next step is spreading out across regions.
### When to use it
Multi-region setups make sense when uptime really matters. That might mean production systems with strict SLAs, customer-facing apps where downtime causes pain, or global services that need to stay responsive close to users. Regulations can also be a factor if you need to keep infrastructure running in specific geographies.
### Example
You can place Machines in more than one region with a single command:
```
fly scale count 6 --region ams,ewr,syd
```
This example runs two Machines in Amsterdam, two in New Jersey, and two in Sydney. Fly Proxy will route users to the closest healthy region.
One important caveat: volumes do not replicate across regions. If your app writes data, you need to think carefully about where that data lives and how it syncs. A common approach is to keep one primary “write” region and treat others as read-only.
### Keep in mind
Multi-region comes with real advantages, but also some trade-offs. Users near your regional Machines will see faster responses, but coordinating data across regions is hard, especially since most apps are not designed for global writes. And while redundancy improves reliability, it also increases your costs: more Machines in more places means more expense, and cross-region traffic can add up quickly.
***
## Summing up
Resiliency builds in layers. At the base, you have a single Machine: cheap, simple, and fragile. Add a second Machine in the same region and you are protected from host failures. Spread Machines across regions and you are resilient to outages that take down an entire site, with the added bonus of faster responses for users around the world.
Each layer improves availability, but it also adds cost and complexity. The right choice depends on what you are building and who is depending on it.
Whatever strategy you choose, practice it. Do not wait for a real outage to find out how your app behaves. Stop a Machine and see if traffic shifts. Kill a region and watch how your data layer responds. A resiliency plan only works if you have rehearsed it enough to trust it.
## Related reading
If you want to dig deeper into how Fly apps stay available and what failure modes to expect, these are good next steps:
* [App availability and resiliency](/apps/app-availability): An overview of the all features that can make your app more resistant to events like hardware failures or outages.
* [Multi-region with Fly-Replay](/blueprints/multi-region-fly-replay): How to serve reads close to users and route writes back to a primary region.
* [Machine placement and regional capacity](/machines/guides-examples/machine-placement): How Fly decides where to place your Machines, how to guide placement yourself, and what happens when a region runs out of capacity.
* [Run Ordinary Rails Apps Globally](https://fly.io/ruby-dispatch/run-ordinary-rails-apps-globally/): Project-level example of Rails distributed across regions with smart routing.
# Git Branch Preview Environments on Github
Source: https://docs.fly.io/blueprints/review-apps-guide
Deploy ephemeral review apps for every pull request using a GitHub Actions workflow and the fly-pr-review-apps Action on Fly.io, with custom specs and secrets.
## Overview
Review apps are a great way to preview new features, changes, and bug fixes. This guide will teach you how to automatically generate ephemeral "review apps" on Fly.io for each pull request (PR) using GitHub Actions. This approach can be applied to other Git services and source code versioning software that supports branching and hooks.
## Quick Start
If you've worked with GitHub Actions before and want to jump right in, here are the steps to take:
1. Create a new repository secret on your GitHub repository called `FLY_API_TOKEN` and provide it with the returned value of `fly tokens org `
2. Create a new workflow in your project at `.github/workflows/fly-review.yml` and add the following [YAML code](https://gist.github.com/anniebabannie/3cb800f2a890a6f3ed3167c09a0234dd).
3. Commit and push the changes containing your GitHub workflow.
And that's it! Any new pull requests will trigger the creation of review apps for your application.
By default, review apps will be available at `https://pr---.fly.dev`.
The workflow described above will spin up a **single application** without other resources; if data stores, multiple Fly Apps, or other resources are required, see [Customizing your workflow](#customize-your-workflow).
For more details about each step, follow the guide below.
## Setting up review apps on Fly.io
### Set your Fly API token
Your GitHub Action will need a Fly API token to deploy and manage new review apps on Fly.io. You can get this token using the following [flyctl](/flyctl/install) command:
```bash theme={null}
fly tokens org
```
This token is specific to a single Fly.io organization and is valid for applications belonging the that organization.
Next, let's save our API token as a secret in our GitHub repository. Visit the Settings tab in your repository, then Secrets and variables → Actions. Alternatively, you can visit `https://github.com///settings/secrets/actions`. There, create a new **Repository Secret** called `FLY_API_TOKEN` with the value of your token.
### Create your GitHub Action
Next, let's create a new GitHub Action by creating a new YAML file in your app's repository for our workflow:
```bash theme={null}
touch .github/workflows/fly-review.yml
```
Then add the following to your new `fly-review.yml` file:
```yaml theme={null}
name: Deploy Review App
on:
# Run this workflow on every PR event. Existing review apps will be updated when the PR is updated.
pull_request:
types: [opened, reopened, synchronize, closed]
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
# Set these to your Fly.io organization and preferred region.
FLY_REGION: iad
FLY_ORG: personal
jobs:
review_app:
runs-on: ubuntu-latest
outputs:
url: ${{ steps.deploy.outputs.url }}
# Only run one deployment at a time per PR.
concurrency:
group: pr-${{ github.event.number }}
# Deploying apps with this "review" environment allows the URL for the app to be displayed in the PR UI.
# Feel free to change the name of this environment.
environment:
name: review
# The script in the `deploy` sets the URL output for each review app.
url: ${{ steps.deploy.outputs.url }}
steps:
- name: Get code
uses: actions/checkout@v4
- name: Deploy PR app to Fly.io
id: deploy
uses: superfly/fly-pr-review-apps@1.2.1
```
In the `deploy` step, `superfly/fly-pr-review` contains the custom Action with a shell script that creates and deploys your review apps. If you're curious, check out [entrypoint.sh](https://github.com/superfly/fly-pr-review-apps/blob/main/entrypoint.sh) for full details on how the script works.
### Commit and push your workflow
Finally, commit and push your changes to your GitHub repository. Once pushed, any new PRs will spin up a review app for you to preview your code on Fly.io.
## Customize your workflow
Often our applications do not operate in isolation and require other resources to function. The sample workflow mentioned above spins up a stand-alone Fly App, but you can customize your `fly-review.yml` GitHub workflow to create additional Fly Apps and resources, such as a Redis, Memcached, etc. You can find examples of these in the [README](https://github.com/superfly/fly-pr-review-apps/tree/main) for `superfly/fly-pr-review-apps`.
### Customize with inputs
The `superfly/fly-pr-review-apps` custom GitHub Action also provides a number of **inputs** that can be added to tweak behavior. See the [README](https://github.com/superfly/fly-pr-review-apps/tree/main) for the complete list.
These inputs can be added to the `deploy` step in your workflow using the `with` key, like so:
```yaml theme={null}
# fly-review.yml
# ...
steps:
# ...
- name: Deploy PR app to Fly.io
id: deploy
uses: superfly/fly-pr-review-apps@1.2.1
with:
name: my-app-name-pr-${{ github.event.number }}
config: fly.review.toml
```
### Defining resource specs
You probably won't need the same specifications for the Fly Machines used by review apps as you do for your production app. There are two ways to override the resources used in review apps.
**1. Specifying a different `config` file:** By default, review apps will use the `fly.toml` in your application root, but you can create a separate TOML file with different specs using the `config` input, like so:
```yaml theme={null}
# fly-review.yml
# ...
steps:
# ...
- name: Deploy PR app to Fly.io
id: deploy
uses: superfly/fly-pr-review-apps@1.2.1
with:
config: fly.review.toml
```
**2. Setting specs in GitHub Action inputs:** Alternatively, you can set the specs for the Fly Machines used by your review apps using the following inputs:
* `vmsize`
* `cpu`
* `cpukind`
* `memory`
* `ha`
### Default secrets and environment variables
We recommend setting default values for any secrets or environment variables your review apps require so you don't have to set them individually. You can do this by defining each secret as either a **repository secret** *or* an **environment secret** in your GitHub repo. If using environment secrets, these should be set on the environment used in your workflow, such as `review`.
Once you have your secrets and environment variables set in GitHub, add each of them to your `fly-review.yml` workflow, separated by a space, like so:
```yaml theme={null}
# fly-review.yml
# ...
steps:
# ...
- name: Deploy PR app to Fly.io
id: deploy
uses: superfly/fly-pr-review-apps@1.2.1
with:
secrets: FOOBAR=${{ secrets.FOOBAR }} SOME_API_KEY=${{ secrets.SOME_API_KEY}}
```
## Related Reading
* [Continuous Deployment with Fly.io and GitHub Actions](/launch/continuous-deployment-with-github-actions) Shows how to set up automated CI/CD pipelines using GitHub Actions and Fly.io — a natural complement to per‑PR review deployments.
* [Get information about an app](/apps/info) A handy guide to inspecting apps, machines, statuses via CLI/API — useful when you’re creating and destroying review apps on pull requests.
# Roll back a deploy
Source: https://docs.fly.io/blueprints/rollback-guide
Undo a bad Fly.io deploy: find a previous release image with fly releases --image, redeploy it, tag images clearly, and handle database migrations separately.
## Overview: Why rollback?
Fly.io runs your apps close to your users, in VMs you can spin up and tear down in seconds. But what if you spin up something broken? If a deploy goes wrong, it’s easy to roll back: just tell Fly to redeploy a previous image. There’s no special `rollback` command because you don’t need one. Rollbacks use the same deploy mechanism you already know.
If you know which image was working before, you can deploy it again. You just need to know where to find the image, and how to tell Fly to use it. The trick is: you’re rolling back the *VM image*, not the database. If you ran migrations in that bad deploy, you’ll have to undo those separately in your application or migration code. Fly isn’t going to time-travel your data.
***
## How to rollback
### Step 1: See what you’ve shipped
Fly keeps a history of your releases, including the exact container image each one used. Run:
```bash theme={null}
fly releases --app myapp --image
```
You’ll see a list of past releases with timestamps and image tags:
```
v42 2025-08-12T16:21Z registry.fly.io/myapp:deployment-01HXXXXX
v41 2025-08-10T09:14Z registry.fly.io/myapp:deployment-01HYYYYY
```
Pick the one before things went sideways.
***
### Step 2: Deploy that exact image
Let’s say `v41` was fine. You can redeploy its image directly:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:deployment-01HYYYYY
```
That’s it — no rebuild, no code checkout, just telling Fly to boot the older VM image. This works as long as the image is still in the Fly registry.
***
### Tip: Stop living in `deployment-01HXXXXX` hell: use human-readable tags
The default auto-generated tags work for machines, not humans. You can make your life easier by tagging builds with something you’ll recognize later — a git commit hash, a version number, or even just `before-i-broke-prod`.
When you deploy, pass `--image-label` to set a tag:
```bash theme={null}
fly deploy --image-label v1.2.3
```
That command:
1. Builds your app
2. Tags the image as `v1.2.3`
3. Deploys it
Rolling back later becomes:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:v1.2.3
```
***
## How long does Fly keep your images?
Right now, Fly doesn’t promise to store app images forever. If an image hasn’t been deployed for a while, it might be pruned from our registry to free up space. If you need a “guaranteed forever” rollback target, push your image to a public container registry (like Docker Hub, GHCR, or ECR) and deploy from there. See [Using the Fly Docker Registry](/blueprints/using-the-fly-docker-registry#to-deploy-a-public-image-from-docker-hub-or-another-public-registry) for details.
***
## Things to watch out for
Rollbacks are simple in Fly, but they aren’t magic. A few details are worth keeping in mind:
* **Database schema drift** → If a deploy ran destructive migrations (dropping a column, renaming a table), rolling back the app image might not help. Plan migrations so they’re safe both forward and backward.
* **Image retention** → Images don’t live forever in Fly’s registry. For long-term rollback insurance, push builds to your own container registry.
* **Config, Secrets and `fly.toml`** → Rollbacks don’t undo config changes. When you deploy an older image, Fly still uses your current `fly.toml`, along with whatever env vars and secrets are set now. If a past deploy depended on an older config, you’ll need to update those settings manually when rolling back.
* **Autoscaling** → If your app scaled up during the bad release (say traffic spiked and Fly added more machines), those extra machines don’t vanish the instant you roll back. They’ll hang around until autoscaling decides to scale them down. This is expected behavior and your rollback still worked; you just have more machines than usual for a while. Read more about [metrics-based autoscaling](/launch/autoscale-by-metric).
* **Monitoring** → Always confirm a rollback stabilized things with logs , probes (`fly logs`, `fly status,` Grafana metrics), not just the success message from `fly deploy`.
* **Regions** → By default, rollbacks redeploy globally. If you run in multiple regions (say `iad`, `lhr`, `syd`), you might test a rollback in just one region before rolling back everywhere. You can do this with the `--region` flag:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:v1.2.3 --region iad
```
***
## Tip: Faster rollbacks with deploy strategies
By default, Fly uses `rolling` deploys: new machines come up before old ones shut down. That’s safe for upgrades, but if you know the current release is bad, you may want a quicker cutover.
### All-at-once rollback:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:v1.2.3 --strategy immediate
```
This stops old machines and boots the rollback image as quickly as possible.
### Controlled speedup:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:v1.2.3 --max-unavailable 0.5
```
This lets you take down up to half your machines at once, moving faster than a cautious rolling deploy while avoiding a full stop.
Use `--strategy immediate` when speed matters more than downtime, and `--max-unavailable` when you want a hedge between safety and speed.
***
## Summing up
Rolling back on Fly is basically just *deploying an older image*. Once you’ve got the muscle memory for `fly releases --image` and `fly deploy --image`, it’s a quick, predictable way to recover from bad deploys. And if you start tagging images with human-readable labels, you’ll have an easier time remembering which one to trust when production is on fire.
## Related reading
* [Seamless Deployments on Fly.io](/blueprints/seamless-deployments) How to set up deploys that avoid downtime and make rollbacks less painful.
* [Using the Fly Docker Registry](/blueprints/using-the-fly-docker-registry) Details on image tagging and retention — key context for understanding how rollbacks work.
* [Staging and Production Isolation](/blueprints/staging-prod-isolation) Tips on environment isolation that reduce the blast radius of a bad release.
# Zero-downtime deployments
Source: https://docs.fly.io/blueprints/seamless-deployments
Get zero-downtime deploys on Fly.io with health checks that work, a rolling, canary, or bluegreen strategy, and release_command for migrations.
## Overview
> [!NOTE]
>
> **Zero-downtime deploys aren’t magic. They’re just health checks that work. Fly.io apps run on individual VMs, and we won’t start routing traffic to a new one until it proves it’s alive. Here's how to get seamless deploys without breaking things or relying on hope as a strategy.**
## Your deploy is only as good as your health checks
Fly.io deployments look like magic when health checks are working. New Machines boot. Old ones go down. Users never notice. But behind the scenes, the [Fly Proxy](/reference/fly-proxy) is watching your Machines and only sending them traffic if they pass your health checks.
[Health checks](/reference/health-checks) are how you tell the platform that your app is alive. If they're missing or broken, you're flying in the dark.
In your `fly.toml`, you'll usually define health checks under `http_service.checks` (or the alternative `services.http_checks` if you're not using the http\_service shortcut) for HTTP-based apps. These checks are simple: make a request to a path like `/`, expect a `200 OK` response, and fail if you get a redirect or timeout. If your app redirects HTTP to HTTPS, add a header like `X-Forwarded-Proto = "https"` to avoid false failures.
The most common health check failure modes are easy to trip over:
* getting 301/302s instead of 200 responses
* hitting a path that loads slowly under cold start conditions
* forgetting to delay checks with a `grace_period` after boot
A typical HTTP health check might look like this:
```
[http_service.checks]
interval = "15s"
timeout = "2s"
grace_period = "10s"
method = "GET"
path = "/healthz"
headers = { X-Forwarded-Proto = "https" }
```
For apps that aren't serving HTTP, or where you want lighter checks, [TCP health checks](/reference/health-checks#service-level-checks) exist too. These just attempt to open a socket to the port you've configured; if the connection is accepted, the Machine is healthy.
[Top-level health checks](/reference/health-checks#top-level-checks)—defined in the `[checks]` section of your `fly.toml`—are for observability, not traffic control. They don't influence routing; if a Machine fails one, we won’t pull it out of rotation. That makes them ideal for internal monitoring and alerting, especially for background workers or other non-public-facing services where you still want to know if something’s gone sideways, but don’t need Fly to reroute traffic automatically.
Deployments get another layer: `machine_checks`. These aren't about routing traffic; they're about catching bad deploys before they go live. A `machine_check` boots a throwaway Machine, sets `FLY_TEST_MACHINE_IP`, and lets you poke it however you want. You can run integration tests, perform simulated traffic, check database access, or validate config. Fail the check, and the deploy halts. This is perfect for canary testing new releases.
**Example:**
```
[services.machine_checks]
command = ["/bin/bash", "-c", "curl -f http://$FLY_TEST_MACHINE_IP:8080/healthz"]
interval = "30s"
timeout = "5s"
```
Note: `machine_checks` run in a temporary Machine with the new image before it's deployed anywhere else. They're isolated, so they won't affect your running app—but they can save you from pushing broken code into production.
If a Machine fails a health check, Fly Proxy pulls it out of the rotation. That Machine can still be running, logging, and debugging, but it's invisible to your users.
## Deployment strategies: choose your own trade-offs
Working with health checks and at least two Machines, you can avoid downtime. The platform won't kill a healthy Machine until a new one is up and running.
Fly.io supports a few deployment strategies in the `[deploy]` section of `fly.toml`:
* **Rolling (default):** Replace Machines one at a time. Use `max_unavailable` to control how many go down at once (e.g., `1` or `25%`).
* **Immediate:** Burn the ships. All Machines update at once, no health checks, hope for the best.
* **Canary:** Start with one Machine. If it's healthy, continue with rolling. This can't be used with attached volumes.
* **Bluegreen:** Boot new Machines alongside old ones. Only switch traffic after all new Machines pass checks. Fastest, safest, but also can't use attached volumes.
Every deploy also includes a smoke check: Fly watches Machines for \~10 seconds after they start. If they crash repeatedly, the deploy fails.
Don't forget to set `wait_timeout` if your image is big or startup is slow. It's easy to hit timeouts before Machines even start.
## One-off tasks with `release_command`
Sometimes you need to run a script before your app is actually deployed, like a database migration or other one-off prep work. The `release_command` in your `fly.toml` lets you do exactly that. It spins up a fresh temporary Machine, runs your command, then shuts it down and destroys it. It doesn’t join your deployed Machines.
This runs once per deploy attempt, using the built image and your app’s environment. It won’t have volumes attached, so it’s not for anything that needs persistent state.
If `release_command` fails, the deploy won’t proceed. This is usually what you want. If the migration didn’t work, it's better to stop there.
We’ve got more details, including how `ENTRYPOINT` and `CMD` are handled, in the [reference for `fly.toml`](/reference/configuration#run-one-off-commands-before-releasing-a-deployment).
## Zero-downtime is a shared responsibility
The Fly.io platform will keep your Machines healthy and traffic flowing. But there's one thing it can't fix: application incompatibility during deploys.
Say you're changing your database schema. You run the migration in a `release_command`, then your Machines update one by one. If the old app code can't handle the new schema, users will see errors while old Machines are still running. Same goes for bluegreen deployments: the new Machines are healthy, but traffic doesn't switch until *all* of them are.
Fixing this is on you. You need to design schema and app changes that can coexist briefly. That usually means:
* App code that works with both the old and new database schemas
* Migrations that add columns or tables but don't drop or rename until later
Every framework has guidance for this. Read it. Follow it. Then health checks, deployment strategies, and the platform can do their jobs.
Deploying without downtime isn't automatic. But with a little help from your health checks and some care in your app logic, it's well within reach.
## Related reading
* [Health Checks](/reference/health-checks)
* [Deploy an App](/launch/deploy)
* [App Configuration (`fly.toml`)](/reference/configuration)
# Setting Hard and Soft Concurrency Limits on Fly.io
Source: https://docs.fly.io/blueprints/setting-concurrency-limits
Tune soft_limit and hard_limit for Fly Machines by load testing a copy of your app, then let autostop/autostart scale within those limits.
## Overview
On Fly.io, machines aren’t created automatically based on traffic; you create them manually. But once a machine is created, it can be stopped when idle (meaning you’re not paying for it), and started again instantly when load picks up. Our “autostart/autostop” feature can drive this process for you — but it only works well if you’ve tuned `soft_limit` and `hard_limit` correctly. Those limits tell Fly when a machine is too busy and it's time to bring more machines online.
## How to tune concurrency limits
Soft and hard limits for Fly Machines don’t have to be a black art. Here’s a quick rundown:
* **hard\_limit**: max concurrent requests the machine will ever handle.
* **soft\_limit**: how many it can handle *comfortably*, before Fly begins starting more machines. Incoming requests are distributed on a round-robin fashion to started machines.
The best way to dial this in is to actually test it. Based on the target app you’re working to scale, create a separate Fly app copy with the same machine config. Now set both limits absurdly high (say, 1000), and hammer it with load using something like `wrk`, `hey`, `ab`, `Locust`, or `k6`. Watch CPU, memory, and latency.
What you’re looking for is the point where responses start to slow down below your comfort threshold. That’s your hard limit—just below that threshold. Your comfort level is up to you; a real-time application will want to keep responses under 100ms, but an ordinary CRUD application might be just fine with request times around 250ms.
Now set your soft limit. The soft limit is there to give our proxy time to bring another machine online before your users notice lag. Example: if things start dragging at 20 concurrent requests, set `hard_limit`: 18 and `soft_limit`: 12.
Once you’ve got your limits dialed in, turning on `auto_start_machines` and `auto_stop_machines` gives you hands-off scaling behavior that still respects the boundaries you’ve set. Your app shouldn’t need babysitting to handle spikes, and you won’t end up running more machines than you meant to. It’s a good balance between elasticity and control.
## Capacity Planning: A Real Example
Let’s say:
* Each request takes \~100ms to serve.
* You’re expecting 750 requests per second.
* One machine handles about 10 concurrent requests before slowing down.
That means you need 75 concurrent request slots to keep up. So: 750 rps \* 100ms = 75 concurrent requests. At 10 per machine, you’ll need 8 machines.
In order to serve requests as they arrive (that is, dispatch 750 requests/second) with an average processing time per request of 100ms, 75 “workers”, or concurrent requests handled, are needed. Based on the above machine capacity of 10 requests, 8 machines should be able to handle the proposed load.
## Related reading
* [Guidelines for Concurrency Settings](/apps/concurrency) Read more in this reference for the `soft_limit`, `hard_limit`, and `type` settings you’ll use in your `fly.toml`.
* [App Configuration: `http_service.concurrency`](/reference/configuration#http_service-concurrency) Check out this section of the `fly.toml` config reference explaining concurrency parameters.
* [Load Balancing on Fly.io](/reference/load-balancing) Find out how your concurrency limits affect how the Fly Proxy routes traffic across machines—and why your soft/hard limits matter.
# Shared Nothing Architecture
Source: https://docs.fly.io/blueprints/shared-nothing
Scale a Rails app as a shared-nothing monolith on Fly.io, with per-tenant SQLite on volumes, Phusion Passenger, Fly-Replay routing, and backups.
[YouTube video player](https://www.youtube-nocookie.com/embed/_cT4Unk5RiU)
> [!NOTE]
>
> This guide from Rails and JS expert Sam Ruby is based on his personal capacity as a ballroom dance nerd. It covers how a serverful, stateful, shared-nothing architecture hit the spot for him when it came time to scale to multiple customers in different parts of the world.
## Introduction
Once upon a time, I found myself double booked at a ballroom dance competition. Read that [story in more detail on my blog](http://intertwingly.net/blog/2022/08/13/Unretiring), but essentially: you’re only supposed to get one shot to dance in a given heat, and I ended up listed twice in the spreadsheet for that heat. I decided to write an app to take the load of manual entry and scheduling off of organizers, and so that no one would ever be asked to do something impossible and be at two places at one time.
Showcase (that’s what I called it) started out as a Ruby on Rails app and a database; it kept track of competitors and their schedules for a single Showcase (that’s what these ballroom dance events are called). I ran it on a single Mac Mini in my home.
Over time, it grew out of my attic and onto VMs in the cloud. It supports more events in different cities, and does more things. Conventional wisdom is that as you grow, you make each type of service reusable and put it into a separate set of machines.
If this approach isn’t the best one for your app, throw it out the window! I did. In my case, it made a lot of sense to put all the services into a single VM, and scale out by repeating that set into new VMs. This works great for me. Maybe it’s right for your app, too! Maybe not! Prepare to bathe in the gory details of my stateful, serverful, shared-nothing monolith.
## Shared-nothing architectures
> [!NOTE]
>
> A shared-nothing architecture is a distributed computing architecture in which each update request is satisfied by a single node (processor/memory/storage unit) in a computer cluster.
As you might expect, a Showcase competition happens live and in a single location. Very few people need access to the system, and they’re often all in the same room; picture a few judges entering scores at the end of a number—that’s your peak simultaneous users. Data’s not shared between events–they don’t want or need to share a database and Redis cache.
Shared-nothing architectures are nothing new. An excerpt from [Wikipedia](https://en.wikipedia.org/wiki/Shared-nothing_architecture):
> The intent is to eliminate contention among nodes. Nodes do not share (independently access) the same memory or storage. One alternative architecture is shared everything, in which requests are satisfied by arbitrary combinations of nodes. This may introduce contention, as multiple nodes may seek to update the same data at the same time.
While serverless is one kind of a shared-nothing architecture - this post explores another variation. One where servers are explicitly deployed,
users are assigned to servers, and users have databases that aren't shared.
This application started, like most applications, small. And that's where the story begins.
## A single event
> [!NOTE]
>
> A Fly.io machine is not merely a Docker container, but rather a [full VM](https://fly.io/blog/docker-without-docker/). Treat it as such: run as many services as you like on a single machine.
My first showcase was in Harrisburg, Pennsylvania. The app I developed initially supported only that one event, and all of the services needed to support the event ran on a single machine. Like most Rails applications,
this consisted of a Rails application, a database, and (eventually) Redis. Over time a number of supporting services were added (NGINX, opensshd, rsyncd, and a small log "heartbeat" script).
Today, these all happily run on a single Fly.io machine.
Conventional wisdom is that as you grow you put each type of service into a separate set of machines. That isn't always the right approach.
An excerpt from DHH's [Majestic Monolith](https://signalvnoise.com/svn3/the-majestic-monolith/):
> Every time you extract a collaboration between objects to a collaboration between systems, you’re accepting a world of hurt with a myriad of liabilities and failure states. What to do when services are down, how to migrate in concert, and all the pain of running many services in the first place.
Another way to look at this is: as needs grow, do you really want to share a database and Redis instances between customers? (Or, in this case, between events?)
If "no" turns out to be a viable answer: go with it. Put all of the services needed to support one customer in one Docker image and then repeat that set as often
as needed. Eliminating the need for a network in producing a response to a request both increases throughput and reliability.
There are a lot of [options for running multiple processes inside a Fly.io App](/app-guides/multiple-processes).
I use a combination of [ENTRYPOINT](https://github.com/rubys/showcase/blob/main/bin/docker-entrypoint) running a
[deploy](https://github.com/rubys/showcase/blob/main/bin/deploy) script and a [procfile](https://github.com/rubys/showcase/blob/main/Procfile.fly).
Given the load for each event is only for one customer, I only need 1 CPU per server. While CPU is not a concern, it still is important to size
the amount of RAM needed.
Even with all of the services running, according to [Graftana/Fly Metrics](https://fly-metrics.net/), each instance uses about 220MB of RAM when idle,
and grows to 450MB when active. That being said, I have seen machines fail due to [OOM](https://en.wikipedia.org/wiki/Out_of_memory) with only 512MB, so I give
each machine 1GB. I also define an additional 2G of [swap](/reference/configuration#swap_size_mb-option) as
I would rather the machine run slower under peak load than crash.
Finally, invest the time to set up a [WireGuard network](/networking/private-networking) that lets you VPN into your own private network of machines.
## Multi-tenancy
> [!NOTE]
>
> The code base supports only a single event. Running multiple events on a single machine is the next step before jumping into multiple machines.
There are a number of blog posts out there on how to do
[multi-tenancy](https://blog.arkency.com/comparison-of-approaches-to-multitenancy-in-rails-apps/)
with Rails. [Phusion Passenger](https://www.phusionpassenger.com/) provides a [different
approach](https://stackoverflow.com/questions/48669947/multitenancy-passenger-rails-multiple-apps-different-versions-same-domain).
For this use case, the Phusion Passenger approach is the best match. The
database for a mid-size local event is about a megabyte in size (about the size
of a single camera image), and can be kept in SQLite. Passenger provides a
[`passenger_min_instances`](https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_min_instances)
`0` option that allow a reasonable number of past, present, and future events
to be hosted essentially without any overhead when not in use. This does mean
that you have to accept the cold start times of the first access, but that
appears to be two seconds or less on modern hardware, which is acceptable.
The NGINX configuration file defines a set of environment variables for each tenant to control
the name of such things as the database, the log file, base URL, and pidfile.
For web sockets (Action Cable), NGINX is [preferred over Apache
httpd](https://www.phusionpassenger.com/library/config/apache/action_cable_integration/).
The [documentation for Deploying multiple apps on a single server
(multitenancy)](https://www.phusionpassenger.com/library/deploy/nginx/) is
still listed as todo, but the following is what I have been able to figure out:
* One action cable process is allocated per server (i.e., listen port).
* In order to share the action cable process, all apps on the same server will
need to share the same Redis URL and channel prefix. The [Rails
documentation](https://guides.rubyonrails.org/action_cable_overview.html#redis-adapter)
suggests that you use a different channel prefix for different applications
on the same server -- **IGNORE THAT**.
* Instead, use environment variables to stream from, and broadcast to, different
action cable channels.
The end result is what outwardly appears to be a single Rails app, with a
single set of assets and a single cable. One additional rails instance
serves the index and provides a global administration interface.
Recapping: a single server can host multiple events, each event is a separate instance of the same Rails application with a separate SQLite database placed on a volume. The server is always on (`auto_stop_machines = false`), but the Rails apps are spun up when needed and spin down when idle (using [`passenger_pool_idle_time`](https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_pool_idle_time))
## Multiple regions
> [!NOTE]
>
> Distributing an app across multiple regions not only lets you service requests close
> to your users, it enables horizontal scalability and isolation.
While most of my users are US based, my showcase app now has a second user in Australia, and now one in Warsaw, Poland.
Sending requests and replies half way around the world unnecessarily adds latency.
As each machine is monolithic and self-contained, replicating service into a new region is
a matter of creating the new machine and routing requests to the correct machine.
This application use three separate techniques to route requests.
The first technique is to [listen](https://github.com/rubys/showcase/blob/main/app/javascript/controllers/region_controller.js) for
[turbo:before-fetch-requests](https://turbo.hotwired.dev/reference/events#http-requests) events and
insert a [fly-prefer-region header](/networking/dynamic-request-routing#the-fly-prefer-region-header).
The region is extracted from a data attribute added to the [body](https://github.com/rubys/showcase/blob/aae08a6d57f92335b2cdbb94756e5416b7b50f83/app/views/layouts/application.html.erb#L16) tag.
This all made possible by how [Turbo works](https://turbo.hotwired.dev/handbook/introduction#turbo-drive%3A-navigate-within-a-persistent-process):
> This happens by intercepting all clicks on `` links to the same domain. When you click an eligible link, Turbo Drive prevents the browser from following it, changes the browser’s URL using the History API, requests the new page using fetch, and then renders the HTML response.
The same source above also defines a `window.inject_region` function to inject the header in other places in the code that may issue fetch requests.
The next (and final) two approaches are done in the NGINX configuration itself. This configuration is defined at startup on
each machine. Here is an example of the definition for `raleigh` on machines in regions other than `iad`:
```
## Raleigh
location /showcase/2024/raleigh/cable {
add_header Fly-Replay region=iad;
return 307;
}
location /showcase/2024/raleigh {
proxy_set_header X-Forwarded-Host $host;
proxy_pass http://iad.smooth.internal:3000/showcase/2024/raleigh;
}
```
This replays action cable (web socket) requests to the correct region, and
[reverse proxies](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/)
all other requests to the correct machine using its [`.internal` address](/networking/private-networking#fly-io-internal-dns). The latter is done because [fly-replay](/networking/dynamic-request-routing#requirements-and-limitations)
is limited to 1MB requests, and uploading audio files may exceed this limit.
## Appliances
> [!NOTE]
>
> For larger tasks, try
> decomposing parts of your applications into event handlers, starting up Machines to handle requests when needed, and stopping them when the requests are done.
While above I've sung the praises of a Majestic Monolith, there are limits. Usage of this application
is largely split into two phases: a lengthy period of data entry, followed by a brief period of report generation.
This pattern then repeats a second time as scores are entered and then distributed.
Report generation is done using [puppeteer](https://pptr.dev/) and [Google Chrome](https://www.google.com/chrome/).
While this app runs comfortably with 1MB of RAM, I've had Google Chrome crash machines that have 2MB.
Generating a PDF from a web page (including authentication) is something that can be run independently of
all of the other services of your application. By making use of [`auto_stop_machines` and `auto_start_machines`](/reference/configuration#the-http_service-section),
a machine dedicated to this task can be spun up in seconds and perform this function.
In my case, no changes to the application itself is needed, just a few lines of NGINX configuration:
```
## PDF generation
location ~ /showcase/.+\.pdf$ {
add_header Fly-Replay app=smooth-pdf;
return 307;
}
```
This was originally published as [Print on Demand](https://fly.io/blog/print-on-demand/),
and the code for that article can be found on [GitHub as fly-apps/pdf-appliance](https://github.com/fly-apps/pdf-appliance).
This architectural pattern can be applied whenever there is a minority of requests that require an outsized amount
of resources. A second example that comes to mind: I've had requests for audio capture and transcription.
## Backups
> [!NOTE]
>
> Running databases on a single volume attached to a single machine is a single
> point of failure. It is imperative that you have recent backups in case of failure.
At this point, databases are small, and can be moved between servers with a simple configuration change.
For that reason, for the moment I've elected to have each machine contain a complete copy of all of the databases.
To make that work, I
[install](https://github.com/rubys/showcase/blob/aae08a6d57f92335b2cdbb94756e5416b7b50f83/Dockerfile#L67),
[configure](https://github.com/rubys/showcase/blob/main/Dockerfile#L114-L137), and
[start](https://github.com/rubys/showcase/blob/aae08a6d57f92335b2cdbb94756e5416b7b50f83/bin/deploy#L26-L27)
[rsync](https://rsync.samba.org/).
With Passenger, I set [`passenger_min_instances`](https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_max_pool_size) to zero, allowing
each application to completely shutdown after five minutes of inactivity. I define a
[`detached_process`](https://www.phusionpassenger.com/library/indepth/hooks.html#detached_process) hook to run
a [script](https://github.com/rubys/showcase/blob/main/bin/passenger-hook).
As that hook will be run after each process exits, what it does first is query the number of running processes. If
there are none (not counting websockets), it begins the backup to each Fly.io machine
using `rsync --update` to only replace the newer files. After that completes, a webhook
is run on an offsite machine (a mac mini in my attic) to rsync all files there. In order for that to work, I run
[an ssh server](/blueprints/opensshd) on each Fly.io machine.
And I don't stop there:
* My home machine runs the same passenger application configured to serve all of the events.
This can be used as a hot backup. I've never needed it, but it comforting to have.
* I also rsync all of the data to a [Hetzner](https://www.hetzner.com/) which
runs the same application.
* Finally, I have a cron job on my home machine that makes a daily backup of all of the databases. A new directory
is created per day, and [hard links](https://en.wikipedia.org/wiki/Hard_link) are used to link to the same file
in the all too common case where a database hasn't changed during the course of the day. As this is in a terabyte
drive, I haven't bothered to prune, so at this point, I have daily backups going back years.
While this may seem like massive overkill, always having a local copy of all of the databases to use to reproduce problems
and test fixes, as well as having a complete instance that I can effectively use as a staging server alone makes all this
worth it.
Finally, since each region contains a copy of all databases, I use [`auto_extend_size_threshold`](/reference/configuration#auto_extend_size_threshold) and
[`auto_extend_size_increment`](/reference/configuration#auto_extend_size_increment) to ensure that I never run out of space.
## Logging
> [!NOTE]
>
> It is important that logs are there when you need them. For this reason, logs need both monitoring and redundancy.
Equally as important as backups are logs. Over the course of the life of this application I've had two cases where
I've lost data due to errors on my part and was able to reconstruct that data by re-applying POST requests extracted
from logs.
The biggest problem with logs is that you take them for granted. You see that they work and forget about them, and
there always is the possibility that they are not working when you need them.
I've written about my approach in [Multiple Logs for Resiliency](https://fly.io/blog/redundant-logs/), but a summary here:
* Since I have volumes already, I [instruct Rails to broadcast logs there](https://github.com/rubys/showcase/blob/aae08a6d57f92335b2cdbb94756e5416b7b50f83/config/environments/production.rb#L102-L110) in addition to stdout.
The API to do this changed in Rails 7.1, and the code I linked to does this both ways. This data isn't replicated to other volumes and
only contains Rails logs, as it really is only intended for when all else fails.
* I have a [logger](https://github.com/rubys/showcase/tree/main/fly/applications/logger) app that I run in two regions.
This writes log entries to volumes, and runs a small web server that I can use to browse and navigate the log entries.
This data too is backed up (forever) to my home server, and is only kept for seven days on the log server.
Two major additions since I wrote up that blog entry:
* I have each machine [create a heartbeat log entry every 30 minutes](https://github.com/rubys/showcase/blob/main/bin/log-heartbeat.sh), and each log server runs a
[monitor](https://github.com/rubys/showcase/blob/main/fly/applications/logger/monitor.ts) once an hour to
ensure that every machine has checked in. If a machine doesn't check in, a [Sentry](https://sentry.io/)
alert is generated.
* I have my log tracker query Sentry whenever I visit the web page to see if there are new events. This
serves as an additional reminder to go check for errors.
I strongly encourage people to look into [Sentry](https://sentry.io/).
## Applying updates
> [!NOTE]
>
> While Fly.io will start machines fast, you have to do your part.
> If your app doesn't start fast, start something else that starts quickly first
> so your users see something while they are waiting.
If you go to [https://smooth.fly.dev/](https://smooth.fly.dev/) the server nearest to you will need to spin up the index application. And if, from there, you click on demo, a second application will be started. While you might get lucky and somebody else (or a bot) may have already started these for you, the normal case is that you will have to wait.
Since all the services that are needed to support this application are on one machine, `fly deploy` can't simply start a new machine, wait for it to start, direct traffic to it, and only then start dismantling the previous machine. Instead, each machine needs to be taken offline, have the rootfs replaced with a new image, and all of its services restarted. Once that is complete, migrations need to be run on that machine; and in my case I may have several SQLite databases that need to be upgraded. I also have each machine that comes up rsync with the primary region. And finally, I generate an operational NGINX configuration and run [nginx reload](https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/).
While this overall process may take some time, NGINX starts fast so I start it first with a [configuration](https://github.com/rubys/showcase/blob/main/config/nginx.startup) that directs everything to a [Installing updates..](http://smooth.fly.dev/showcase/503.html) page that auto refreshes every 15 seconds which normally is ample time for all this to occur. Worst case, this page simply refreshes again until the server is ready.
Even with this in place, it still is worthwhile to make the application boot fast. Measuring the times of the various components of startup, in my case Rails startup time dominates, and it turns out that effectively `bin/rails db:prepare` starts Rails once for each database.
I found that I can avoid that by [getting a list of migrations](https://github.com/rubys/showcase/blob/cb3a05fd48c22a140effcf03ccfa9bd038a0df30/config/tenant/nginx-config.rb#L186),
and then [comparing that list to the list of migrations already deployed](https://github.com/rubys/showcase/blob/cb3a05fd48c22a140effcf03ccfa9bd038a0df30/config/tenant/nginx-config.rb#L207-L215).
I also moved preparation of my demo database to [build time](https://github.com/rubys/showcase/blob/cb3a05fd48c22a140effcf03ccfa9bd038a0df30/Dockerfile#L139-L140). In most cases, this means that my users
see no down time at all, just an approximately two second delay on their next request as Rails restarts.
## Administration
> [!NOTE]
>
> While Fly.io provides a [Machines API](/machines/api),
> you can go a long way with old-fashioned scripting using the [flyctl](/flyctl)
> command.
Now lets look at what happens when I create a new event, in a new region. I (a human) get an email, I bring up my admin UI (on an instance of the index app I mentioned above). I fill out a form for the new event. I go to a separate page and add a new region. I then go to a third form where I can review and apply the changes. This involves regenerating my map, and I need to update the NGINX configuration in every existing region. There are shortcuts I can utilize which will update files on each machine and reload the NGINX configuration, but I went that way I would need to ensure that the end result exactly matches the reproducible result that a subsequent `fly deploy` would produce. I don't need that headache, so I'm actually using `fly deploy` which can add tens of seconds to the actual deploy time.
I started out running the various commands manually, but that grew old fast. So eventually
I invested in creating an admin UI.
Here's what the front page of the admin UI looks like:
From there, I can add, remove, and reconfigure my events, and apply changes. I can also visit
this app on three different servers, and see the logs from each. Finally, I have links to handy
tools. The Sentry link that is current gray turns red when there are new events.
I'm actually very happy with the result. The fact that I could get an email and add an event and have everything up and running on a server half a world away in less than five minutes is frankly amazing to me.
And to be honest, most of that is waiting on the human (me), followed by the amount of time it takes to build and deploy a
new image in nine regions.
And in most cases, I can now make and deploy changes from my smartphone rather than requiring
my laptop.
For those who are curious, here's the [deployment script](https://github.com/rubys/showcase/blob/main/bin/apply-changes.rb).
## Costs
> [!NOTE]
>
> This is a decent sized personal project, deployed across three continents. Even with a
> significant amount of over-provisioning, the costs per month are around \$75, before taking
> into account plans and allowances.
I'm currently running this app in nine regions. I have two machines running a log server.
I have two machines that will run puppeteer and Chrome on demand.
[Fly.io pricing](/about/pricing) includes a number of plans which include a number of allowances. To
keep things simple, I'm going to ignore all of that and just look at the total costs.
From there, add your plan and subtract your allowances to see what this would cost for you.
And be aware that things may change, the below numbers are current as of October 2026.
Let's start with compute. 9 shared-cpu-1x machines with 1GB each at \$6.70 per month total \$60.30.
Two shared-cpu-1x machines with 512MB each at \$3.69 per month total \$7.38.
The print on demand machines are on bigger machines, but are only run when requested.
I've configured them to stay up for 15 minutes when summoned. Despite all of this the
costs for a typical month are around \$0.03. So we are up to \$67.71.
Next up volumes. I have 9 machines with 3GB disks, and 2 machines that only need 1G each.
29GB at \$0.15 per month is \$4.35.
I have a dedicated IPv4 address to use ssh. This adds \$2.00 per month.
Outbound transfer is well under the free allowance, but for completeness I'm seeing about 16GB in North America and Europe,
which at \$0.02 per GB amounts to \$0.32. And about 0.1GB in Asia Pacific, which at \$0.04 per GB adds less than a cent,
but to be generous, let's accept a total of \$0.33.
Adding up everything, this would be 67.71 + 4.35 + 2.00 + 0.33 for a total of \$74.39 per month.
Again, this is before factoring in plans and allowances, and reflects the current pricing in October 2026.
For my needs to date, I'm clearly over-provisioning, but it is nice to have a picture of what
a fully fleshed out system would cost per month to operate.
## Summary
> [!NOTE]
>
> Shared Nothing architectures are a great way to scale up applications where the data involved is easily partitionable.
Key points (none of these should be a surprise, they were highlighted
in the text above):
* Run as many services as you like on a single machine; this both increases throughput and reliability.
* You can even run multiple instances of the same app on a machine.
* Distributing your apps across multiple regions lets you service requests close to your users.
* For larger tasks, start up machines on demand.
* Every backup plan should have a backup plan.
* Not only should you have logs, logs should be monitored and have redundancy.
* Make sure that you start something quickly -- even if it is only a "please wait" placeholder -- to make sure that your app is always servicing requests.
* Create an administration UI and write scripts you can use to change your configuration.
And finally, this app currently supports 77 events in 36 cities with 9 servers for less than you probably pay for a dinner out a month or a smart phone plan.
## Background
> [!NOTE]
>
> Feel free to skip or skim this section. It describes how the application came to be, what it does, and a high level description of its usage needs.
Background information on how this application came to be, and got me involved with Fly.io can be found
at [Unretiring](http://intertwingly.net/blog/2022/08/13/Unretiring) on my personal blog.
Here's an excerpt from that post:
> I was at a local ballroom dance competition and found myself listed twice in one heat - a scheduling mishap where I was expected to be on the floor twice with two different dance partners.
>
> After the event, Carolyn and I discovered that the organizers used a spreadsheet to collect entries and schedule heats. A very manual and labor prone process, made more difficult by the need to react to last minute cancellations due to the ongoing pandemic.
>
> Carolyn told the organizers that I could do better, and volunteered me to write a program. I thought it would be fun so I agreed.
This application is [on GitHub](https://github.com/rubys/showcase#readme), and while each individual event requires authentication for access, a full function demo can be accessed at [https://smooth.fly.dev/](https://smooth.fly.dev/). It was originally deployed a single Mac mini, and now is running on a small fleet of machines in the Fly.io network.
***
Couples ballroom dance competitions are called a "Showcase".
The application supports:
* managing all the contestants
* storing and playing dance music for solos
* invoicing students and studios
* managing the schedule
* printing heat sheets, back numbers, and labels
* recording the judges' scores
* publishing results
As you might expect, a Showcase competition happens live and in one location. There are very few users who need access to the system and they often are all physically in the same location.
Here we'll cover how a Ruby on Rails application using a SQLite database can be deployed to the nearest Fly.io region for the event and how elegantly it solved this problem, including the ability to easily deploy new servers from the application.
***
The core Showcase application is implemented on top of Ruby on Rails, augmented by two separate services implemented in JavaScript an run using [Bun](https://bun.sh/). While the details may vary, many of the core principles described below apply to other applications,
even applications using other frameworks. In fact, many of the pieces of advice provided below apply to apps you host yourself, or host on
other cloud providers.
But first, some useful approximations that characterize this app, many of which will guide the choice of architecture:
* Each event is a separate instance of the Rails application, with a its own separate database.
* A typical database is one megabyte. Not a typo: one megabyte. Essentially, the app is a spreadsheet with a [Turbo](https://turbo.hotwired.dev/handbook/introduction) web front end.
* All the services needed to support an event are run on a single VM.
* Persistence consists of the database, storage (used for uploaded audio files), and logs.
* Typical response time for a request is 100ms or less.
* Peak transaction per second per database is approximately 1. Again, not a typo -- think spreadsheet.
* Peak number of simultaneous users is often one, but even when there are multiple they tend to be all in the same geographic region, often in the same room (example: multiple judges entering scores).
* Transactions are generally a blend of reads and writes with neither dominating.
This lends itself well to horizontal scaling, where every event effectively has a dedicated VM. Again, this is a useful approximation - as not every event is "active", multiple events can share a VM. As needs change, databases can be moved to different VMs.
## Related reading
* [Getting Started with N‑Tier Architecture](/blueprints/n-tier-architecture) A contrasting architecture pattern where state is shared across the data tier — useful for understanding when shared‑nothing *isn’t* the best fit.
* [Session Affinity (a.k.a. Sticky Sessions)](/blueprints/sticky-sessions) Focused on routing and state‑isolation patterns that often come up when you adopt a shared‑nothing approach and need to keep state local to a machine.
* [Networking](/networking) The comprehensive networking overview (private networks, Anycast, WireGuard tunnels, routing) — helpful when building globally distributed isolated nodes in a shared‐nothing architecture.
# Separate staging and production environments
Source: https://docs.fly.io/blueprints/staging-prod-isolation
Isolate production from staging on Fly.io by giving each environment its own organization, with separate members, apps, and fly.toml files.
## Overview
You want to isolate your production environment from your test or staging environments by limiting production access to the smallest possible group. You can get dependable isolation by using multiple organizations. Organization access is invitation-only and orgs are isolated from one another at a low level by our private networking architecture: every organization has its own [private network](/networking/private-networking) by default.
All Fly.io accounts start with a "personal" organization. You can create as many organizations as you need, not just for different environments, but also for different projects or clients.
Before you create a new organization, here's what you need to know:
* You can consolidate billing for multiple organizations under a single parent organization, or you can create regular organizations with separate payment methods and billing. Learn more about [unified billing](/about/billing#unified-billing).
* You invite or remove members in each organization separately.
* An organization can have multiple apps and those apps can communicate with each other securely over the org's private network.
Adjust the pattern to fit your needs. For example:
* personal organization: the org you started with, and the one you keep for personal projects
* staging organization: `-staging` used for development and testing
* production organization: `-production` used for your production app
## Work with multiple organizations
It's best if you use one Fly.io account to manage all your organizations, so you can access them without logging in and out. App names are unique across the Fly.io platform, so your staging and production apps will have different names and their own `fly.toml` files for configuration.
### Manage organizations in your dashboard
To create a new org from the [dashboard](https://fly.io/dashboard/), select **Create new organization** from the **Organization** dropdown.
To view or send invites to members, use the **Organization** dropdown to choose an org, then go to **Team**.
### Manage organizations with flyctl
To create new organizations and invite or remove members, use [`fly orgs` flyctl commands](/flyctl/cmd/fly_orgs).
When you have more than one org, flyctl prompts you to choose an organization when required. You can run commands on a specific app in any org using the `--app` option if you aren't in the app's project source directory in your terminal.
## Other kinds of isolation and access control
* When you only need app isolation within an org, you can use [custom private networks](/networking/custom-private-networks) to isolate apps from one another by creating an app with `fly apps create` and the `--network` option.
* When you want user or 3rd-party access control, you can use [deploy](https://community.fly.io/t/deploy-tokens/11895) and [org-scoped tokens](https://community.fly.io/t/org-scoped-tokens/13194) to limit access to apps or orgs.
## Related reading
* [Going to production checklist](/apps/going-to-production) A broad checklist for production‑readiness that includes isolating environments and access control.
* [Staging environments with GitHub actions](/django/advanced-guides/staging-environments-with-github-actions) A practical guide focused on using GitHub Actions to deploy dedicated staging environments. Complements the isolation pattern nicely.
# Sticky sessions (session affinity)
Source: https://docs.fly.io/blueprints/sticky-sessions
Send a user's requests to the same Fly Machine with the fly-force-instance-id request header from the client or the fly-replay response header from your server.
## Overview
In a number of scenarios, it is important to ensure that certain requests are routed to a specific Machine. This frequently is expressed in the form of wanting an entire user's session to be processed by the same Machine.
There are two approaches to addressing this with Fly.io:
* [fly-force-instance-id](#fly-force-instance-id) - Client-side request header
* [fly-replay](#fly-replay) - Server-side response header
## Fly-Force-Instance-Id
This approach requires you to have control over the client, typically a browser, but allows for immediate routing without an additional hop. The example below uses Rails' Hotwire [Turbo](https://turbo.hotwired.dev/) with [Stimulus](https://stimulus.hotwired.dev/) to send the required header.
For this to work, the client needs some way of knowing what Machine to route requests to. This can be accomplished by adding attributes to the `` tag in HTML responses. With Rails, those tags would be found in
`app/views/layouts/application.html.erb`.
An example `` tag:
```erb theme={null}
```
These attributes identify the Machine instance to direct future requests to, and the name of the Stimulus controller to be used to make this happen.
Place the following into `app/javascript/controllers/sticky-session.js`:
```js theme={null}
import { Controller } from "@hotwired/stimulus"
// Connects to data-controller="sticky-session"
export default class extends Controller {
connect() {
document.documentElement.addEventListener(
'turbo:before-fetch-request',
this.beforeFetchRequest
)
}
disconnect() {
document.documentElement.removeEventListener(
'turbo:before-fetch-request',
this.beforeFetchRequest
)
}
beforeFetchRequest = event => {
event.detail.fetchOptions.headers['fly-force-instance-id'] =
this.element.dataset.instance;
}
}
```
This code will subscribe and unsubscribe from
[turbo:before-fetch-request](https://turbo.hotwired.dev/reference/events#turbo%3Abefore-fetch-request) events and insert a `fly-force-instance-id` header into requests.
## Fly-Replay
This approach is implemented entirely on the server. Your application examines each request, determines which Machine should handle it, and returns a `fly-replay` response header to route the request accordingly. Initial requests will require an additional "hop" to route, and [requests are limited to payloads of 1 megabyte](/networking/dynamic-request-routing#requirements-and-limitations).
The example below creates [Express Middleware](https://expressjs.com/en/guide/using-middleware.html) that routes requests based on a session cookie:
```js theme={null}
// Import required modules
import express from "express";
import cookieParser from "cookie-parser";
// Create an Express app
const app = express();
// Middleware to parse cookies
app.use(cookieParser());
app.use((request, response, next) => {
// This assumes your application has already established a session using a session_id cookie via your session middleware.
const sessionId = request.cookies["session_id"];
if (!sessionId) {
// No session - let any machine handle it
next();
return;
}
// Determine which machine should handle this session
// This could be based on:
// - Database lookup (session -> machine mapping)
// - Consistent hashing of session ID
// - Regional routing logic
// - Load balancing algorithm
// You'll need to implement determineTargetMachine() based on your routing strategy
const targetMachineId = determineTargetMachine(sessionId);
if (targetMachineId === process.env.FLY_MACHINE_ID) {
// This is the correct machine - handle the request
next();
} else {
// Route to the correct machine
response.set('Fly-Replay', `instance=${targetMachineId}`);
response.status(307);
response.send();
}
});
// Start the Express server
const port = 3000;
app.listen(port, () => {
console.log(`Server is running on port ${port}`);
});
```
### Optimizing with Replay Caching
For production workloads, having your application replay every request can create unnecessary load and latency. You can configure Fly Proxy to cache replay decisions, so only the first request in a session needs to consult your application.
Add replay cache rules to your `fly.toml`:
```toml theme={null}
[http_service]
internal_port = 8080
force_https = true
[[http_service.http_options.replay_cache]]
path_prefix = "/"
ttl_seconds = 300
type = "cookie"
name = "session_id"
```
With this configuration:
1. The first request with a given `session_id` hits your app
2. Your app returns a `fly-replay` response
3. Fly Proxy caches this decision for that specific `session_id` value
4. For the next 5 minutes, requests with the same `session_id` are automatically routed without consulting your app
Caching is an optimization, not a guarantee, so your origin app still needs to handle replaying requests that aren't routed by a cached replay.
Replays can be cached on any existing cookie or request header. For sticky sessions with an API, for example, you might choose to cache based on the `Authorization` header.
For complete details on replay caching configuration, see [Session-based Replay Caching](/networking/dynamic-request-routing#session-based-replay-caching).
## Related reading
* [Dynamic Request Routing with `fly‑replay`](/networking/dynamic-request-routing) Explains how `fly‑replay` lets you route requests to a specific machine or region—core technique for sticky sessions.
* [Load Balancing](/reference/load-balancing) Details how the Fly Proxy routes traffic among Machines, which underpins how session‑affinity decisions get made.
* [Networking](/networking) A broad overview of how Fly handles public/private networking, Anycast IPs, WireGuard mesh, domain routing, and more.
# Crontab with Supercronic
Source: https://docs.fly.io/blueprints/supercronic
Run cron jobs inside your app container on Fly.io with Supercronic: add a crontab, install it in your Dockerfile, and scale a single cron process group.
## Overview
`crontab` is a little too opinionated for containers—it's a great little tool, but when run inside of containers it doesn't grab `ENV` vars how we'd like it to. Fortunately there's a Go binary called `supercronic` that's a drop-in replacement for containers.
The good news is that it's pretty easy to get it running on Fly with a little bit of copy and pasting. Let's get to it.
## Create a crontab file
In the root of your project, add a `crontab` file.
```
touch ./crontab
```
If you need to run a job every 5 minutes, your `crontab` file would something like this:
```
*/5 * * * * echo "hello world!"
```
Check out [cron.help](https://cron.help) if you need a quick crontab syntax reference.
## Install `supercronic` in the container
The [latest releases for supercronic](https://github.com/aptible/supercronic/releases) are on [GitHub](https://github.com/aptible/supercronic), where they include copy pasta 🍝instructions for getting it in your Dockerfile. The example below uses v0.2.29. Check the [releases page](https://github.com/aptible/supercronic/releases) for the latest version and its checksum, and update `SUPERCRONIC_URL` and `SUPERCRONIC_SHA1SUM` to match:
```
# Latest releases available at https://github.com/aptible/supercronic/releases
ENV SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.29/supercronic-linux-amd64 \
SUPERCRONIC=supercronic-linux-amd64 \
SUPERCRONIC_SHA1SUM=cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b
RUN curl -fsSLO "$SUPERCRONIC_URL" \
&& echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - \
&& chmod +x "$SUPERCRONIC" \
&& mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" \
&& ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic
# You might need to change this depending on where your crontab is located
COPY crontab crontab
```
After you put that in your Dockerfile, we're going to configure Fly to run only one instance of `supercronic`. Why? Because if you have a cronjob that does something like deliver emails to customers, you only want them to get that once. The last thing you want is them getting as many emails from your services that matches the number of instances you're running.
## Setup a web & cron process
At the bottom of the `fly.toml` file, add the following:
```
[processes]
# The command below is used to launch a Rails server; be sure to
# replace with the command you're using to launch your server.
web = "bin/rails fly:server"
cron = "supercronic /app/crontab"
```
Then we have to tell Fly that your `web` process matches up with a service by having this under the `[[services]].`
```
# Make sure there's only one `processes` entry under `[[services]]`
[[services]]
processes = ["web"]
```
This change tells Fly that your `web` processes that you defined under `[processes]` will be exposed to the public internet. Your `cron` process won't be exposed to the public Internet because it doesn't need to!
## Deploy & scale the new processes
Now that we've added `supercronic` to our `Dockerfile`, put the `crontab` at the root of our project folder, and reconfigured the `fly.toml` file, we're ready to deploy to production.
```
$ fly deploy
```
Then we'll need to scale the processes so that we only run one virtual machine container with the `cron` process. Be sure to change `web` to whatever number you had before.
```
$ fly scale count cron=1 web=
```
That's it! If all went well you now have cron running in a `cron` process in a Fly virtual machine. When you `fly deploy` it will get the latest code changes and reboot the virtual machines.
## Resources
* [https://github.com/fly-apps/supercronic](https://github.com/fly-apps/supercronic)
* [https://cron.help](https://cron.help)
## Related Reading
* [Task scheduling guide with Cron Manager and friends](/blueprints/task-scheduling) A broader look at scheduling jobs on Fly.io — covers Cron Manager, Supercronic, scheduled Machines, and more.
* [Run multiple process groups in an app](/launch/processes) Explains how to define processes in `fly.toml`, run multiple programs (web, cron, workers), scale by group.
* [App Configuration (`fly.toml`)](/reference/configuration) In‑depth look at the config file that orchestrates your services, processes, health checks, etc.
# Task scheduling guide with Cron Manager and friends
Source: https://docs.fly.io/blueprints/task-scheduling
Compare ways to run scheduled jobs on Fly.io, from Cron Manager's one-off Machines to Supercronic, scheduled Machines, and in-app schedulers.
## Overview
Running on a schedule: it's not just for trains. If your app needs to rebuild a cache every night, prune old data, or email a weekly newsletter to your users, you’re probably looking for a way to run a cron job. Fly.io gives you a few ways to get that done.
We’ll walk through options for task scheduling, starting with the most robust and ending with the “it works, okay?” tier.
## Cron Manager
[Cron Manager](https://github.com/fly-apps/cron-manager) is our “batteries-included” solution for running scheduled jobs. It’s a small Fly app that spins up temporary Machines—one per job—and tears them down afterward. Think of it as a very polite butler that rings a bell and quietly disappears before you remember you don’t have a butler.
### How It Works
You deploy Cron Manager as its own Fly app. It stays online, watching a `schedules.json` file. When a job’s scheduled time hits, it boots a one-off Machine with whatever image, region, and resources you specify. That Machine runs the command, exits, and disappears like a good container should.
Jobs run in complete isolation—no shared environment, no weird state carried over from the last job. And because they’re just Fly Machines, you get all the perks: region placement, resource control, image selection, and clean logs.
### Why Use This
* **Isolation**: Each job gets its own Machine. Nothing leaks.
* **Central config**: `schedules.json` lives in Git. Changes are versioned, auditable, and deployable.
* **Easy updates**: Want to change a command or schedule? Update the file and redeploy.
* **Job logs**: Each Machine has its own logs and lifecycle. Debugging isn’t a treasure hunt.
### Setup
1. Clone [cron-manager](https://github.com/fly-apps/cron-manager)
2. Create a new Fly app and deploy it with `fly deploy`
3. Set your `FLY_API_TOKEN` as a secret for the app
4. Define your job schedules in `schedules.json`
Here’s a sample job:
```json theme={null}
{
"name": "daily-cache-rebuild",
"app_name": "my-app",
"schedule": "0 3 * * *",
"region": "lhr",
"command": ["bin/rebuild-cache"],
"command_timeout": 300,
"enabled": true,
"config": {
"image": "my-app-image",
"guest": { "cpu_kind": "shared", "cpus": 1, "memory_mb": 256 },
"restart": { "policy": "no" }
}
}
```
### Managing Jobs
Once deployed, you can:
* List jobs: `cm schedules list`
* Check history: `cm jobs list `
* Peek at details: `cm jobs show `
* Trigger one manually: `cm jobs trigger `
SSH into the Cron Manager VM if you need to do some spelunking.
***
## Supercronic
Sometimes you don’t need a full orchestration layer—you just want a crontab to run inside your app container. Supercronic does that. It’s a container-friendly cron runner that handles environment variables correctly.
### How It Works
[Supercronic](/blueprints/supercronic) runs as a background process inside your app. You add a `crontab` file, install Supercronic in your Dockerfile, and define a `cron` process in `fly.toml`. Done.
Well, almost done—remember to scale it correctly.
### Setup
1. Add a crontab file (`/app/crontab`) using standard syntax
2. Install Supercronic in your Dockerfile
3. In `fly.toml`:
```
[processes]
app = "bin/server"
cron = "supercronic /app/crontab"
```
1. Scale it:
```bash theme={null}
fly scale count app=2 cron=1
```
Only run one copy of the cron process. Unless your goal is to send four copies of the same reminder email. (In which case, carry on.)
***
## Scheduled Machines
Fly Machines support basic scheduling out of the box: you can tell a Machine to start hourly, daily, weekly, or monthly. The API is simple. The downside? You don’t get fine-grained control—just interval buckets.
This is fine for non-critical cleanup tasks or anything where “roughly once a day” is good enough.
***
## In-App Scheduling
If you’re already using a scheduler like `apscheduler` in a Python app—or you’ve duct-taped one together in Node—you can use it to call `fly machine run --rm` and spin up ephemeral Machines manually.
This gives you full control and is technically elegant, though it does require some careful footwork. It’s the “build your own trigger system” option, best for when you’re already halfway there.
***
## Additional Options
In addition to the options above, you can use an external scheduling service, like [GitHub Actions schedule](https://docs.github.com/en/actions/reference/events-that-trigger-workflows#schedule), or [easycron](https://www.easycron.com/), an online cron service which can send out API requests to wake machines up.
***
## Summary
| Option | Use When… | Good To Know |
| - | - | - |
| **Cron Manager** | You want isolated, auditable job runs | Requires a separate app |
| **Supercronic** | You want quick cron jobs inside your container | Keep the process count to one |
| **Scheduled Machines** | You want simple, low-frequency jobs (\~1x/day) | Not for precise timing |
| **In-App Scheduler** | You want full control and don't mind plumbing | Write your own orchestration |
If you're not sure where to start, Cron Manager is the most production-hardened option. But Supercronic gets you 80% of the way with almost no setup, and sometimes that’s all you need.
***
## Related Reading
* [Cron Manager](https://github.com/fly-apps/cron-manager)
* [Crontab with Supercronic](/blueprints/supercronic)
* [Deferring long-running tasks to a work queue](/blueprints/work-queues)
# Use Docker base images for faster deploys
Source: https://docs.fly.io/blueprints/using-base-images-for-faster-deployments
Build a shared base image as its own Fly.io app, push it to the Fly registry, and use it in FROM so each deploy rebuilds only your app, not every dependency.
## Overview
This guide explains how to use base images for faster deployments. A base image is any image that is intended to be used as the `FROM` line in a Dockerfile. Each app that is deployed on Fly.io can add an image to the Fly registry.
Every app in the same organization can access the image for any other app and use it as the `FROM` line in their Dockerfile. This means that it is very easy to make a base image by making a second app to use as the base image.
## Why use a base image?
Most developers have a specific reason for using a base image in their project. Usually it's one of the following:
* The base image can be built with all of their app's dependencies. Rebuilding the app on top of this base image means the deploy process is only running the app-specific build processes, not the entire dependency tree.
* The same app is deployed for a number of different end users. Each end user has some specific files or configurations to build into the image. A base image can get *most of the way* to the final configuration, and their app can do the final steps.
## How to make a base image
This guide walks through building and deploying an example application called go-fly-a-site, which runs a simple Go web server. After deploying the app, you'll create a base image from its non-app-specific parts. Finally, you'll update the app to use that base image.
### Making the app
Our program will be very simple. It will respond to HTTP requests on port 8080 and respond with "Hello World!".
```GO theme={null}
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", HelloServer)
http.ListenAndServe(":8080", nil)
}
func HelloServer(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "Hello, world!\r\n")
}
```
Create a Dockerfile based on Debian 12. Install curl and go, compile the Go code, and configure the image to run the compiled binary.
```Dockerfile theme={null}
FROM debian:12
# Install Go
RUN apt-get update; apt-get upgrade; apt-get -y install golang
# Copy the code for our server to /opt/main.go
COPY main.go /opt/main.go
# Build /opt/server from the code in /opt/main.go
RUN go build -o /opt/server /opt/main.go
# Run our code from main.go as /opt/server
CMD [ "/opt/server" ]
```
With `main.go` and the `Dockerfile` in your project directory, deploy the application:
```bash theme={null}
$ fly launch --name go-fly-a-site --vm-size shared-cpu-1x --no-deploy
....
Your app is ready! Deploy with `flyctl deploy`
```
Now we have an additional file, `fly.toml` and have created the app `go-fly-a-site`.
Deploy the app:
```bash theme={null}
$ fly deploy
==> Verifying app config
Validating /Users/fly/Code/go-fly-a-site/fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
==> Building image with Depot
--> build:
...
Visit your newly deployed app at https://go-fly-a-site.fly.dev/
```
We can see it's running:
```bash theme={null}
$ curl https://go-fly-a-site.fly.dev/
Hello, world!
```
Now our app is built and running. This is great, but each time we `fly deploy` we might be reinstalling Go through apt. We probably want to do that much less often than we want to update our `main.go`, so we can make a base image that installs go for us.
### Make a base image
Let's make a new file called `base.Dockerfile`. For this file, we will keep the Go install. We will replace the previous `CMD` statement with `[ "sleep", "inf" ]` so that we can easily SSH into this image to inspect it if needed in the future.
```Dockerfile theme={null}
FROM debian:12
# Install Go
RUN apt-get update; apt-get upgrade; apt-get -y install golang
CMD [ "sleep", "inf" ]
```
Copy `fly.toml` to `fly-base.toml` and make the following modification:
* Append -base to the app name.
* Explicitly include dockerfile in the `[build]` section.
* Remove the `[http_service]` section.
The resulting file looks like this:
```toml theme={null}
app = 'go-fly-a-site-base'
primary_region = 'lax'
[build]
dockerfile = "base.Dockerfile"
[[vm]]
size = 'shared-cpu-1x'
```
Create the app using the following options:
* `--ha=false` ensures the app uses a single Machine.
* `--config base.fly.toml` specifies the correct config file.
When prompted about the existing `fly.toml` file, select yes to copy its configuration. You can skip tweaking the settings.
```bash theme={null}
$ fly launch --ha=false --config base.fly.toml
An existing fly.toml file was found for app go-fly-a-site-base
? Would you like to copy its configuration to the new app? Yes
Using build strategies '[the "base.Dockerfile" dockerfile]'. Remove [build] from fly.toml to force a rescan
Creating app in /Users/fly/Code/go-fly-a-site
We're about to launch your app on Fly.io. Here's what you're getting:
...
Region: Los Angeles, California (US) (from your fly.toml)
App Machines: shared-cpu-1x, 256MB RAM (from your fly.toml)
...
? Do you want to tweak these settings before proceeding? (y/N)
Created app 'go-fly-a-site-base' in organization 'personal'
Admin URL: https://fly.io/apps/go-fly-a-site-base
Hostname: go-fly-a-site-base.fly.dev
Wrote config file base.fly.toml
...
--> build:
--> Building image done
image: registry.fly.io/go-fly-a-site-base:deployment-01JTKF7JXJ1ZGMA76GCJ5WGZQZ
image size: 233 MB
```
Some things to note:
* We have the base image name we can use, it's the URL provided in `image:`.
* When we go to the admin URL, we can click **Registry** and see the same image URL.
* The **Registry** will have the list of images that have been created.
* Any app in your organization can use the image URL for a `FROM` statement in a Dockerfile.
* Apps in any other organization will get an error if they try to use this image.
### Updating the app to use the base image
After creating the base image, update the Dockerfile for go-fly-a-site:
```Dockerfile theme={null}
FROM registry.fly.io/go-fly-a-site-base:deployment-01JTKF7JXJ1ZGMA76GCJ5WGZQZ
# Copy the code for our server to /opt/main.go
COPY main.go /opt/main.go
# Build /opt/server from the code in /opt/main.go
RUN go build -o /opt/server /opt/main.go
# Run our code from main.go as /opt/server
CMD [ "/opt/server" ]
```
With the updated Dockerfile in place, deploy the app with `fly deploy`.
```bash theme={null}
$ fly deploy
==> Verifying app config
Validating /Users/fly/Code/go-fly-a-site/fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
==> Building image with Depot
--> build:
[+] Building 20.7s (8/8) FINISHED
...
=> [internal] load metadata for registry.fly.io go-fly-a-site-base:deployment-01JTKF7JXJ1ZGMA76GCJ5WGZQZ
...
```
We can see that it used the new base image, and the application is running:
```bash theme={null}
$ curl https://go-fly-a-site.fly.dev/
Hello, world!
```
We can shutdown the one Machine for the base image:
```
$ fly machine list --config base.fly.toml
1 machines have been retrieved from app go-fly-a-site-base.
View them in the UI here
go-fly-a-site-base
ID NAME ...
d890175f6940e8 wandering-wave-6179 ...
```
Grab the Machine ID and then run `fly machine stop` with the app and Machine id:
```bash theme={null}
$ fly machine stop d890175f6940e8 --config base.fly.toml
Sending kill signal to machine d890175f6940e8...
d890175f6940e8 has been successfully stopped
```
You’ll only be charged for the root file system storage of the image that's being stored.
Now we have a base image for our project! We can modify and deploy our `main.go` code without having to reinstall Go. If we ever need to add additional dependencies to the base image, we just modify the `base.Dockerfile`, deploy it again, and get the new `image:` from the deploy command or in the **Registry** section of the app's dashboard.
## Command Reference
Get Machines for the base image: `fly machine list --config base.fly.toml`
Start Machine for the base image: `fly machine start --config base.fly.toml`
Stop Machine for the base image: `fly machine stop --config base.fly.toml`
Get SSH access to the base image: `fly ssh console --config base.fly.toml` (Make sure the Machine is started)
Rebuild the base image: `fly deploy --config base.fly.toml` (Update `base.Dockerfile` with your changes first, and remember to stop the Machine when you're done)
## Additional Notes
If you have a base image that many different apps will use, it can make sense to keep it all in its own app directory. You can use `fly.toml` and `Dockerfile` like a normal app.
The base image will stick around in the registry and as long as an image is associated with a Machine -- even if the Machine is stopped -- it won't be deleted.
You can find the registry URL of the images for your base image in the dashboard under the **Registry** tab for the app, or by running `fly releases --image`
## Related use case
Want to skip the Fly builder and use Docker tools directly to manage your images? See [Managing Docker Images with Fly.io's Private Registry](/blueprints/using-the-fly-docker-registry), a guide that walks through building, pushing, deploying, and managing images via Fly.io’s Docker registry!
## Related reading
* [Deploy with a Dockerfile](/languages-and-frameworks/dockerfile) The general guide on how to deploy apps from Dockerfiles — useful context when you’re building a base image.
* [App Configuration (`fly.toml`)](/reference/configuration) Details on how your `fly.toml` orchestrates builds, images, and deployments.
# Managing Docker Images with Fly.io's Private Registry
Source: https://docs.fly.io/blueprints/using-the-fly-docker-registry
Tag and push Docker images to registry.fly.io, deploy them with fly deploy --image, and reuse images across apps in your Fly.io organization without fly launch.
## Overview
Fly.io lets you deploy Docker containers as lightweight Firecracker VMs, running globally. While `fly launch` supports many frameworks and auto-generates deployment configurations, you can also build and push your own Docker images to Fly.io’s private registry and deploy them manually.
This guide walks through building, pushing, deploying, and managing images via Fly.io’s Docker registry.
## Tagging images for the Fly.io registry
Each Fly.io app gets a dedicated registry path in the form of:
```
registry.fly.io/
```
To create a new app:
```sh theme={null}
fly apps create
```
App names must be globally unique. You can generate a unique one automatically:
```sh theme={null}
fly apps create --generate-name
```
Then tag your image for the app's registry:
```sh theme={null}
docker build -t registry.fly.io/: .
```
## Authenticating and pushing to the registry
Authenticate Docker to use the Fly.io registry:
```sh theme={null}
fly auth docker
```
This command updates your `~/.docker/config.json` with the required auth token for `registry.fly.io`. Once authenticated, push your image:
```sh theme={null}
docker push registry.fly.io/:
```
## Deploying the image
You can deploy images from the Fly.io registry or public registries, depending on where the image is hosted.
### To deploy a private image pushed to Fly.io's registry:
Use the `--image` option with the `fly deploy` command:
```sh theme={null}
fly deploy --app --image registry.fly.io/:
```
### To deploy a public image from Docker Hub or another public registry:
Specify the image in your `fly.toml` file:
```toml theme={null}
[build]
image = "flyio/hellofly:latest"
```
## Reusing images across apps
Although registry URLs are structured per app, access is scoped per organization. This means you can push an image to `registry.fly.io/app-1` and use it in another app (e.g. `app-2`) if both apps are in the same Fly.io organization:
```sh theme={null}
fly deploy --app app-2 --image registry.fly.io/app-1:tag
```
This pattern is useful for SaaS platforms that build once and deploy many times.
## Verifying image existence
You can verify that an image exists in the Fly registry without deploying it using Docker:
```sh theme={null}
docker manifest inspect registry.fly.io/my-app-name:my-tag
```
This command returns data if the image exists, or exits with code 1 if it does not. It requires prior authentication using `fly auth docker`.
For most workflows, use the `fly deploy --image` option with the Fly Docker registry. For advanced flows like multi-app SaaS provisioning, track image tags using `docker manifest inspect` or associate them with Machines to retain them.
## Listing image tags
There is no API or GraphQL query to list all image tags in the registry. Tags only become visible to Fly when they are part of a release. You can view deployed image references with:
```sh theme={null}
fly releases -a my-app-name --image
```
If you're using the GraphQL API to query releases, use the following query format:
```json theme={null}
{
"query": "query($appName: String!, $limit: Int!) { app(name: $appName) { releases: releasesUnprocessed(first: $limit) { nodes { id version description reason status imageRef stable user { id email name } createdAt } } } }",
"variables": {
"appName": "my-app-name",
"limit": 25
}
}
```
This will only return images used in past deploys.
## Related use case
Want to optimize your deploys with shared layers and pre-installed dependencies? See the [Using base images for faster deployments](/blueprints/using-base-images-for-faster-deployments#how-to-make-a-base-image) blueprint.
## Related reading
* [Deploy with a Dockerfile](/languages-and-frameworks/dockerfile) The general guide on how to deploy apps from Dockerfiles — useful context when you’re building a base image.
* [App Configuration (`fly.toml`)](/reference/configuration) Details on how your `fly.toml` orchestrates builds, images, and deployments.
# Using Fly Volume forks for faster startup times
Source: https://docs.fly.io/blueprints/volume-forking
Preload ML models, SQLite databases, or other large files onto a Fly Volume once, then fork it so new Machines boot with the data already in place.
> [!NOTE]
>
> **This guide shows how to preload large files onto volumes using forking to improve startup performance. This is especially useful for apps with big model files, binaries, or databases.**
## Overview
If your app needs large local files like ML models, SQLite databases, or game assets, you're probably copying them into place at boot. Maybe you bake them into your image, or have your app download them on first run. This works, but it has downsides:
* **Slow startup times**: waiting for multi-GB file downloads.
* **Wasted bandwidth**: each new machine pulls the same data.
* **Image bloat**: large files increase your OCI/Docker image size.
There's a better way. Write those files to a Fly Volume so they *persist* after they've been downloaded. Fly Volumes support [forking](/volumes/volume-manage/#create-a-copy-of-a-volume-fork-a-volume), a fast, storage-efficient way to create new volumes preloaded with data, skipping downloads and startup delays.
***
## Why use this pattern?
You’re cloning machines and want them to boot fast, with all their large files already in place.
This isn’t about long-term persistence (though volumes can be used for that, too). It’s about getting cold-start performance close to warm-start, by skipping the “download half the internet” phase of your boot process.
It’s especially useful when:
* You’ve already run a setup step once and don’t want to repeat it.
* Your app reads large files but doesn’t modify them.
* You're scaling up on demand and startup time matters.
***
## How volume forking works
When you fork a volume:
* **Data is copied from the source volume**, block-for-block.
* **Forked volumes are usable immediately**, even while they are in the hydrating state, thanks to lazy fetching (a system where data blocks are only fetched from the source volume when your app actually tries to read them).
Your app can start reading files as soon as the fork succeeds. We start background replication right away, but behind the scenes we use a network block device to fetch blocks on demand. So startup is fast, and the data just shows up when your app asks for it.
One tradeoff: disk access might be a little slower while the replication is ongoing. But for many apps, this beats the startup delay of downloading or unpacking files.
***
## Example: cloning a machine with preloaded data
### 1. Make sure the source volume you want to fork is available
```
fly volumes list -a your-app
```
Pick the volume with the data you want to reuse.
### 2. Fork the volume
```
fly volumes fork vol_abc123 -a your-app
```
You’ll get output like this:
```
ID: vol_new_id
Name: volume_name
App: your-app
Region: yul
Zone: b5a1
Size GB: 20
Encrypted: true
Created at: 05 Sep 25 21:52 UTC
```
Note the `vol_new_id`.
### 3. Clone the machine and attach the forked volume
```
fly machine clone --attach-volume vol_new_id
```
Now your new machine starts with the same data in place—no need for your app to download or generate it again.
***
## What to know
* **Forks aren’t instant behind the scenes.** You can read from them immediately, but the full copy happens in the background. Disk I/O might be slower until replication finishes.
* **Data is a snapshot, not a live link.** Changes to the original after forking don’t show up in the copy (and vice versa).
* **You can fork cross‑region.** The `--region` flag allows you to target another region for the fork. If you skip it, the volume ends up in the same region by default. Cross-region forks are expected to hydrate more slowly, since all data must be replicated over the network between regions.
* **Forks have storage costs.** As blocks are fetched or written, forked volumes grow. Keep an eye on storage usage costs if you fork a lot.
***
## (Optional) Automate it in CI
If you're scripting deploys or scaling machines dynamically, you can fold this pattern into a CI job or automation script.
Here’s a rough sketch:
```
# Step 1: Fork the seed volume
FORKED_VOL=$(fly volumes fork vol_abc123 -a your-app | grep ID | awk '{ print $2 }')
# Step 2: Clone the machine and attach the fork
fly machine clone --attach-volume $FORKED_VOL
```
This lets you create and attach preloaded volumes as part of your rollout process which is useful if you're standing up a new VM per job, or if you're scaling inference machines on demand.
This is totally optional. Most folks will do this manually, or only once per release. But if you're already scripting `fly machine` operations, this fits right in.
## Related reading
* [Fly Volumes overview](/volumes/overview)
* [Create and manage volumes](/volumes/volume-manage)
* [Volume states](/volumes/volume-states)
* `fly volumes fork` [reference](/flyctl/cmd/fly_volumes_fork)
# Warm pools of user Machines
Source: https://docs.fly.io/blueprints/warm-pool-user-machines
Make per-user apps start instantly: keep a pool of pre-provisioned Fly apps, volumes, and booted Machines, then claim one and configure it when a user signs up.
This page covers how to make per-user infrastructure feel instant: pre-provision a pool of generic Fly Machines in the background, then claim one and personalize it when a user shows up. It's the "pre-create a pool of these" advice from [Per-User Dev Environments](/blueprints/per-user-dev-environments), worked out in full: the database schema, the API calls, the claim query, and the maintenance loop.
## The problem
If you run per-user apps (dev environments, AI agents, code sandboxes), onboarding a user means creating a Fly app, a [volume](/volumes), and a Machine, then waiting for the software inside to boot. Each step is fast, but they add up: a minute or more before the user has something to talk to. Users expect seconds.
Most of that work is generic. The app, the volume, the Machine, the boot: none of it depends on who the user is. Only the last step, configuration, is user-specific. So do the generic work ahead of time. A background worker keeps N instances provisioned, booted, and health-checked. When a user signs up, you claim one from the pool, write their config into it, and hand it over. Allocation drops to about a second. If the pool is empty, you fall back to provisioning directly. That's slower, but nothing breaks.
## One app per pool entry
Each pool entry is a dedicated Fly app, not just a Machine in a shared app. [The app is your isolation boundary](/machines/guides-examples/one-app-per-user-why): creating it with its own network name puts each entry on its own [private 6PN network](/networking/private-networking), so one user's Machine can never reach another's. A [Flycast address](/networking/flycast) gives your control plane private routing to the Machine without a public IP. And deleting the app cascades to its Machines and volumes, so teardown is a single API call, which matters more than it sounds, because crash-recovery cleanup uses the same call.
This control plane creates and destroys apps continuously, so it needs an org-scoped API token (`fly tokens create org`), not an app-scoped deploy token. Keep it server-side and never expose it to a user Machine.
```bash theme={null}
# Create the app on its own private network
curl -X POST "https://api.machines.dev/v1/apps" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"app_name": "pool-x7f3k2",
"org_slug": "my-org",
"network": "pool-x7f3k2"
}'
# Allocate a Flycast (private) address into the app's own network
curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/ip_assignments" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"type": "private_v6", "network": "pool-x7f3k2"}'
```
Pass the same `network` name you created the app with. The `network` field on the allocation targets which 6PN network the Flycast address routes within, and it does not default to the app's own network. Leave it empty and the address lands in your organization's default network, where it cannot reach a Machine that lives in the app's isolated per-app network.
App creation returns `422` if the name is taken. Your worker will retry after crashes, so treat that as idempotent: `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the `422` was a real validation error, so rethrow it.
## Track pool state in a database
The Machines API is your source of truth for infrastructure. A small table is your source of truth for who owns what. One row per pool entry:
| column | purpose |
| - | - |
| `id` | your identifier |
| `app_name` | the entry's dedicated Fly app |
| `machine_id` | written as soon as the Machine is created |
| `volume_id` | written as soon as the volume is created |
| `region` | the concrete region Fly actually placed it in |
| `status` | `provisioning` → `ready` → `failed` |
| `provisioned_at` | set when the entry becomes `ready` |
| `allocated_at` | `NULL` = available; set = claimed |
| `allocated_to` | the user or instance that claimed it |
| `provision_version` | bump to invalidate the pool when your image changes |
Two rules make this crash-safe. First, insert the row *before* touching the Machines API, so nothing exists on Fly.io that your database can't see. Second, write resource IDs back incrementally: the volume ID right after volume creation, the Machine ID right after Machine creation. If the worker dies mid-provision, a sweeper can find the half-finished row, delete the app (cascading whatever got created), and move on.
## Provision in the background
The sequence for each entry: create app → allocate Flycast IP → create volume → create Machine → wait for boot → health-check → mark `ready`.
```bash theme={null}
# Create the volume first; Fly resolves the region
curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/volumes" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"name": "user_data", "region": "iad", "size_gb": 1}'
# Create the Machine in the *volume's* region
curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"name": "pool",
"region": "iad",
"config": {
"image": "registry.fly.io/my-user-app@sha256:...",
"guest": {"cpu_kind": "shared", "cpus": 1, "memory_mb": 2048},
"init": {"swap_size_mb": 2048},
"mounts": [{"volume": "vol_abc123", "path": "/data"}],
"services": [{
"internal_port": 8080,
"ports": [{"port": 80, "handlers": ["http"]}],
"autostart": true,
"autostop": "off"
}],
"checks": {
"health": {"type": "tcp", "port": 8080, "interval": "15s",
"timeout": "5s", "grace_period": "120s"}
},
"restart": {"policy": "on-failure", "max_retries": 3},
"metadata": {"pool_entry": "fp_abc123"}
}
}'
# Wait for the VM to boot (timeout defaults to 60s, which is also the max)
curl "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/wait?state=started&timeout=60" \
-H "Authorization: Bearer ${FLY_API_TOKEN}"
```
A few settings in that config matter more than they look.
**Volume and Machine must co-locate.** A volume is pinned to one physical host, and the Machine that mounts it has to land on that same host. Create the volume first, then create the Machine with that volume ID in `config.mounts` and use the volume's returned region. The mount is what places the Machine on the volume's host; matching the region alone isn't enough, and creating the Machine first can fail to attach even in the right region. Store the Machine's returned region in your database too, since that's where it actually landed.
**Pin the image by digest, not a tag.** `provision_version` is meant to mark a uniform pool generation. If you build on a moving tag like `:latest`, two entries with the same version can end up running different images. Reference an immutable digest (`@sha256:...`) and bump `provision_version` whenever it changes.
**`autostop: "off"` is the point.** Pool Machines are idle by design. [Autostop](/launch/autostop-autostart) would helpfully shut them down, and a stopped pool Machine is a cold start with extra steps.
**`started` doesn't mean ready.** The `wait` endpoint tells you the VM booted, not that your app inside is serving. Poll your app's health endpoint through the [exec endpoint](/api/machines/machines/execute-command) before marking the entry `ready`:
```bash theme={null}
curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/exec" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"command": ["curl", "--fail", "--silent", "--max-time", "5", "http://localhost:8080/healthz"]}'
```
The exec call returns HTTP 200 even when the command inside failed, so check the response body's `exit_code`, not the API status. `--fail` makes the inner `curl` return non-zero on a 5xx, so a wedged app reads as unhealthy instead of ready. (Whatever probe you use has to exist in the image.)
This is what makes claiming instant later. A `ready` entry is one whose app is already serving, so the only work left at claim time is configuration. Skip this and your "instant" claims hand users a Machine that's still booting.
Pool Machines don't know their user yet, so boot them with a minimal placeholder config (and dummy values for any env vars your entrypoint insists on). The real config arrives at claim time. If any provisioning step fails, delete the app (it cascades) and mark the row `failed` for the sweeper.
## Claim atomically with SKIP LOCKED
When a user signs up, claim an entry with one atomic statement:
```sql theme={null}
UPDATE pool SET allocated_at = NOW(), allocated_to = $user_id
WHERE id = (
SELECT id FROM pool
WHERE status = 'ready' AND allocated_at IS NULL
ORDER BY provisioned_at ASC -- oldest first
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *
```
`FOR UPDATE SKIP LOCKED` is doing the heavy lifting: concurrent signups never grab the same entry and never wait on each other's locks. No advisory locks, no queue, no coordinator.
Prefer the user's [region](/reference/regions), but don't insist on it. Run the query filtered to nearby regions first, then unfiltered. A warm Machine in the wrong region beats a cold start in the right one. If the query returns zero rows, provision directly using the same sequence as the background worker; the pool is an optimization, never a dependency.
Be careful how you unwind a failed claim. If the claim fails *before* you write anything user-specific, clearing `allocated_at` safely returns the entry to the pool. Once you've written the user's config or secrets to the volume, that entry is contaminated: returning it to the pool can hand the next user the previous one's data. After any user-specific write, mark the entry `failed` and let the maintenance loop destroy it. Don't recycle it.
## Personalize with exec, not a Machine update
There are two ways to turn a generic pool Machine into *this user's* Machine, and the difference between them is most of the payoff:
| approach | mechanism | latency |
| - | - | - |
| exec write | write the user's config to the volume via exec; the app hot-reloads it | \~1-2s |
| Machine update | `POST /machines/{id}` with new config, full Machine restart | \~20s |
If your app can watch a config file and reload it, write the file and you're done. Base64-encode the user's config, drop it in where the exec command decodes it, and write to a temp file before renaming so a reader never sees a half-written config:
```bash theme={null}
# CONFIG_B64 = base64 of the user's config; splice it into the command below
curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/exec" \
-H "Authorization: Bearer ${FLY_API_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"command": ["sh", "-c", "umask 077; echo CONFIG_B64 | base64 -d > /data/config.json.tmp && mv /data/config.json.tmp /data/config.json"]}'
```
This also decides where per-user secrets should live: in the config file on the volume, not in Machine env vars. Env changes require a Machine update and a restart. A file write doesn't.
When you do need a real Machine update (rename, metadata, env vars), use the [lease flow](/machines/api/machines-resource#create-a-machine-lease) so concurrent updaters can't clobber each other: acquire a lease, send the update with the `fly-machine-lease-nonce` header, release the lease in a `finally`. And know that Machine updates replace the entire config; they don't merge. `GET` the current Machine first and send back its full config with only your fields changed, or you'll silently wipe mounts, services, checks, and env.
## Maintain the pool
A small worker, ideally its own [process group](/launch/processes), loops on an interval. Each tick does four things, in order:
1. **Cycle.** Delete unallocated entries whose `provision_version` doesn't match the current one. Bump the version whenever your image or Machine config changes, and the pool replaces itself over the next few ticks.
2. **Clean.** For `failed` entries, delete the app and the row.
3. **Validate.** `GET /machines/{id}` for each `ready` entry; discard anything whose `state` isn't `started`. Also discard rows stuck in `provisioning` for more than \~10 minutes. That's a crashed provisioner, and the incrementally-written IDs let you clean up whatever it left behind.
4. **Replenish.** Count available entries (`status = 'ready' AND allocated_at IS NULL`) plus in-flight `provisioning` entries against your target, and create the shortfall. Counting claimed-but-still-`ready` rows here is the classic bug: the pool looks full and quietly stops refilling as users claim entries. Spread the new entries across regions by weight (say, 3 in `us`, 1 in `eu`, 1 in `apac`, where those are your own buckets of concrete Fly regions like `iad`, `ams`, and `sin`). Counting in-flight `provisioning` entries stops you from over-provisioning during a burst of claims. Guard against overlapping runs.
## Pointers and footguns
* **Warm Machines cost money.** Pool size is a bet on your peak claim rate versus your tolerance for cold-start fallbacks. Start small (2-5), log ready/provisioning counts every tick, and grow only when you see fallbacks under real traffic. If your app boots fast, a pool of *stopped* Machines that you `start` on claim may be the better trade, since stopped Machines cost much less than running ones.
* **Mint unique secrets per entry** (auth tokens, gateway passwords) at provision time and store them encrypted in your database. A claimed Machine should never be reachable with another entry's credentials.
* **Record what the API returned, not what you asked for.** Region inputs resolve at creation time; the Machine's actual region is what your claim query needs to match against.
* **Machines are tied to physical hardware.** A host failure destroys the Machine and its volume. For pool entries that's fine, since the maintenance loop replaces them, but anything durable a user creates after claiming should also be backed up off-Machine, to somewhere like [Tigris](/tigris) or S3.
* **TCP checks beat HTTP checks for pooled apps.** Your app's HTTP endpoints may require auth or per-user config that doesn't exist pre-claim. A TCP check on the listening port with a generous `grace_period` is the reliable signal.
## Related reading
* [Per-User Dev Environments with Fly Machines](/blueprints/per-user-dev-environments): The architecture this pattern plugs into, including routing with `fly-replay`.
* [Connecting to User Machines](/blueprints/connecting-to-user-machines): How to get traffic to a fleet of per-user Machines once you've assigned them.
* [Why one app per user?](/machines/guides-examples/one-app-per-user-why): The isolation reasoning behind app-per-entry.
* [Machines API reference](/machines/api): Apps, Machines, volumes, leases, and exec.
# Background job queues with Celery, BullMQ, and Fly Machines
Source: https://docs.fly.io/blueprints/work-queues
Move slow work like PDFs, emails, and API calls out of web requests into a work queue, using Celery for Python, BullMQ for Node, or on-demand Fly Machines.
## Overview
Some things just take a while. Generating a PDF. Calling a slow third-party API. Running a machine learning model. Sending an email newsletter, and retrying if the mail server flakes out.
Whatever the reason, if your web app is sitting around waiting for that work to finish, you're doing it wrong. Not just in the “inefficient” sense—but in the “your app will time out and confuse users and make debugging miserable” sense.
Instead, you should offload that work to a background worker.
This isn’t a new idea. The pattern’s been around forever, often called a **work queue,** **job queue,** or **task queue.** Here’s the gist:
* The web app gets a request.
* Instead of doing everything right away, it drops a job onto a queue.
* A separate process—called a worker—pulls jobs from that queue and handles them in the background.
This means your web handler can respond immediately. It also means your background work can run on different machines, retry on failure, and scale independently from your frontend.
You’ve probably already used something like this without realizing it. Ever sent an email via a task queue? That was a worker. Ever seen a spinner on a dashboard while a report gets generated? There's probably a job queue behind it.
In this guide, we’ll show you how to set this up—using Celery if you're in Python, BullMQ if you're in Node, and Fly Machines if you're allergic to running idle containers.
If you're already comfortable with queues, skip ahead. But if you’ve been stuffing async tasks into HTTP handlers and hoping for the best, this is your exit ramp. Let’s clean it up.
***
## Why use a work queue?
* **Responsiveness**: Return HTTP responses instantly. Let workers take their time.
* **Scalability**: Scale web servers and workers independently. No need to waste compute on idle async calls.
* **Retries**: Jobs can fail and retry cleanly, without blocking a request or annoying a user.
* **Cost efficiency**: With Fly Machines, you only pay for worker time while the job is running.
* **Persistence**: Jobs don’t care if a user closes their browser tab. Work still gets done.
***
## Redis and Valkey
Most of the tools in this guide use [Redis](https://redis.io/about/) as a message broker—a fast, in-memory data store that's great for short-lived messages and task queues. If you're deploying on Fly.io, we recommend trying [Valkey](https://valkey.io/), a fork of Redis with a more active community and governance model. You can set up a Valkey cluster without needing an Enterprise license. It works the same way but gives you more flexibility and long-term stability. You can deploy either Redis or Valkey yourself or use a managed service like [Upstash Redis](/upstash/redis).
***
## Option 1: Celery (Python)
[Celery](https://docs.celeryq.dev/en/latest/getting-started/introduction.html) is a full-featured distributed task queue commonly used with Django and Flask.
### How it works
1. Your app enqueues a task to a message broker (Redis or RabbitMQ).
2. A worker process picks up the task from the queue and executes it.
3. Task results can optionally be stored (e.g., in Redis or your database).
### Setup
Install the basics:
```bash theme={null}
pip install "celery[redis]" django-celery-results
```
Create a `celery.py`:
```python theme={null}
from celery import Celery
app = Celery("my_app")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
```
Use `@shared_task` for defining tasks:
```python theme={null}
from celery import shared_task
@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=3, retry_kwargs={"max_retries": 5})
def crunch_numbers(self, arg):
...
```
Run your worker:
```bash theme={null}
celery -A my_app worker -l info
```
### Deploying to Fly.io
Use `fly launch` to generate config and add Upstash Redis.
Update `fly.toml`:
```
[processes]
app = "gunicorn my_app.wsgi"
worker = "celery -A my_app worker -l info"
```
Then:
```bash theme={null}
fly deploy
fly scale count worker=2
```
> [!NOTE]
>
> **Region tip**: Put your Redis instance and workers in the same region (e.g. `ord`, `fra`) to avoid unnecessary latency and weird race conditions.
***
## Option 2: On-Demand Workers with Fly Machines
You don’t always need a persistent queue and worker pool. For low-frequency or one-off jobs, you can spin up a Fly Machine per task.
### How it works
1. Your app writes a job (params, metadata) to Redis/Valkey.
2. It calls the Fly Machines API to spin up a temporary worker.
3. The worker reads job data from Redis, runs it, and writes back the result.
4. Your app polls Redis to check status and fetch results.
5. You shut down the machine and clean up.
This is great for jobs that don’t happen often, and it keeps your infrastructure lean. You can reuse your app image or build a dedicated worker image.
Just make sure your job data and results are JSON-serializable.
***
## Option 3: BullMQ (Node.js)
BullMQ is a Redis-backed queue system for Node.js apps. It works similarly to Celery:
* Your app enqueues jobs to Redis/Valkey.
* Workers pull jobs and execute them.
* You get features like retries, rate limits, job delays, and concurrency control.
BullMQ plays nicely with TypeScript, has good docs, and integrates well with modern backend frameworks like NestJS.
Provision Redis via Upstash or use `fly redis create`.
Scale workers as needed based on job load and machine size:
```javascript theme={null}
const worker = new Worker("my-queue", async job => {
// do your work here
}, { concurrency: 4 });
```
***
## Other Options
If none of the above tools fit your stack or mental model, there are plenty of alternatives:
* **Sidekiq** (Ruby): The de facto standard for background jobs in Ruby and Rails apps. Redis-backed, battle-tested, and fast.
* **Oban** (Phoenix) A robust background job processing library for Elixir
* **RQ** (Python): A simpler, more lightweight job queue than Celery, great for smaller projects or simpler needs.
Most of these integrate with Redis/Valkey, some support more exotic brokers (like Postgres or Kafka), and all of them can be deployed on Fly.io with the right setup.
***
## Final Notes
Whatever you do, don’t handle long-running jobs inside your HTTP request handlers. Use a job queue. Pick the right tool for your stack and use case, and let your web server get back to doing what it’s good at—responding fast and not melting down under load.
## Related reading
* [Async workers on Fly Machines](https://fly.io/blog/python-async-workers-on-fly-machines/)
* [Celery and Django for async tasks](https://fly.io/django-beats/celery-async-tasks-on-fly-machines/)
# Working with Docker on Fly.io
Source: https://docs.fly.io/blueprints/working-with-docker
How Fly.io turns Docker images into VMs: no Docker daemon, ephemeral filesystems, prebuilt images for faster deploys, image tags, and multi-container Machines.
> [!NOTE]
>
> **Fly.io runs apps close to your users by turning Docker images into VMs running in our global network. This is a practical guide for using Docker effectively on Fly.io.**
## Overview
If you've deployed an app with `fly launch`, you've already worked with Docker on Fly.io. Behind the scenes, Fly.io builds every deployed app as a Docker image. But here's the twist: Fly.io doesn't run Docker containers. It uses Docker images as a packaging format. Once deployed, your app runs inside a lightweight VM booted from an unpacked image. This post explains what that means, how to work with images and tags, and how to speed up deployments by prebuilding images.
Fly now also supports multi-container Machines. If you're looking for more information and use cases, check out [this forum post](https://community.fly.io/t/using-containers-with-flyctl/24729) and [this guide](/machines/guides-examples/multi-container-machines).
## Why you should care
Understanding how Docker images work on Fly.io unlocks faster deployments, better image reuse, and more control over your app lifecycle. If you're running large builds, deploying the same app to multiple Fly apps, or trying to debug tricky release issues, Docker image fluency pays off.
## How Docker images work on Fly.io
When you deploy to Fly.io, we take your Docker image and unpack it into a root filesystem for a virtual machine. That’s it. No Docker daemon. No containers. Just your app running inside a lightweight VM.
At boot, each Fly.io machine runs a minimal init process that sets up the environment and kicks off whatever you’ve configured in your Dockerfile’s `ENTRYPOINT` and `CMD`. When your app exits, the VM shuts down.
The filesystem is ephemeral. If your app writes files, installs packages, or modifies anything at runtime, those changes vanish when the machine stops. On the next boot, it’s back to the clean image. Want to keep data around? Mount a [volume](/volumes/overview).
## Prebuilding for faster deploys
By default, `fly deploy` builds and deploys in one step. That can be slow if your Dockerfile does a lot of work. If you're deploying often, it can help to split build from release:
```bash theme={null}
fly deploy --build-only --push
```
This builds your image and pushes it to the registry, but doesn't deploy it.
When you're ready, deploy that prebuilt image with:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:your-tag-here
```
This shaves valuable seconds (or minutes) off deployment time, and lets you treat builds and releases as separate steps.
## Managing image tags
When you run `fly deploy`, we build a Docker image and push it to Fly.io's private registry. That image gets a tag like `deployment-20231208-123456`.
To deploy that same image again, or to another app in your org, you can reference it explicitly:
```bash theme={null}
fly deploy --image registry.fly.io/myapp:deployment-20231208-123456
```
Want friendlier tags? Use the `--image-label` flag:
```bash theme={null}
fly deploy --image-label my-cool-feature-abc123
```
You can then re-deploy it later with `--image` as shown above. This is especially handy for canary deploys or promoting the same image to staging and production.
Want to manage images manually? Authenticate your Docker client with:
```bash theme={null}
fly auth docker
```
This lets you tag, push, and pull images using standard Docker commands.
## Troubleshooting common Docker image issues
* **My app is writing data, but it’s gone after restart!**\
That’s expected. Fly.io Machines reset their root filesystem each time they start. Use [volumes](/volumes/overview) for persistence.
* **Why is my deploy slow?**\
Split build and deploy steps: run `fly deploy --build-only --push`, then `fly deploy --image` to skip rebuilding every time.
* **I’m getting weird build errors.**\
Use `fly deploy --build-local` to build with your local Docker daemon. It's often easier to debug than remote builds. Or just use plain old Docker: `docker build -t myapp .` to build locally and `docker run` to test the image before deploying.
* **My image builds fine but fails at runtime.**\
Check your `ENTRYPOINT` and `CMD`. Your app runs as PID 1 in a VM, not in a Docker container. Use `fly logs` to investigate. You can also `docker run` the image locally to spot runtime issues before deploying.
* **My image isn’t deploying correctly to another app.**\
Make sure both apps are in the same Fly.io org, and reference the full tag from the Fly registry.
* **My app crashes or misbehaves on startup. What’s going on?**\
Fly.io doesn’t run readiness checks before starting your app by default. If your app depends on something like a database or external service, you need to make sure it waits until that service is ready before it starts doing work. Add retries or startup logic to handle this gracefully. Logs are your friends - run `fly logs` on a separate console to see what the app does or says at startup.
* **It works locally, but not in production.**\
That probably means “it runs on my machine,” not “the image I’m deploying runs correctly.” Build the image locally with `docker build` and test it with `docker run`. That’s usually the fastest way to spot missing files, bad assumptions, or environment mismatches. Also, pin your base images, audit your `.dockerignore`, and make sure everything your app needs is actually included in the image.
## Tips for optimizing your Dockerfile
A few well-placed changes to your Dockerfile can dramatically improve build speed, runtime performance, and security:
* **Use multi-stage builds.** Compile code and install build-only dependencies in one stage, then copy only what's needed into a smaller final image.
* **Choose minimal base images.** Use Alpine distroless, or slim variants of Debian/Ubuntu to reduce image size and attack surface. Slim images often provide a good middle ground between minimalism and compatibility.
* **Leverage layer caching.** Put the least frequently changing steps (like system packages) early in the file.
* **Combine `RUN` steps.** Use `&&` to chain commands and reduce image layers.
* **Clean up after installs.** Remove package caches and temp files in the same `RUN` block.
* **Use `.dockerignore`.** Keep your build context clean: exclude `node_modules`, `.git`, logs, etc.
* **Pin image versions.** Don’t rely on `latest`; specify exact versions to keep builds consistent.
* **Avoid hardcoded secrets.** For *runtime* [secrets](/apps/secrets), use `fly secrets`. It’s built for exactly this, and injects secrets securely into your app’s environment. For *build-time* secrets (like private package tokens), use [build secrets](/apps/build-secrets) to avoid leaking them into your final image.
* **Run as non-root.** Use `USER` to limit container privileges.
* **Scan for vulnerabilities.** Tools like Trivy, Docker Scout, and others help keep your images safe. You can also try `fly registry vulns -a ` for a quick CVE scan of your deployed image (just note that it’s still experimental).
These tips and practices can make your images smaller, safer, and more predictable to deploy.
## Related reading
* [**Deploy with a Dockerfile**](/languages-and-frameworks/dockerfile) Find out how `fly launch` and `fly deploy` work when you're starting from a Dockerfile.
* [**Managing Docker images with the Fly.io registry**](/blueprints/using-the-fly-docker-registry) Understand how image storage, tagging, and reuse work in Fly.io’s private registry.
* [**Using base images for faster deploys**](/blueprints/using-base-images-for-faster-deployments) Learn how shared base images can shave time off your CI/CD loop.
* [**Optimizing your Dockerfile**](https://docs.docker.com/build-cloud/optimization/) Advice straight from Docker on structuring builds for speed and reusability.
* [**Customizing your Dockerfile for Laravel**](/laravel/advanced-guides/customizing-dockerfile) A Laravel-specific deep dive, but useful even if you’re not using Laravel.
# Databases and storage
Source: https://docs.fly.io/database-storage-guides
Pick a data option on Fly.io: Managed Postgres, Upstash for Redis, Fly Volumes for persistent NVMe disk, Tigris object storage, or an external provider.
## Managed database service
[Fly.io Managed Postgres](/postgres) is a production-ready Postgres service that handles the hard parts for you: high availability, automatic failover, encrypted backups, monitoring and metrics, seamless scaling, and 24/7 support to keep your data fast, safe, and always online.
## Key-value stores
**[Upstash for Redis](/upstash/redis)** - [Redis](https://redis.io/) is an in-memory database commonly used for caching. A managed service by [Upstash](https://upstash.com/).
***
## Fly Volumes - Disk storage
The Fly Machines in your app provide ephemeral storage, so you get a blank slate on every startup. For hardware-local, persistent storage on Fly.io, use Fly Volumes. You can attach volumes on an app directly, or run a separate database app with volume storage and connect an app to that.
**[Fly Volumes](/volumes/overview):** A Fly Volume is a slice of NVMe disk storage attached to the server that hosts your Machine. Read the [Fly Volumes overview](/volumes/overview) to find out if volumes are the best solution for your use case.
## Object storage service
**[Tigris Global Object Storage](/tigris)** - [Tigris](https://www.tigrisdata.com/) is a globally distributed S3-compatible object storage service hosted on Fly.io infrastructure.
***
## Have a project with special database or storage requirements?
If your project calls for something else, most external databases and storage providers work well with Fly.io. You might just need to spend a bit more time wiring up the networking.
# Your Fly App
Source: https://docs.fly.io/deep-dive/application
The Fly.io deep dive demo runs on two Fly Machines with autostop and autostart. Scale them vertically or with fly scale count, edit fly.toml, and redeploy.
Your demo app is running by default on two [Fly Machines](/machines), our fast-launching VMs.
Both Machines stop when not in use, and start automatically when a new request comes in. [Autostop/autostart](/launch/autostop-autostart) is entirely configurable. You can choose to suspend instead of stop, configure a minimum number of Machines to keep running, or even decide never to stop Machines at all.
The purpose of two Machines is twofold: redundancy and scalability. If one Machine goes down, the other can continue on. If both are available, when your app has higher traffic, both can be started to handle requests. You can [vertically scale](/launch/scale-machine/) the CPU and RAM on Machines.
You can also [horizontally scale](/launch/scale-count) the number of Machines. You can scale out to different regions with the `fly scale count` command too. If you have a co-worker on another continent, create a Machine there.
Familiarize yourself with the [fly.toml](/reference/configuration) config file. Make a change there--or to any part of your app--and run `fly deploy` to update your app.
Your Fly App takes advantage of [Anycast routing](/networking/services), a [load-balancing proxy](/reference/fly-proxy), and [private networking](/networking/private-networking). You can also create [DNS certificates for custom domains](/networking/custom-domain).
# Django demo reference
Source: https://docs.fly.io/deep-dive/django
How the Django dictaphone demo fits together: Django Storages on Tigris, Channels on Redis, Postgres via DATABASE_URL, and Celery jobs that call Whisper.
The Django source is on [GitHub](https://github.com/fly-apps/django-dictaphone/).
`fly launch` will provide a [`Dockerfile`](https://docs.docker.com/reference/dockerfile/)
and a [`fly.toml`](/reference/configuration) config file.
When you make changes to your application, you can use
[dockerfile-django](https://github.com/fly-apps/dockerfile-django?tab=readme-ov-file#dockerfile-generator-for-django-projects)
to produce updated Dockerfiles.
How the pieces are put together:
* The original web-dictaphone app (minus the HTML) is in the
[static](https://github.com/fly-apps/django-dictaphone/tree/main/static) directory,
and contains icons, scripts, styles, and a web manifest; all served as
static files.
* [`clips/templates/clips/index.html`](https://github.com/fly-apps/django-dictaphone/blob/main/clips/templates/clips/index.html) contains the HTML template.
* [Django Models and databases](https://docs.djangoproject.com/en/5.1/topics/db/) is [configured](https://github.com/fly-apps/django-dictaphone/blob/2a9ec4ceb728e657aeec76477d87ea1a8627523a/dictaphone/settings.py#L89-L102) to use Sqlite3 as the database in development mode, and
PostgreSQL in production. Access to the database is through the `DATABASE_URL` secret.
* [Django Storages](https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html) is [configured](https://github.com/fly-apps/django-dictaphone/blob/2a9ec4ceb728e657aeec76477d87ea1a8627523a/dictaphone/settings.py#L104-L117) to write to the filesystem
in development and Tigris in production. The following secrets are used to
establish the connection: `AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_SECRET_ACCESS_KEY`, and `BUCKET_NAME`.
* [Django Channels](https://channels.readthedocs.io/en/latest/) is [configured](https://github.com/fly-apps/django-dictaphone/blob/2a9ec4ceb728e657aeec76477d87ea1a8627523a/dictaphone/settings.py#L119-L135) to use an in memory channel in development and Redis in production
based using the `REDIS_URL` secret.
Key points of logic:
* [`static/scripts/app.js`](https://github.com/fly-apps/django-dictaphone/blob/main/static/scripts/app.js)
has been modified to make server requests at various points using
[the Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch).
* [`clips/views.py`](https://github.com/fly-apps/django-dictaphone/blob/main/clips/views.py) contains
the logic to build responses to requests for the index page, and to GET, PUT,
and DELETE audio clips.
* The server side realtime implementation code is primarily in [Django Channels](https://channels.readthedocs.io/en/latest/), so all that is needed is a few lines in [`clips/models.py`](https://github.com/fly-apps/django-dictaphone/blob/b1bb70cc513bd4977dc74fd56dfff86b8f2d12ec/clips/models.py#L20-L25).
The client side is provided by [`static/scripts/websocket.js`](https://github.com/fly-apps/django-dictaphone/blob/main/static/scripts/websocket.js).
When update notifications are received, the index page is re-retrieved and the body of the DOM is replaced with new contents.
* When the `WHISPER_URL` secret is set, `PUT` requests will cause the audio clips
to be passed to the Whisper server, and responses will be used to update the
PostgreSQL database. This is done using [Celery](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html).
The code for this is in [`clips/tasks.py`](https://github.com/fly-apps/django-dictaphone/blob/main/clips/tasks.py).
# Deep dive demo
Source: https://docs.fly.io/deep-dive/index
An hour-long Fly.io demo deploying MDN's Web Dictaphone with Postgres, Tigris, and Upstash Redis WebSockets in Django, Next.js, Node, Phoenix, or Rails.
Welcome to our deep dive demo, where you can explore Fly.io more thoroughly, but in a time-boxed way. In one hour or less: get a fully-functioning app running in the first few minutes, and then have enough time left over to understand what you just did and explore how the pieces fit together.
## Goals of the deep dive
It's really just one goal:
**Help you find out if Fly.io is the right place for you.**
If you came from our [Speedrun](https://fly.io/speedrun) or [Getting started](/getting-started) pages, maybe you want to go beyond `hello world` and explore the platform so you can feel confident that you're choosing a provider that can support you now, and scale with you later on as you grow.
Maybe you're concerned about lock in. You want to use services from other places and even eject entirely and go elsewhere if things don't work out.
Perhaps it is a simple as you feel it is easier to tune and evolve a working
system than it is to debug why your application doesn’t launch on an unfamiliar
platform.
If you don't get a good feeling within an hour, you're out of here. If you do get a good feeling, then stick around and try launching the app that you brought here to launch.
## Beyond `hello fly`
A real-world application has, at a minimum, the following components:
* An HTML form and a database; typically a relational database.
* The ability to handle media files or documents, generally using S3.
* A multi-user and realtime component, where changes made by one person in one location are reflected instantly in the browser of another person.
To get a fully-functional app running smoothly, you need a whole bunch of things; things like Anycast routing, load balancers, DNS certificates, WebSockets, an internal private network, a relational database, an object store, and an in-memory database. And the knowledge to connect them all together.
Connecting complex components together generally takes an unpredictable amount of time due to surprises. Usually, set up for a complete app typically takes a *minimum* of an afternoon's worth of work, even if you're familiar with a cloud platform. We will be up and running in minutes.
Fly.io minimizes surprises by taking a lot of work off your plate with our [Public Network Services](/networking/services), [Fly Proxy routing](/reference/fly-proxy), out-of-the-box [Private Networking](/networking/private-networking), and preconfigured databases.
While we can't promise no surprises, we can show you how we'll partner with you to handle some of the dev ops complexities of working in a public cloud.
## So what is this deep dive demo app?
The deep dive demo uses industry standard components that you can run on your laptop, a VPS, AWS EC2, Google Compute Engine, or Azure.
### Web Dictaphone -- Fly.io edition
The deep dive demo is based on [MDN's Web Dictaphone](https://github.com/mdn/dom-examples/tree/main/media/web-dictaphone).
You can play with a [live demo hosted on GitHub](https://mdn.github.io/dom-examples/media/web-dictaphone/). The Web Dictaphone app is about as basic of an HTML form as you can get, and it has the added bonus of providing the ability to generate as many media files as you want using only your voice.
The basic Web Dictaphone is client side only, requiring a web server that can deploy static assets (HTML, CSS, JS, images), like nginx, Apache HTTPd, or Caddy. Storing the data in databases for our deep dive demo requires a server that can handle HTTP GET, POST, PUT, and DELETE requests. The choice of server varies depending on the language or framework.
The demo includes the following components:
* A [PostgreSQL](https://www.postgresql.org/) relational database to store the names of the audio clips
* A [Tigris bucket](https://www.tigrisdata.com/) to store the audio files
* [Upstash for Redis](/upstash/redis) and [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) to handle the connections for the realtime requirement
### Available runtimes
Our deep dive demo comes in five flavors (so far):
* [Django](/deep-dive/django)
* [Next.js](/deep-dive/nextjs)
* [Node.js](/deep-dive/nodejs)
* [Phoenix](/deep-dive/phoenix)
* [Rails](/deep-dive/rails)
# Launch the demo
Source: https://docs.fly.io/deep-dive/launch-deep-dive
Install flyctl, then run fly launch --from with the Django, Node.js, Next.js, Phoenix, or Rails dictaphone repo to deploy the deep dive demo on Fly.io.
Let's get right to it.
**Step 0:** To get set up to run the deep dive demo on Fly.io, all you need to do is install Node, Phoenix, Python, or Ruby and create an empty project directory.
After that there are only two steps.
**Step 1:** [Install flyctl](/flyctl/install).
**Step 2:** Run one of the following:
```bash theme={null}
fly launch --from https://github.com/fly-apps/django-dictaphone
```
-or-
```bash theme={null}
fly launch --from https://github.com/fly-apps/node-dictaphone
```
-or-
```bash theme={null}
fly launch --from https://github.com/fly-apps/nextjs-dictaphone
```
-or-
```bash theme={null}
fly launch --from https://github.com/fly-apps/phoenix-dictaphone
```
-or-
```bash theme={null}
fly launch --from https://github.com/fly-apps/rails-dictaphone
```
If you're new to Fly.io, the second step will take you to a page where you can register before continuing.
The `fly launch` output describes what you'll be getting for the app, and gives you an opportunity to tweak the settings. (Suggestion: don't. The defaults are fine for this demo and we'll walk you through how to adjust them later.) And then it will build and assemble and wire up your app.
Take your time and play with it. Open the application in multiple browser windows. Send a link to a friend on another continent and watch your browser update in realtime.
And then relax. We promised you it would be less than an hour. You're already up and running. If you are so inclined, try bringing up this exact same application on another cloud provider. We don't mind. In fact, we encourage it. Just please don't count the time you spent there against the hour you set aside for this demo!
## Explore the result of `fly launch`
Once you're back and/or rested up, explore the app and add-on components. Feel free to review the following in any order, or chose to skip ahead:
* [Your Fly App](/deep-dive/application)
* [PostgreSQL database](/deep-dive/postgresql)
* [Tigris Object Storage](/deep-dive/tigris)
* [Upstash Redis](/deep-dive/redis)
## Runtimes
Optional, but if you haven't yet dived in, details are provided for each runtime:
* [Django](/deep-dive/django)
* [Next.js](/deep-dive/nextjs)
* [Node.js](/deep-dive/nodejs)
* [Phoenix](/deep-dive/phoenix)
* [Rails](/deep-dive/rails)
***
# Next.js demo reference
Source: https://docs.fly.io/deep-dive/nextjs
Walk through the Next.js dictaphone source: server.js WebSockets, the app/audio route for clip GET, PUT, and DELETE, pg for Postgres, pubsub.js for Redis.
The Next.js demo source is on [GitHub](https://github.com/fly-apps/nextjs-dictaphone).
`fly launch` will provide a [`Dockerfile`](https://docs.docker.com/reference/dockerfile/)
and a [`fly.toml`](/reference/configuration) config file. When you make changes to your application, you can run `npx @flydotio/dockerfile` to produce updated Dockerfiles.
How the pieces are put together:
* The original web-dictaphone app is placed in the [Next.js app](https://nextjs.org/docs/app) directory:
* [`app/dictaphone.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/dictaphone.js)
contains the JavaScript.
* [`app/globals.css`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/globals.css) contains
the stylesheets.
* [`app/page.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/page.js) contains the HTML
* [`server.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/server.js) and
[`app/audio/[name]/route.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/audio/%5Bname%5D/route.js) contain
the server implementation.
* The [pg](https://www.npmjs.com/package/pg) module is used to access PostgreSQL.
Access to the database is through the `DATABASE_URL` secret. This module is low level
which is sufficient for this demo, but most modern Node.js applications use
an [ORM](https://amplication.com/blog/top-6-orms-for-modern-nodejs-app-development).
* The [AWS SDK for JavaScript](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/welcome.html)
is used to access Tigris. The following secrets are used to
establish the connection: `AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_SECRET_ACCESS_KEY`, and `BUCKET_NAME`.
* The [express-ws](https://github.com/HenningM/express-ws?tab=readme-ov-file#express-ws-)
module is used for [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) support.
* The [redis](https://github.com/redis/node-redis?tab=readme-ov-file#node-redis) module
is used to access Redis. The `REDIS_URL` secret is used to access the database.
Key points of logic:
* [`app/page.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/page.js) contains the
home page. All of the clip data comes from web sockets.
* [`app/dictaphone.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/dictaphone.js)
has been modified to make server requests at various points using
[the Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch); the delete
and rename actions are split out so that they can be referenced by the React JSX Home page view.
* [`server.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/server.js) contains
the WebSocket code, and delegates to Next.js to render the rest.
* [`pubsub.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/pubsub.js) contains the
interface to Redis. It is split out from the server to avoid circular dependencies.
* [`app/audio/[name]/route.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/audio/%5Bname%5D/route.js) contains
the logic to build responses to requests to GET, PUT,
and DELETE audio clips.
* [`app/page.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/ac1abcdc9cb1dfa7de08c682b26034e76ad34cc6/app/page.js#L12-L21)
contains the client implementation of web sockets.
* When the `WHISPER_URL` secret is set, `PUT` requests will cause the audio clips
to be passed to the Whisper server, and responses will be used to update the
PostgreSQL database. The code for this is in [`app/audio/[name]/route.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/459774ec9ebebdf6f991adaa6c88509854ffe34e/app/audio/%5Bname%5D/route.js#L90-L118).
# Node.js demo reference
Source: https://docs.fly.io/deep-dive/nodejs
Tour the Express.js dictaphone demo: EJS views, the pg module, AWS SDK access to Tigris, express-ws WebSockets, and a Redis queue that updates browsers.
The Node.js demo source is on [GitHub](https://github.com/fly-apps/node-dictaphone).
`fly launch` will provide a [`Dockerfile`](https://docs.docker.com/reference/dockerfile/)
and a [`fly.toml`](/reference/configuration) config file. When you make changes to your application, you can run `npx @flydotio/dockerfile` to produce updated Dockerfiles.
How the pieces are put together:
* The original web-dictaphone app (minus the HTML) is in the
[public](https://github.com/fly-apps/node-dictaphone/tree/main/public) directory,
and contains icons, scripts, styles, and a web manifest; all served as
static files.
* [`views/index.ejs`](https://github.com/fly-apps/node-dictaphone/blob/main/views/index.ejs) contains the HTML template.
* [app.js](https://github.com/fly-apps/node-dictaphone/blob/main/app.js) contains
the server implementation, based on [Express.js](https://expressjs.com/). There
are lots of JavaScript frameworks out there, this is perhaps the oldest and most
minimalist.
* The [pg](https://www.npmjs.com/package/pg) module is used to access PostgreSQL.
Access to the database is through the `DATABASE_URL` secret. This module is low level
which is sufficient for this demo, but most modern Node.js applications use
an [ORM](https://amplication.com/blog/top-6-orms-for-modern-nodejs-app-development).
* The [AWS SDK for JavaScript](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/welcome.html)
is used to access Tigris. The following secrets are used to
establish the connection: `AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_SECRET_ACCESS_KEY`, and `BUCKET_NAME`.
* The [express-ws](https://github.com/HenningM/express-ws?tab=readme-ov-file#express-ws-)
module is used for [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) support.
* The [redis](https://github.com/redis/node-redis?tab=readme-ov-file#node-redis) module
is used to access Redis. The `REDIS_URL` secret is used to access the database.
Key points of logic:
* [`public/scripts/app.js`](https://github.com/fly-apps/node-dictaphone/blob/main/public/scripts/app.js)
has been modified to make server requests at various points using
[the Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch).
* [`app.js`](https://github.com/fly-apps/node-dictaphone/blob/main/app.js) contains
the logic to build responses to requests for the index page, and to GET, PUT,
and DELETE audio clips.
* [`pubsub.js`](https://github.com/fly-apps/node-dictaphone/blob/main/pubsub.js) contains
the logic to subscribe to a `dictaphone:timestamp` redis queue, and to
forward updates to all open websocket connections.
* [`websocket.js`](https://github.com/fly-apps/node-dictaphone/blob/main/public/scripts/websocket.js)
contains the client implementation of web sockets. When update notifications
are received, the index page is re-retrieved and the body of the DOM is
replaced with new contents.
* When the `WHISPER_URL` secret is set, `PUT` requests will cause the audio clips
to be passed to the Whisper server, and responses will be used to update the
PostgreSQL database. The code for this is in [app.js](https://github.com/fly-apps/node-dictaphone/blob/1e84a4dece6888dfc68880d146b46511d47391b3/app.js#L102-L129).
# Phoenix demo reference
Source: https://docs.fly.io/deep-dive/phoenix
See how the Phoenix dictaphone demo uses LiveView, Ecto with Postgres, ExAws.S3 for Tigris, and Phoenix.PubSub.Redis, plus the Whisper audio controller.
The Phoenix demo source is on [GitHub](https://github.com/fly-apps/phoenix-dictaphone).
`fly launch` will provide a [`Dockerfile`](https://docs.docker.com/reference/dockerfile/)
and a [`fly.toml`](/reference/configuration) config file.
How the pieces are put together:
* The original web-dictaphone app is spread throughout your application:
* [`assets/js/dictaphone.js`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/assets/js/dictaphone.js)
contains the JavaScript.
* [`assets/css/dictaphone.css`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/assets/css/dictaphone.css) contains
the stylesheets.
* [`priv/static/favicon.ico`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/priv/static/favicon.ico) contains your image
* [`lib/dictaphone_web/live/dictaphone.html.heex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/live/dictaphone.html.heex) contains the HTML
* [`lib/dictaphone_web/live/dictaphone.ex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/live/dictaphone.ex) and
[`lib/dictaphone_web/controllers/audio_controller.ex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/controllers/audio_controller.ex) contain
the server implementation.
* [Ecto.Adapters.Postgres](https://hexdocs.pm/ecto_sql/Ecto.Adapters.Postgres.html) is used to access PostgreSQL.
Access to the database is through the `DATABASE_URL` secret.
* [ExAws.S3](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html)
is used to access Tigris. The following secrets are used to
establish the connection: `AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_SECRET_ACCESS_KEY`, and `BUCKET_NAME`.
* [Phoenix.LiveView](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html)
module is used for [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) support.
* [Phoenix.PubSub.Redis](https://hexdocs.pm/phoenix_pubsub_redis/Phoenix.PubSub.Redis.html)
is used to access Redis. The `REDIS_URL` secret is used to access the database.
Key points of logic:
* [`lib/dictaphone_web/live/dictaphone.html.heex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/live/dictaphone.html.heex) contains the
home page.
* [assets/js/dictaphone.js](https://github.com/fly-apps/phoenix-dictaphone/blob/main/assets/js/dictaphone.js)
has been modified to:
* upload clips using [the Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch)
* use [Click Events](https://hexdocs.pm/phoenix_live_view/bindings.html#click-events) for delete
* use [Client Hooks](https://hexdocs.pm/phoenix_live_view/js-interop.html#client-hooks-via-phx-hook) for rename actions
the WebSocket code, and delegates to Next.js to render the rest.
* [`lib/dictaphone_web/live/dictaphone.ex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/live/dictaphone.ex) contains the
interface to Redis. It is split out from the server to avoid circular dependencies.
* [`app/audio/[name]/route.js`](https://github.com/fly-apps/nextjs-dictaphone/blob/main/app/audio/%5Bname%5D/route.js) contains
the logic to respond to events from both browser clients, and redis pubsub.
* [`lib/dictaphone_web/controllers/audio_controller.ex`](https://github.com/fly-apps/phoenix-dictaphone/blob/main/lib/dictaphone_web/controllers/audio_controller.ex)
contains the server access to `GET` and `PUT` audio clips.
* When the `WHISPER_URL` secret is set, `PUT` requests will cause the audio clips
to be passed to the Whisper server, and responses will be used to update the
PostgreSQL database. The code for this is in [`lib/dictaphone_web/controllers/audio_controller.ex`](https://github.com/fly-apps/phoenix-dictaphone/blob/396d22a537b4f688f99ce88b707d6e045d828e6f/lib/dictaphone_web/controllers/audio_controller.ex#L42-L58).
# Deep Dive: PostgreSQL
Source: https://docs.fly.io/deep-dive/postgresql
Fly Postgres in the deep dive demo starts as one Machine in its own app. Add an HA cluster and read replicas, and keep it on the private network only.
Fly Postgres is deployed as a separate app, and that app comes initially configured with a single [Fly Postgres](/unmanaged-postgres) Machine. That's fine for development, but for production you need redundancy and scalability. With a [few commands](/unmanaged-postgres/advanced-guides/high-availability-and-global-replication) you can create an HA cluster in your primary region and read-only replicas elsewhere.
And there is no lock in here. You're free to host your database literally anywhere.
Before moving on, one last observation on the relational database. While you want and need your application to be on the internet, you are much better off if your relational database is NOT directly exposed to the internet, but can only be accessed via your application. That’s the value of an [internal private network](/networking/private-networking). The private connection from the demo app to the Postgres app was configured automatically for you.
# Rails demo reference
Source: https://docs.fly.io/deep-dive/rails
Rails dictaphone demo internals: Stimulus controllers, Active Record on Postgres, Active Storage on Tigris, Action Cable on Redis, Sidekiq Whisper jobs.
The Rails source is on [GitHub](https://github.com/fly-apps/rails-dictaphone/).
`fly launch` will provide a [`Dockerfile`](https://docs.docker.com/reference/dockerfile/)
and a [`fly.toml`](/reference/configuration) config file. `fly launch` will
also work with Rails-provided Dockerfiles. When you make changes to your application, you can run
`bin/rails generate dockerfile` to generate updated Dockerfiles.
How the pieces are put together:
* The original web-dictaphone app distributed throughout the application:
* [`app/javascript/controllers/main_controls_controller`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/javascript/controllers/main_controls_controller.js)
and [`app/javascript/controllers/sound_clip_controller`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/javascript/controllers/sound_clip_controller.js)
contain the JavaScript, implemented as [Stimulus](https://stimulus.hotwired.dev/handbook/origin) controllers.
* [`app/assets/stylesheets/dictaphone.css`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/assets/stylesheets/dictaphone.css) contains
the stylesheets.
* [`app/views/clips/index.html.erb`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/views/clips/index.html.erb) contains the HTML
* [Active Record](https://guides.rubyonrails.org/active_record_basics.html) will run using Sqlite3 as the database in development mode, and
PostgreSQL in production.
Access to the database is through the `DATABASE_URL` secret.
* [Active Storage](https://guides.rubyonrails.org/active_storage_overview.html) is configured to write to the filesystem
in development and Tigris in production. The following secrets are used to
establish the connection: `AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT_URL_S3`, `AWS_REGION`, `AWS_SECRET_ACCESS_KEY`, and `BUCKET_NAME`.
* [Action Cable](https://guides.rubyonrails.org/action_cable_overview.html) is configured to use an Async adapter
in development and the Redis adapter in production.
Key points of logic:
* [`app/controllers/clips_controller.rb`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/controllers/clips_controller.rb) contains
the logic to build responses to requests for the index page, and to GET, PUT,
and DELETE audio clips.
* The realtime implementation code is primarily in [ActionCable](https://guides.rubyonrails.org/action_cable_overview.html), so all that is needed is one line in [`app/models/clip.rb`](https://github.com/fly-apps/rails-dictaphone/blob/6bdf4f639640c9fb55530546dbbed682b65a7df9/app/models/clip.rb#L2)
and one line in [`app/views/layouts/application.html.erb`](https://github.com/fly-apps/rails-dictaphone/blob/6bdf4f639640c9fb55530546dbbed682b65a7df9/app/views/layouts/application.html.erb#L9)
* When the `WHISPER_URL` secret is set, `PUT` requests will cause the audio clips
to be passed to the Whisper server, and responses will be used to update the
PostgreSQL database. This is done using [Active Job](https://guides.rubyonrails.org/active_job_basics.html)
and [Sidekiq](https://sidekiq.org/).
The code for this is in [`app/jobs/whisper_transcribe_job.rb`](https://github.com/fly-apps/rails-dictaphone/blob/main/app/jobs/whisper_transcribe_job.rb).
# Recap
Source: https://docs.fly.io/deep-dive/recap
Wrap up the Fly.io deep dive after deploying a full-stack app with object storage, realtime updates, and AI, plus links to production, pricing, support.
We promised that you could get through this in under an hour. You successfully deployed a full-stack app with both object storage and realtime requirements, and then went on to add AI functionality.
Up front, we set a goal: **Help you find out if Fly.io is the right place for you.**
Fly.io can support your apps now, with flexible built-in features, and scale with you later as you grow, with low-level control of Machines and powerful compute in 35 [regions](/reference/regions).
You're never locked in; you can always eject entirely and move your entire application elsewhere if things don’t work out.
How did we do? Visit [community](https://community.fly.io/) and let us know.
## Learn more
* [Going to production](/apps/going-to-production)
* [Support](/about/support)
* [Pricing](/about/pricing)
* [Application Monitoring by Sentry](/monitoring/sentry)
* [Application Security by Arcjet](/security/arcjet)
# Upstash Redis
Source: https://docs.fly.io/deep-dive/redis
Upstash Redis in the deep dive demo broadcasts pubsub updates to each Machine, which tells browsers over WebSockets. It also caches for Node and Rails.
[Upstash Redis](/upstash/redis) is used by the deep dive demo app for its [pubsub](https://redis.io/docs/latest/commands/?group=pubsub) capabilities, but it can do [so much more](https://upstash.com/docs/redis/overall/rediscompatibility).
In particular, Redis is useful for caching:
* [Node and Redis](https://redis.io/learn/develop/node/nodecrashcourse/caching)
* [Rails and Redis](https://guides.rubyonrails.org/caching_with_rails.html#activesupport-cache-rediscachestore)
In the deep dive demo app, updates are broadcast to all Machines via Redis, and then each Machine informs browser clients
of the update via [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API). The client requests
updated information from the application using HTTP GET.
# Tigris
Source: https://docs.fly.io/deep-dive/tigris
Tigris object storage serves the demo's audio files across regions with no setup. Make a bucket public, or migrate from S3 gradually with shadow buckets.
[Tigris](/tigris) is a globally distributed object storage service. It essentially requires no configuration, and seamlessly handles multi-regions. If you upload an audio file to a host in Virginia, you can access it from Amsterdam. Even better: subsequent accesses from regions other than the primary region are served locally.
Unlike relational databases, there may be reasons why you want to make your object store available via the internet. In this demo, the object store starts out private, but you can make it [public](/tigris#public-buckets) if you want. Public buckets serve objects without authentication at `https://bucket-name.t3.tigrisfiles.io/key-name`.
And if you happen to have an existing S3 object store, check out [shadow buckets](/tigris#migrate-from-another-s3-compatible-provider-with-zero-downtime), which enable you to incrementally migrate your data.
# Whisper
Source: https://docs.fly.io/deep-dive/whisper
Add speech-to-text to the deep dive demo with fly launch --attach, which runs OpenAI Whisper on an L40S GPU Machine that is reachable only over Flycast.
Since you are now officially a Fly.io expert, let's dive right in and run the following two commands in the project directory of your deep dive demo app:
```
fly launch --attach --from https://github.com/rubys/cog-whisper
fly deploy
```
Next try capturing a new audio clip. It will be transcribed automatically by OpenAI Whisper, an automatic speech recognition (ASR) program that converts speech into text.
## What just happened
You provisioned a new Machine, this time with an [L40S](https://www.nvidia.com/en-us/data-center/l40s/) GPU.
It will stop when not in use. It will restart when a new request comes in.
It is only available on the private network using [Flycast](/networking/flycast).
It runs [OpenAI Whisper](https://openai.com/index/whisper/) accessed via a [COG](https://github.com/replicate/cog) interface.
This process involves taking audio clips from Tigris, passing them to Whisper, and updating Postgres with the results.
The Node code for this is about [two dozen lines of code](https://github.com/fly-apps/node-dictaphone/blob/1e84a4dece6888dfc68880d146b46511d47391b3/app.js#L102-L129),
and for Rails is about a [dozen](https://github.com/fly-apps/rails-dictaphone/blob/6bdf4f639640c9fb55530546dbbed682b65a7df9/app/jobs/whisper_transcribe_job.rb#L5-L16).
And, as always, there's no lock in. You can opt to replace this with a machine hosted by [Replicate](https://replicate.com/) or elsewhere.
# Django: Advanced Guides
Source: https://docs.fly.io/django/advanced-guides
Advanced Django guides for Fly.io, currently covering staging environments: spin up a separate staging app for each change using GitHub Actions.
## Staging environments with GitHub actions
How to create staging environments on the Fly.io with GitHub actions.
# Staging environments with GitHub actions
Source: https://docs.fly.io/django/advanced-guides/staging-environments-with-github-actions
Create a Django staging environment for each pull request on Fly.io with GitHub Actions and fly-pr-review-apps, optionally using a staging Postgres cluster.
Creating staging environments for testing changes to our apps can be a challenge. This
guide shows how to use GitHub actions to smoothly create a separate staging
environment for each pull request using the
[`fly-pr-review-apps`](https://github.com/marketplace/actions/pr-review-apps-on-fly-io)
action, which will create and deploy our Django project with changes from the specific
pull request. It will also destroy it when it's no longer needed, such as after closing
or merging a pull request. The entire staging process enclosed in one GitHub action, so
we don't have to worry about anything else (**NoOps**).
## Basic flow
[GitHub action](https://github.com/features/actions) workflows are defined by YAML files
in the `.github/workflows/` directory of our repository. Let's add a new flow that will
create and deploy staging environment for each pull request. But first we need to create
a new repository secret called `FLY_API_TOKEN` to use for authentication. Go to a GitHub
repository page and open:
*Settings (tab) → Security → Secrets and variables → Actions → Repository secrets → New repository secret*
next, create a new secret called `FLY_API_TOKEN` with a value from:
```bash theme={null}
fly tokens org
```
> [!NOTE]
>
> It's also possible to create a token from the organization dashboard, under the "Tokens" tab.
Now, we're ready to add a new flow. This is how we can define it in the
`.github/workflows/fly_pr_preview.yml` file:
```yaml theme={null}
# .github/workflows/fly_pr_preview.yml
name: Start preview app
on:
pull_request:
types: [labeled, synchronize, opened, reopened, closed]
concurrency:
group: ${{ github.workflow }}-pr-${{ github.event.number }}
cancel-in-progress: true
permissions:
contents: read
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
jobs:
preview-app:
if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
runs-on: ubuntu-latest
name: Preview app
environment:
name: pr-${{ github.event.number }}
url: ${{ steps.deploy.outputs.url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy preview app
uses: superfly/fly-pr-review-apps@1.2.0
id: deploy
with:
region: waw
org: personal
```
It's time to split our configuration up into its component parts:
* `on → pull_request → types`: specifies [events](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)
on which a new staging project will be deployed:
```yaml theme={null}
pull_request:
types: [labeled, synchronize, opened, reopened, closed]
```
* `concurrency`: prevents concurrent deploys for the same PR (Pull Request). The `group`
name contains a PR number and workflow name to create a separate group for each
workflow and PR:
```yaml theme={null}
concurrency:
group: ${{ github.workflow }}-pr-${{ github.event.number }}
cancel-in-progress: true
```
* `env`: makes the `FLY_API_TOKEN` secret available to use for authentication:
```yaml theme={null}
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
```
* `jobs → preview-app → if`: skips deploy on PRs without the
*PR preview app*
label. Both for safety reasons and to avoid creating a staging environments when no
needed:
```yaml theme={null}
if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
```
* `jobs → preview-app → environment`: describes the deployment target to show up in a
pull request UI. `steps.deploy.outputs.url` is filled by the
`superfly/fly-pr-review-apps` and will contain the URL of a deployed staging project:
```yaml theme={null}
environment:
name: pr-${{ github.event.number }}
url: ${{ steps.deploy.outputs.url }}
```
* `jobs → preview-app → steps → with`: specifies all inputs that we want to pass to our
flow. You can check available options in the
[README](https://github.com/superfly/fly-pr-review-apps#inputs). We pass the Fly.io
[region](/reference/regions#fly-io-regions) and organization as a
starting point:
```yaml theme={null}
with:
region: waw
org: personal
```
The action configured in this way will deploy an app with the name created according to
the following pattern:
`pr-{{ PR number }}-{{ repository owner }}-{{ repository name }}`
to the: `https://{{ app name }}.fly.dev`, e.g.
`https://pr-9-felixxm-fly-pr-preview-example.fly.dev/`
(for PR number 9 in the repository called `pr-preview-example`).
## Using Postgres cluster
Using a dedicated staging database is a good practice for test environments. This gives
us more control and appropriate separation from the production environment which
eliminates a potential data leak vector.
If you don't have a [Postgres cluster](/unmanaged-postgres) specifically for
testing purposes you can create one with [`fly postgres create`](/flyctl/cmd/fly_postgres_create):
```bash theme={null}
fly postgres create --name pg-fly-pr-staging-preview
```
Once created, we can specify it in our action (`jobs → preview-app → steps → with`)
using the `postgres` input:
```yaml theme={null}
# .github/workflows/fly_pr_preview.yml
name: Start preview app
on:
pull_request:
types: [labeled, synchronize, opened, reopened, closed]
concurrency:
group: ${{ github.workflow }}-pr-${{ github.event.number }}
cancel-in-progress: true
permissions:
contents: read
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
jobs:
preview-app:
if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
runs-on: ubuntu-latest
name: Preview app
environment:
name: pr-${{ github.event.number }}
url: ${{ steps.deploy.outputs.url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy preview app
uses: superfly/fly-pr-review-apps@1.2.0
id: deploy
with:
postgres: pg-fly-pr-staging-preview # ← Added
region: waw
org: personal
```
With that in place, our staging Postgres cluster will be automatically attached to the
test app, which will make a `DATABASE_URL` environment variable available in the test
VM.
## Additional release steps
For performing additional release steps when deploying our staging environment, we can
use a custom [Fly.io TOML configuration file](/reference/configuration)
dedicated for staging with a release script
[to run before a deployment](/reference/configuration#run-one-off-commands-before-releasing-a-deployment).
First, make a copy of an existing configuration:
```bash theme={null}
mkdir staging
```
```bash theme={null}
cp fly.toml staging/fly_staging.toml
```
Next, create a script (`staging/post_deploy.sh`) to prepare our staging database:
```bash theme={null}
# staging/post_deploy.sh
#!/usr/bin/env bash
# Migrate database.
python /code/manage.py migrate
# Load fixtures with test data.
python /code/manage.py loaddata /code/staging/test_groups.json
```
Finally, we need to add `release_command` to the TOML configuration calling our script:
```toml theme={null}
# staging/fly_staging.toml
console_command = "/code/manage.py shell"
[build]
[env]
PORT = "8000"
[deploy]
release_command = "sh ./staging/post_deploy.sh" # ← Added.
...
```
and specify the custom TOML configuration in our action
(`jobs → preview-app → steps → with`) using the `config` input:
```yaml theme={null}
# .github/workflows/fly_pr_preview.yml
name: Start preview app
on:
pull_request:
types: [labeled, synchronize, opened, reopened, closed]
concurrency:
group: ${{ github.workflow }}-pr-${{ github.event.number }}
cancel-in-progress: true
permissions:
contents: read
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
jobs:
preview-app:
if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
runs-on: ubuntu-latest
name: Preview app
environment:
name: pr-${{ github.event.number }}
url: ${{ steps.deploy.outputs.url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Deploy preview app
uses: superfly/fly-pr-review-apps@1.2.0
id: deploy
with:
config: staging/fly_staging.toml # ← Added
postgres: pg-fly-pr-staging-preview
region: waw
org: personal
```
With this small effort we have staging database created on Fly.io!
🚨 Be aware that `release_command` is run on a temporary VM and cannot modify the
local storage or state. It's fine to run database operations but not to perform release
steps that attempt to modify a local storage, e.g. collecting static files. Such steps
should be added to the `Dockerfile`. For example, if we want to collect static files,
then add the `collectstatic` management command to the `Dockerfile`:
```docker theme={null}
ARG PYTHON_VERSION=3.10-slim-bullseye
FROM python:${PYTHON_VERSION}
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN mkdir -p /code
WORKDIR /code
COPY requirements.txt /tmp/requirements.txt
RUN set -ex && \
pip install --upgrade pip && \
pip install -r /tmp/requirements.txt && \
rm -rf /root/.cache/
COPY . /code
# ↓ Added ↓
RUN set -ex && \
python /code/manage.py collectstatic --noinput
EXPOSE 8000
CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "hello_pr_preview_example.wsgi"]
```
# Deploy an Existing Django App
Source: https://docs.fly.io/django/getting-started/existing
Move an existing Django app to Fly.io: configure settings, launch with fly launch, attach Postgres, and troubleshoot your first deploy.
If you have an existing Django app that you want to move over to Fly, this guide
walks you through the initial deployment process and shows you techniques you
can use to troubleshoot issues you may encounter.
## Configure Django
The official [How to Deploy Django](https://docs.djangoproject.com/en/stable/howto/deployment/) guide and [Django Deployment Checklist](https://docs.djangoproject.com/en/stable/howto/deployment/checklist/) are a good first step to ensure your app is ready for deployment.
The third-party package [WhiteNoise](https://whitenoise.evans.io/en/stable/) is recommended for serving static files in production as this requires [additional configuration](https://docs.djangoproject.com/en/stable/howto/static-files/deployment/).
A production web server must be installed, typically either [Gunicorn](https://docs.gunicorn.org/en/latest/install.html) or [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/). The database adapter [Psycopg](https://www.psycopg.org/psycopg3/docs/) is commonly used along with [dj-database-url](https://pypi.org/project/dj-database-url/) to establish a Django database connection via a `DATABASE_URL` environment variable.
Be sure to have generated an up-to-date `requirements.txt` file for any new packages added for deployment.
## flyctl
Fly.io has its own command-line utility for managing apps, [flyctl](/flyctl). If not already installed, follow the instructions on the [installation guide](/flyctl/install) and [log in to Fly](/getting-started/sign-up-sign-in).
## Provision Django and Postgres Servers
To configure and launch the app, use the command `fly launch` and follow the wizard. You can set a name for the app, choose a default region, launch and attach a Postgres database. You can also set up a Redis database though we will not be doing so in this example.
```bash cmd theme={null}
fly launch
```
```output output theme={null}
Scanning source code
Detected a Django app
Creating app in ~/django-existing-app
We're about to launch your Django app on Fly.io. Here's what you're getting:
Organization: Jane Smith (fly launch defaults to the personal org)
Name: django-existing-app (derived from your directory name)
Region: Ashburn, Virginia (US) (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM (most apps need about 1GB of RAM)
Postgres: (not requested)
Redis: (not requested)
? Do you want to tweak these settings before proceeding? Yes
Opening https://fly.io/cli/launch/mo1ootho9ualooghoch3iih6cha2shah ...
Waiting for launch data... Done
Created app 'django-existing-app' in organization 'personal'
Admin URL: https://fly.io/apps/django-existing-app
Hostname: django-existing-app.fly.dev
Set secrets on django-existing-app: SECRET_KEY
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app django-existing-app-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:44b698752cf113110f2fa72443d7fe452b48228aafbb0d93045ef1e3282360a6
Waiting for machine to start...
Machine 217811c53e0896 is created
==> Monitoring health checks
Waiting for 217811c53e0896 to become healthy (started, 3/3)
Postgres cluster django-existing-app-db created
Username: postgres
Password: lt5JoEIVons5INJ
Hostname: django-existing-app-db.flycast
Flycast: fdaa:3:9f45:0:1::2
Proxy port: 5432
Postgres port: 5433
Connection string: postgres://postgres:lt5JoEIVons5INJ@django-existing-app-db.flycast:5432
Save your credentials in a secure place -- you won't be able to see them again!
Connect to postgres
Any app within the Jane Smith organization can connect to this Postgres using the above connection string
Now that you've set up Postgres, here's what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user
Postgres cluster django-existing-app-db is now attached to django-existing-app
The following secret was added to django-existing-app:
DATABASE_URL=postgres://django_existing_app:lt5JoEIVons5INJ@django-existing-app-db.flycast:5432/django_existing_app?sslmode=disable
Postgres cluster django-existing-app-db is now attached to django-existing-app
Wrote config file fly.toml
[INFO] Python 3.10.12 was detected. 'python:3.10-slim-bullseye' image will be set in the Dockerfile.
Validating ~/django-existing-app/fly.toml
Platform: machines
✓ Configuration is valid
Your Django app is ready to deploy!
For detailed documentation, see https://fly.dev/docs/django/
```
## Dockerfile and fly.toml
The `fly launch` command creates two new files in the project that are automatically configured: `Dockerfile` and `fly.toml`.
The [Dockerfile](https://docs.docker.com/engine/reference/builder/) is essentially instructions for creating an image.
The [`fly.toml`](/reference/configuration) file is used by Fly.io to configure applications for deployment. Configuration of builds, environment variables, internet-exposed services, disk mounts and release commands go here.
## ALLOWED\_HOSTS & CSRF\_TRUSTED\_ORIGINS
The dedicated URL for your deployment will be `.fly.dev`. Update the [ALLOWED\_HOSTS](https://docs.djangoproject.com/en/stable/ref/settings/#allowed-hosts) and [CSRF\_TRUSTED\_ORIGINS](https://docs.djangoproject.com/en/stable/ref/settings/#csrf-trusted-origins) configurations with your `` to include it.
## Deploy Your Application
To deploy the application use the following command:
```bash theme={null}
fly deploy
```
This will take a few seconds as it uploads your application, verifies the app configuration, builds the image, and then monitors to ensure it starts successfully. Once complete, visit your app with the following command:
```bash theme={null}
fly apps open
```
If everything went as planned you will see your Django application homepage.
## Troubleshooting your initial deployment
Since this is an existing Django app, it is highly likely it might not boot because you probably need to configure secrets or other service dependencies. Let's walk through how to troubleshoot these issues so you can get your app running.
### View Log Files
If your application didn't boot on the first deploy, run `fly logs` to see
what's going on.
```bash theme={null}
fly logs
```
This shows the past few log file entries and tails your production log
files. [Additional flags](/flyctl/cmd/fly_logs) are available for filtering.
### Console
To SSH into your hosted Django application use the command `fly ssh console`. For example, to execute Django's `createsuperuser` command to log into the admin do the following:
```bash theme={null}
fly ssh console
# python manage.py createsuperuser
```
If you prefer, this can be run as one command instead:
```bash theme={null}
fly ssh console --pty -C 'python /code/manage.py createsuperuser'
```
> [!NOTE]
>
> The `--pty` flag tells the SSH server to run the command in a pseudo-terminal, which `createsuperuser` requires.
### Secrets
Secrets allow sensitive values, such as credentials and API keys, to be securely passed to your Django applications. You can set, remove, or list all secrets with the [fly secrets](/apps/secrets) command.
### Git
Deployments are initiated via the `fly deploy` command--git isn't needed to deploy to Fly.io. The advantage of this approach is that your git history will be clean and not full of `git push` commits such as occurs on other hosting platforms.
This also means that `.gitignore` files are not ignored. If you have secrets or other sensitive information in your git history, it is recommended to create a `.dockerignore` file and add the git repo there.
### Databases
Fly.io has a [Postgres offering](/unmanaged-postgres) to automate provisioning, maintenance, and snapshot tasks for your Postgres database, but it does not manage it. If you run out of disk space, RAM, or other resources on your Fly Postgres instances, you'll have to scale those virtual machines from the Fly CLI. If you prefer, you can instead connect to an [external fully-managed Postgres database](/postgres).
### Custom Domain & SSL Certificates
After you finish deploying your application to Fly.io and have tested it extensively, [read through the Custom Domain docs](/networking/custom-domain) and point your domain at Fly.
In addition to supporting [`CNAME` DNS records](/networking/custom-domain#cname-records), Fly.io also supports [`A` and `AAAA` records](/networking/custom-domain#a-and-aaaa-records-recommended) for those who want to point `example.com` (without the `www.example.com`) directly at Fly.
### flyctl Commands
The [flyctl CLI docs](/flyctl) have an extensive inventory of `fly` commands. Here are a few common commands especially if you are coming from another hosting service like Heroku.
| Task | Command |
| - | - |
| Log in | `fly login` |
| Launch an app | `fly launch` |
| Deployments | `fly deploy` |
| Open web dashboard | `fly dashboard` |
| SSH | `fly ssh console` |
| Tail log files | `fly logs` |
| Secrets | `fly secrets` |
| View releases | `fly releases` |
| Help | `fly help` |
# Deploy a Django App
Source: https://docs.fly.io/django/getting-started/index
Deploy a Django app to Fly.io. Build a sample app, generate a Dockerfile with fly launch, serve it with Gunicorn, and add Postgres.
In this guide we recreate and deploy [this Django application](https://github.com/fly-apps/hello-django) to demonstrate how quickly Django apps can be deployed to Fly.io!
## Initial Local Setup
Make sure that [Python](https://www.python.org/) is already installed on your computer along with a way to create virtual environments.
This allows you to run your project locally, and test that it works, before deploying it to Fly.io.
> We recommend the latest [supported versions](https://devguide.python.org/versions/#supported-versions) of Python.
Create a folder for your project. Here we'll call it `hello-django`. Enter the folder with `cd`:
```bash theme={null}
mkdir hello-django && cd hello-django
```
### Virtual Environment
For this guide, we use [venv](https://docs.python.org/3/library/venv.html#module-venv) but any of the other popular choices such as [Poetry](https://python-poetry.org/), [Pipenv](https://github.com/pypa/pipenv), or [pyenv](https://github.com/pyenv/pyenv) work too.
```shell theme={null}
# Unix/macOS
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $
```
```powershell theme={null}
# Windows (PowerShell)
> python -m venv .venv
> .venv\Scripts\Activate.ps1
(.venv) ...>
```
> [!NOTE]
>
> From this point on, the commands won't be displayed with `(.venv)` but we assume you have your Python virtual environment activated.
### Install Django
With your virtual environment **activated**, install the [latest](https://www.djangoproject.com/download/#supported-versions) version of Django using [pip](https://pip.pypa.io/en/stable/):
```bash theme={null}
python -m pip install Django
```
### Create a Django Project
Inside the `hello-django` folder, create a Django project named `hello_django`:
```bash theme={null}
django-admin startproject hello_django .
```
> Don't forget the `.` at the end. It's crucial because it tells the script to create the Django project directory structure in the current directory, our folder `hello-django`.
Note that by convention, we name Django projects using `snake_case`: words written in lowercase with spaces replaced by underscore (`_`). Hyphens (`-`) are not valid identifiers and you might get this error message:
```terminal theme={null}
'hello-django' is not a valid project name. Please make sure the name is a valid identifier.
```
By this point, our project structure should look like this:
```terminal theme={null}
hello-django/
|-- .venv
|-- hello_django/
| |-- __init__.py
| |-- asgi.py
| |-- settings.py
| |-- urls.py
| |-- wsgi.py
|-- manage.py
```
### Create and Map a URL to a View
Now let's configure a basic view that returns the text, `Hello, Fly!` by updating the
[`hello_django/urls.py`](https://github.com/fly-apps/hello-django/blob/main/hello_django/urls.py) file:
```python theme={null}
# hello_django/urls.py
from django.contrib import admin
from django.http import HttpResponse
from django.urls import path
# ↓ New basic view returning "Hello, Fly!" ↓
def hello(request):
return HttpResponse("Hello, Fly!")
urlpatterns = [
path("admin/", admin.site.urls),
path("", hello, name="hello"), # ← Added!
]
```
### Run migrations
As part of Django's core functionality, some existing apps are included by default to provide you with out-of-the-box features. Some of those apps require their own database tables.
To initialize the local database and set up those tables, run the `migrate` command:
```bash theme={null}
python manage.py migrate
```
### Start the Web Server
Now `runserver` to start up Django's local web server:
```bash theme={null}
python manage.py runserver
```
If you open `http://127.0.0.1:8000/` in your web browser it now displays the text `Hello, Fly!`.
## Django Deployment Checklist
By default, Django is configured for local development. The [How to Deploy Django](https://docs.djangoproject.com/en/stable/howto/deployment/) and [Django Deployment Checklist](https://docs.djangoproject.com/en/stable/howto/deployment/checklist/) guide list the various steps required for a secure deployment.
> You can also find a complete guide [Deploying Django to Production](https://fly.io/django-beats/deploying-django-to-production/) in our [Django Beats Blog](https://fly.io/django-beats/).
However, for demonstration purposes, we can take some shortcuts.
First, in the [`hello_django/settings.py`](https://github.com/fly-apps/hello-django/blob/main/hello_django/settings.py) file update the `ALLOWED_HOSTS` configuration to accept
a host on which it's deployed. Use the [`FLY_APP_NAME`](/machines/runtime-environment#fly_app_name)
environment variable for that:
```python theme={null}
# hello_django/settings.py
import os
APP_NAME = os.environ.get("FLY_APP_NAME")
ALLOWED_HOSTS = [f"{APP_NAME}.fly.dev"] # ← Updated!
```
Second, install [Gunicorn](https://gunicorn.org/) as our [production server](https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver):
```bash theme={null}
python -m pip install gunicorn
```
Third, create a [`requirements.txt`](https://github.com/fly-apps/hello-django/blob/main/requirements.txt) file listing all the packages in the current Python virtual environment:
```bash theme={null}
pip freeze > requirements.txt
```
That's it! We're ready to deploy on Fly.io.
## flyctl
Fly.io has its own command-line utility for managing apps, [flyctl](/flyctl). If not already installed, follow the instructions on the [installation guide](/flyctl/install) and [log in to Fly.io](/getting-started/sign-up-sign-in).
## Configure and Deploy your Fly App
To configure and launch the app, run the `fly launch` command and follow the wizard. You can set a name for the app and choose your primary region. You can also choose to launch and attach a Postgres database and/or a Redis database though we are not using either in this example.
```bash cmd theme={null}
fly launch
```
```output output theme={null}
Scanning source code
Detected a Django app
Creating app in /flyio/hello-django
We're about to launch your Django app on Fly.io. Here's what you're getting:
Organization: Joe Doe (fly launch defaults to the personal org)
Name: hello-django (derived from your directory name)
Region: Amsterdam, Netherlands (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM (most apps need about 1GB of RAM)
Postgres: (not requested)
Redis: (not requested)
? Do you want to tweak these settings before proceeding? Yes
Opening https://fly.io/cli/launch/mo1ootho9ualooghoch3iih6cha2shah ...
Waiting for launch data... Done
Created app 'hello-django' in organization 'personal'
Admin URL: https://fly.io/apps/hello-django
Hostname: hello-django.fly.dev
Set secrets on hello-django: SECRET_KEY
Wrote config file fly.toml
[INFO] Python 3.10.12 was detected. 'python:3.10-slim-bullseye' image will be set in the Dockerfile.
Validating /flyio/hello-django/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
```
This creates two new files in the project that are automatically configured: a [`Dockerfile`](https://github.com/fly-apps/hello-django/blob/main/Dockerfile) and [`fly.toml`](https://github.com/fly-apps/hello-django/blob/main/fly.toml) file to configure applications for deployment.
To deploy the application use the following command:
```bash theme={null}
fly deploy
```
This will take a few seconds as it uploads your application, verifies the app configuration, builds the image, and then monitors to ensure it starts successfully. Once complete visit your app with the following command:
```bash theme={null}
fly apps open
```
YAY! You are up and running! Wasn't that easy?
## Recap
We started with an empty directory and in a matter of minutes had a running Django application deployed to the web. A few things to note:
* Your application is running on a Virtual Machine that was created based on the `Dockerfile` image.
* The `fly.toml` file controls your app configuration and can be modified as needed.
* `fly dashboard` can be used to monitor and adjust your application. Pretty much anything you can do from the browser window you can also do from the command line using [`fly` commands](/flyctl). Try `fly help` to see what you can do.
Now that you have seen how to deploy a simple Django application, it is time to move on to [Existing Django Apps](/django/getting-started/existing) that feature static files and a Postgres database.
# Django on Fly.io
Source: https://docs.fly.io/django/index
Run Django on Fly.io: fly launch detects and deploys Django apps. Start with the starter app guide, move an existing project off Heroku, or go advanced.
Fly.io is a great place to run your Django application. `fly launch` can detect, configure and deploy Django apps.
If you have questions or comments, please reach out on the [Django category of the Community Board](https://community.fly.io/c/django/) for Django on Fly.io specific discussions. The general [Community Board](https://community.fly.io/) is great for all things Fly.io: networking, databases, regions, and more. You can use the `django` tag as well to help interested people find a topic more easily.
## [Getting Started](./getting-started)
Run through the [Starter Django App](./getting-started/) guide to get a feel for what it's like to deploy to Fly.io. You'll have a Django app running in just a few minutes.
## [Existing Django Apps](./getting-started/existing/)
The [Existing Django Apps](./getting-started/existing/) guide walks through deploying a production-ready project to Fly.io. If you are coming from another hosting platform like Heroku there are additional tips for managing the migration.
## [Advanced Guides](./advanced-guides)
# Elixir: Advanced Guides
Source: https://docs.fly.io/elixir/advanced-guides
Advanced Elixir on Fly.io: GitHub Actions CI/CD, Observer and Livebook against production, clustering from your laptop, and SQLite3 as a database.
Elixir can do some pretty neat things by inheriting abilities from the Erlang
Virtual Machine (BEAM). This provides a set of advanced guides for Elixir
applications running on Fly.
## GitHub Actions for Elixir CI/CD
Setup GitHub CI/CD for your Elixir project.
## Connecting Observer to Your App in Production
Guide for connecting a locally running Observer instance to your application running on Fly.
## Connecting Livebook to Your App in Production
Guide for connecting a locally running Livebook to your application running on Fly.
## Do Interesting Things with Livebook and Your Production App
Guide shows interesting things we can do when Livebook connects to our deployed app running on Fly.
## Easy Clustering from Home to Fly.io
Guide for connecting a locally running Elixir application to another application running on Fly.
## SQLite3
This guide shows you how to use SQLite3 as your database
# Easy Clustering from Home to Fly.io
Source: https://docs.fly.io/elixir/advanced-guides/clustering-from-home-to-your-app-in-fly
Cluster a local Elixir node with an Elixir app running on Fly.io over WireGuard, with cookie and IPv6 setup plus a bash script that automates the connection.
This explains how to cluster a locally running Elixir application with another Elixir application running on Fly.io. Additionally, a [**bash script** (named `cluster_with_remote`)](https://gist.github.com/brainlid/9e02e95f7d9c65a23312a4df95094d2a) is provided to automate the process of starting the local node and clustering it with the server.
Here we cover *why* we might want to do this, *what* is required to make it work, and *how* to make it happen.
## Why cluster a local application with the one on the server?
Besides being really cool that we can do this, there are some practical reasons as well.
### Develop and debug a distributed application
Building a globally distributed application can be challenging to model locally. With [Fly.io Regions](/reference/regions), we can deploy our cluster-aware application where it makes sense. Then, our local application joins the global cluster, giving us a close-up view of how the application behaves in a truly globally distributed environment.
## How it works
Elixir supports [clustering](/elixir/the-basics/clustering) multiple running Elixir applications together (thanks to [Erlang](https://www.erlang.org/doc/reference_manual/distributed.html)), even while running on separate machines.
Fly.io makes creating a [WireGuard VPN tunnel](/networking/private-networking#private-network-vpn) between your local machine and your Fly.io organization easy.
This means we can connect and cluster our locally running Elixir application with an Elixir application deployed at Fly.io. There are some special configuration requirements needed to make this possible, but we cover it all here.
### Prerequisites
There are a few prerequisites to consider when clustering a local Elixir application to one running in Fly.io. We'll provide tools or instructions to help with each.
1. A [WireGuard connection](/networking/private-networking#private-network-vpn) must be setup and open from your local machine to Fly.io
2. The same version of Elixir and Erlang OTP must be running on both ends.
3. The locally running Elixir application needs the Erlang COOKIE used on the server.
4. We need the full node name of the Elixir node running on the server (**NOTE:** The bash script provided later does this for you.)
5. The local Elixir application must be started with IPv6 networking support enabled. By default it does not.
### Setting the Erlang cookie
The [recommended way for setting the Erlang COOKIE](/elixir/the-basics/clustering#making-the-cookie-changes) value in your deployed application is to set an ENV named `RELEASE_COOKIE`. This gives the server a stable, predictable cookie value. In order for the nodes to connect, they need the same cookie.
When [set in the ENV through Fly.io](/flyctl/cmd/fly_config_env), we can read it from the application's information. The script does this so we don't have to do anything else.
### Starting with IPv6 support
Fly.io uses an IPv6 network internally for private IPs. The BEAM needs IPv6 support to be enabled explicitly. On the server, that’s taken care of through a Dockerfile. Locally, however, it needs to be enabled so the local application can cluster with the remote node.
The issue is, if IPv6 support is enabled globally, like in a `.bashrc` file, then setting it in the `cluster_with_remote` script essentially flips it OFF. If NOT set globally, then it should be set in the script. Choose the approach that best fits your situation.
When set globally in your `.bashrc` file (recommended), it looks like this:
```
export ERL_AFLAGS="-kernel shell_history enabled -proto_dist inet6_tcp"
```
This one includes the added benefit of turning on shell history in IEx. Yay!
If not set globally, it can be set in the script command like this:
```
iex --erl "-proto_dist inet6_tcp" --sname local [...]
```
Where the `--erl "-proto_dist inet6_tcp"` portion is the key.
### Hidden by default: A note about production systems
Joining your local application to a production cluster in a public way may result in other application sending work, tasks, or executing processes in your local node, depending on the behavior of your specific application. This may not be what you want!
For this reason, the script uses the `--hidden` option to hide the local node from the rest of the cluster.
To see the ***all*** the connected nodes, including any hidden ones, use:
```elixir theme={null}
Node.list(:hidden)
```
> [!NOTE]
>
> If you want the local Elixir application to be *fully visible* to the rest of the cluster, remove the `--hidden` argument in the bash script.
## Bash script file
Create [this file](https://gist.github.com/brainlid/9e02e95f7d9c65a23312a4df95094d2a) locally in your root of your Elixir project.
```sh expandable theme={null}
#!/bin/bash
# After opening a WireGuard connection to your Fly.io network, copy this file
# into the directory for your Elixir application. Run the script to start the
# local Elixir project and cluster it to an application running on Fly.io.
# In order for this to work:
# - Your wireguard connection must be up.
# - When run from a directory with a `fly.toml` file, `flyctl` command is used
# to access information about the application.
# - Set the ENV `CLUSTER_APP_NAME` to specify a different hosted app name.
# - Set the ENV `RELEASE_COOKIE` to override the Erlang cookie used for
# clustering. It uses the the value from the deployed app if it is set there.
# - Run the script.
set -e
if ! command -v jq &> /dev/null; then
echo "jq is not installed. Please install it before running this script. It is a command-line JSON processor."
exit 1
fi
# Check if CLUSTER_APP_NAME is set and use it if found
if [[ -n $CLUSTER_APP_NAME ]]; then
# Use the override app_name from the ENV
json_data=$(fly status --app ${CLUSTER_APP_NAME} --json)
else
# Use the app_name for the current app
json_data=$(fly status --json)
fi
# Use an explicit RELEASE_COOKIE value if provided
if [ -n "$RELEASE_COOKIE" ]; then
release_cookie=$RELEASE_COOKIE
else
# Extract the RELEASE_COOKIE value from the deployed app
release_cookie=$(echo "$json_data" | jq -r '.Machines[] | select(.state == "started") | .config | .env | .RELEASE_COOKIE' | head -n 1)
if [ $release_cookie == "null" ]; then
echo "The deployed application did not set RELEASE_COOKIE ENV. If the cookie is static on the server, provide it locally through RELEASE_COOKIE."
exit 1
fi
fi
# Extract the app_name
app_name=$(echo "$json_data" | jq -r '.Name')
# Extract private_ip for the first started machine
private_ip=$(echo "$json_data" | jq -r '.Machines[] | select(.state == "started") | .private_ip' | head -n 1)
# Extract image_ref tag hash for the first started machine
image_tags=$(echo "$json_data" | jq -r '.Machines[] | select(.state == "started") | .image_ref.tag | sub("deployment-"; "")' | head -n 1)
if [ -z "$private_ip" ]; then
echo "No instances appear to be running at this time."
exit 1
fi
# Assemble the full node name
full_node_name="${app_name}-${image_tags}@${private_ip}"
echo Attempting to connect to $full_node_name
# IMPORTANT:
# ==========
# Fly.io uses an IPv6 network internally for private IPs. The BEAM needs IPv6
# support to be enabled explicitly.
#
# The issue is, if it's enabled globally like in a `.bashrc` file, then setting
# it here essentially flips it OFF. If not set globally, then it should be set
# here. Choose the version that fits your situation.
#
# It's the `--erl "-proto_dist inet6_tcp"` portion.
# export ERL_AFLAGS="-kernel shell_history enabled -proto_dist inet6_tcp"
# Toggles on IPv6 support for the local node being started.
# iex --erl "-proto_dist inet6_tcp" --sname local --cookie ${release_cookie} --hidden -e "IO.inspect(Node.connect(:'${full_node_name}'), label: \"Node Connected?\"); IO.inspect(Node.list(:hidden), label: \"Connected Nodes\")" -S mix phx.server
# Does NOT toggle on IPv6 support, assuming it is enabled some other way.
iex --sname local --cookie ${release_cookie} --hidden -e "IO.inspect(Node.connect(:'${full_node_name}'), label: \"Node Connected?\"); IO.inspect(Node.list(:hidden), label: \"Connected Nodes\")" -S mix phx.server
```
Make the script file executable.
```
chmod +x cluster_with_remote
```
## Script usage
With the perquisites out of the way, we'll use the `cluster_with_remote` script to start our local Elixir application. The script automates much of what needs to be done to make the process work smoothly. Feel free to customize the script as needed.
The script is designed to be copied into a project with little to no modification required. The only expected customizations are handled through ENV values that can be controlled per-project.
There are two primary ways to use the script:
1. Cluster the local application to the same project deployed at Fly.io. This is the same application running in two places.
2. Cluster a local application to a difference project deployed at Fly.io. The deployed application's name must be provided.
### Cluster to the same application running on the server
The simplest variation is when we cluster our local application to a deployed version of itself. The `fly.toml` file in the directory with a copy of the `cluster_with_remote` is for the deployed application.
To do this, just execute the script:
```
./cluster_with_remote
```
The script outputs if the clustering connection succeeded and prints the names of the connected nodes.
### Cluster to a different application running on the server
There are times when the local application is connecting to a *different* application than where the script is running from.
To do this, we need to tell the script the Fly.io app name of the application we want to connect with. It can be done like this:
```
CLUSTER_APP_NAME=server-app-name ./cluster_with_remote
```
This can also be set as a project-specific ENV using a tool like [direnv](https://direnv.net/) or [dotenv](https://www.dotenv.org/). The custom app name to cluster with can be written to env file and then running the script just works.
```
./cluster_with_remote
```
If the Erlang cookie of the deployed application is not set using the recommended `RELEASE_COOKIE` ENV setting, it can still be provided to the script using a local ENV named `RELEASE_COOKIE`. See the script for details.
Now you're *really* doing distributed Elixir!
## Summary
When we couple Elixir's clustering ability with Fly.io's networking, VPN, and API discoverability, we can easily cluster a locally running Elixir application with a deployed Elixir application. This makes it easy to develop and debug distributed applications.
The ready-to-use script automates much of the process.
# Connecting Livebook to Your App in Production
Source: https://docs.fly.io/elixir/advanced-guides/connect-livebook-to-your-app
Connect a local Livebook to your production Elixir app on Fly.io over WireGuard, covering Elixir version requirements, the node name, and the shared cookie.
[Livebook](https://github.com/elixir-nx/livebook) is an exciting advancement in the Elixir community. It was created for the Elixir machine learning library [Nx](https://github.com/elixir-nx/nx). You can think of it as Elixir's version of [Jupyter Notebooks](https://jupyter.org/).
Livebook ends up being very flexible and powerful. Because Elixir nodes can easily be clustered together, you can run Livebook on your local machine, connect it to your Elixir servers, and do some really interesting things.
Visualizing the setup:
## Requirements
There are a few requirements for this to work.
* **Livebook requires Elixir version 1.14.2 or higher**. Livebook runs locally on your machine, so that part is easy to control. However, the **server** side needs to have the same version of Elixir as well. When Livebook connects to the server, it loads some code into the server environment as well. So your server version of Elixir also needs to be 1.14.2 or higher. I recommend making your local version be the same as the production version to reduce potential problems.
* **Known cookie value on your server**. Livebook needs to know the cookie value used on the server. Follow [this guide to give your app a static cookie value](/elixir/the-basics/clustering#the-cookie-situation).
* **WireGuard setup on your local machine**. Follow the Fly.io [Private Network VPN](/networking/private-networking#private-network-vpn) guide to walk through that.
> [!NOTE]
>
> **Elixir Version Tip**
>
> Check the version of Elixir and Erlang specified in your Dockerfile for the deploy. That's the actual version of Elixir your server is running.
>
> You may need to update the version in your Dockerfile and deploy the application before continuing.
## Installing Livebook
There are several ways to [install and run Livebook](https://livebook.dev/#install). [Livebook Desktop](https://news.livebook.dev/introducing-the-livebook-desktop-app-4C8dpu) is really slick and deploying a [Livebook on Fly](https://fly.io/launch/livebook) is super easy.
Livebook Desktop can be installed as an app on Mac and Window. For Linux, we can pull the source or install it as an [Escript](https://github.com/elixir-nx/livebook#escript).
### Using Livebook Desktop
When running [Livebook Desktop](https://livebook.dev/#install), Livebook will invoke on boot a file named `~/.livebookdesktop.sh` on macOS or `%USERPROFILE%\.livebookdesktop.bat` on Windows. This file can be modified to set environment variables used by Livebook, such as:
```
export LIVEBOOK_DISTRIBUTION=name
export ERL_AFLAGS="-proto_dist inet6_tcp"
```
### Install as an Escript
This section assumes you are already setup for [local Elixir development](https://elixir-lang.org/install.html).
Following the Livebook [Escript README section](https://github.com/elixir-nx/livebook#escript), this is how to install Livebook.
```
mix do local.rebar --force, local.hex --force
mix escript.install hex livebook
```
Then locally launching Livebook is:
```bash theme={null}
livebook server
```
When connecting to a Fly.io app, we can pass the needed ENV value like this:
```bash theme={null}
LIVEBOOK_DISTRIBUTION=name ERL_AFLAGS="-proto_dist inet6_tcp" livebook server
```
> [!NOTE]
>
> **Upgrading an Escript Installed Livebook**
>
> When upgrading your Livebook version while using [asdf](https://asdf-vm.com/) for managing your Elixir and Erlang installs, then here's a helpful tip:
>
> * Remove the existing `livebook` shim: `rm ~/.asdf/shims/livebook`
> * Install the new version: `mix escript.install hex livebook`
> * Re-shim to get the command: `asdf reshim elixir`
> * Verify you have the upgrade: `livebook --version`
### ENV Values
The `ERL_AFLAGS` settings is passed to the BEAM on startup. The `LIVEBOOK_DISTRIBUTION=name` setting configures Livebook to run with a fully qualified name and to be ready for clustering.
The Livebook README page [documents the supported ENV values](https://github.com/livebook-dev/livebook#environment-variables).
## Running Livebook
Once started, Livebook generates a link in the console you use to connect to it. The Livebook link in the console looks something like this:
```
[Livebook] Application running at http://localhost:8080/?token=uzbwvvpexj7mdftwjilezi4qaygt2bfc
```
With Livebook running locally, let's get started!
### Create or Load a Notebook
Let's create a new notebook to test the remote connection.
Create a new notebook titled, "Remote Connect Test" and add a Section.
On the left side, click the "Runtime settings" as pictured below:
Before we can run any Elixir code, we need to connect to the remote server. To connect to the remote server, we will "Configure" the server we want to connect to.
Now we use "Attached Node" to connect to an existing Elixir node.
With the "Attached Node" mode selected, we need to provide 2 additional pieces of information before we can connect.
1. **Name** - Our node's name
2. **Cookie** - The cookie value used by the deployed server
### Getting the Node Name
Let's see how to get the node name and private IP address needed. To list your deployed apps:
```bash cmd theme={null}
fly apps list
```
```output output theme={null}
...
icy-leaf-7381 personal running 49m26s ago
```
Get the private IPv6 address for the desired app.
```bash cmd theme={null}
fly ips private --app icy-leaf-7381
```
```output output theme={null}
ID REGION IP
44040812 lax fdaa:0:1da8:a7b:bf88:71ca:52:2
```
Combine the app name with the private IPv6 address to get the full node name. For our example here, it would be:
```
icy-leaf-7381@fdaa:0:1da8:a7b:bf88:71ca:52:2
```
### Using the Cookie
We also need the cookie used by the server. See [this guide](/elixir/the-basics/clustering) for setting that up for your app.
### Attaching to a Fly Server
**REMEMBER:** At this point, make sure your WireGuard connection to Fly is up!
When you have the name and cookie, enter those and "**Connect**" to the server.
**REMEMBER:** Each time we deploy our apps, the private IP will change. We need to get the current IP before we can connect again.
Once connected, we have code completion available in the Elixir cells for the app we are connected to. The [HelloElixir app](https://github.com/fly-apps/hello_elixir) doesn't have anything useful to run so we can just prove to ourselves that our code is being executed remotely.
Add the following code to a Livebook Elixir cell and execute it.
```elixir theme={null}
require Logger
Logger.info("Testing logger")
```
Now, check the logs of the production app. You should find your logger output there!
```bash cmd theme={null}
fly logs
```
```output output theme={null}
...
2021-06-21T22:47:21.415426797Z app[44040812] sjc [info] 22:47:21.414 [info] Testing logger
```
### Success!
We successfully executed Elixir code through Livebook connected to a remote server running on Fly.io! The code was actually executed on the server!
Looking for things to do now that you're connected? Check out: [Do Interesting Things with Livebook and Your Production App](/elixir/advanced-guides/interesting-things-with-livebook)
## Troubleshooting Checklist
If you have problems connecting, here are a few things to check:
* Ensure your WireGuard connection is up.
* Check that the version of Elixir on the server is 1.14.2 or higher.
* Check that the version of Elixir declared in the project's Dockerfile.
* Check that the version of Elixir on the client matches the server.
* If you re-deployed, you need to get the new IP address for the deployed app.
* Check the server logs. Failed connections may still log errors at the server.
* Check the local livebook logs. Failed connections may log helpful messages.
# Connecting Observer to Your App in Production
Source: https://docs.fly.io/elixir/advanced-guides/connect-observer-to-your-app
Run Erlang Observer locally and connect it to a production Elixir app on Fly.io over a WireGuard tunnel to inspect and kill processes, with a helper script.
Elixir, Erlang, and really just the BEAM has a slick feature called "[Observer](https://elixir-lang.org/getting-started/debugging.html#observer)". It's a powerful UI that connects to a running Elixir or Erlang node and let's you "observe" what's going on inside. It has some limited ability to modify things as well, most notably you can kill running processes. This can help when something is misbehaving or you just want to play "chaos monkey" and kill parts of the system to see how it recovers.
This shows a process tree for the application. Using this I can inspect individual processes or even kill them!
One very cool way to run Observer is to run it on your local machine (which has the ability to display the UI) and connect to a production server (with no windowing UI available) and "observe" it from a distance. So yeah... have a problem in production? Not sure what's going on? You can literally tunnel in, crack the lid and poke, prod, and peek around to see what's going on.
Next we'll cover how your project can support this feature and see how to do it on Fly.io!
## What We Will Do
Fly.io natively supports [WireGuard](https://www.wireguard.com/), Jason Donenfeld's amazing VPN protocol. If you’ve ever lost hours of your life trying to set up an IPSec VPN, you’ll be blown away by how easy WireGuard is. It’s so flexible and performant that Fly uses it as our network fabric. And it’s supported on [every major platform](https://www.wireguard.com/install/), including macOS, iOS, Windows, and Linux. What that means for you is that if your app runs on Fly, you can open a secure, private, direct connection from your dev machine to your production network, in less time than it took me to write this paragraph. Cool, right?
This is what we're going to do.
We will bring up a secure WireGuard tunnel that links to your servers on Fly. In this graphic, there are two `my_app` Elixir nodes clustered together running on Fly.
From the local machine, we can open an IEx terminal configured to **join** that cluster of remote Elixir nodes. Our local machine supports running Observer and drawing the UI. We use our local observer to talk to the remote nodes in the cluster!
Let's do it. This will be fun!
## Sharing a Cookie
The deployed servers running on Fly need a predictable, stable, known cookie value used for allowing nodes to join each other to create a cluster. This is required for your local node to be able to connect to the remote nodes. Your local node needs the cookie value too!
See [this guide on creating a static cookie value](/elixir/the-basics/clustering#the-cookie-situation) in your Elixir project.
## WireGuard Tunnel
Setup WireGuard on your local machine. Follow the Fly.io [Private Network VPN](/networking/private-networking#private-network-vpn) guide to walk through that.
## Connecting to Production
To make the connection, there are several steps in the process. We'll create a short script to automate the process for us!
### Knowing the Cookie
Our script needs to know the cookie value. The easiest way to do this and keep the script generic is to set the value in the ENV. This lets us copy the script unchanged to multiple projects.
To help manage project-specific ENV values, I like using [direnv](https://direnv.net/). When changing into a directory with an `.envrc` file, it loads those values into my ENV, when I leave that directory, it unloads them. This means I can set a COOKIE value (or other config) specific to each project and it works great.
You don't have to use a tool like [`direnv`](https://direnv.net/) though. The `./observer` script file can be customized to set the COOKIE value explicitly if you prefer that approach. Refer to the full [script file here](https://github.com/fly-apps/hello_elixir/blob/main/observer#L14) in the comments to see how you can do that.
### Script File
This is a bash script to kick off a correctly configured local IEx session, connect a new local node to the remote cluster, and start Observer.
Here's a copy of the **[`observer`](https://github.com/fly-apps/hello_elixir/blob/main/observer) script file**.
This should work fine on Linux and MacOS. On Windows, if you are using [WSL2](https://docs.microsoft.com/en-us/windows/wsl/install-win10) then it will work because it's Linux. Otherwise, refer to the manual steps outlined below.
After creating the file locally, make the script file executable:
```bash theme={null}
chmod +x observer
```
Execute the script:
```bash cmd theme={null}
./observer
```
```output output theme={null}
Attempting to connect to hello-elixir-01HR8CXEYKQ9RXYQFEWSE5PTE0@fdaa:0:1da8:a7b:a160:73d2:f48b:2
Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit]
Node Connected?: true
Connected Nodes: [:"hello-elixir-01HR8CXEYKQ9RXYQFEWSE5PTE0@fdaa:0:1da8:a7b:a160:73d2:f48b:2"]
...
Interactive Elixir (1.16.1) - press Ctrl+C to exit (type h() ENTER for help)
```
When observer first opens, it might looks something like this:
Notice that the window title shows `my_remote@...`? This means it's showing the stats of my local IEx node that isn't actually running any of my code. So this data isn't very interesting yet.
If everything worked and it's connected, under the Nodes menu you should see the connected remote node.
When the remote node is selected, then all the stats and information changes to reflect what's going on in the selected node.
It worked! I'm seeing the information for the production node!
### Success!
Let's review briefly what was accomplished.
* I setup a WireGuard tunnel from my personal computer into my private Fly network.
* I started a local Elixir node that shares the same cookie value.
* My local node connected over WireGuard to the production cluster.
* I launched Observer.
Now, using WireGuard and this script, I can easily launch Observer and observe any node in the cluster!
## Disconnecting
When done, close Observer. It leaves you with an open IEx shell that is still connected to the remote cluster. You can safely CTRL+C, CTRL+C to exit it.
At this point you can shutdown your WireGuard connection as well if desired.
## Tips and Troubleshooting
The script is a simple tool to make it easy to launch observer and connect to the cluster. It doesn't diagnose or handle all the things that can fail. For instance, if your WireGuard connection isn't up, it just won't find the server but it also won't complain. If you encounter issues, you can go through the manual steps below to help diagnose any problems.
In order for everything to work, here's the checklist overview:
* Your wireguard connection must be up.
* Your application defines a release that specifies the cookie value to use.
* The local COOKIE value must be the same as the cookie value used in production.
* Observer needs to be working in your local environment. That requires WxWidget support in your Erlang install.
### Networking and the BEAM
Fly.io uses an IPv6 network internally for private IPs. The BEAM needs IPv6 support to be enabled explicitly. That's taken care of for the server through the Dockerfile. Locally, however, it needs to be enabled as well so the machine running Observer can actually *connect* to the remote node.
The issue is, if IPv6 support is enabled globally, like in a `.bashrc` file, then setting it in the `observer` script essentially flips it OFF. If NOT set globally, then it should be set in the script. Choose the version that fits your situation by modifying the script.
It's the `--erl "-proto_dist inet6_tcp"` portion.
Example:
```
iex --erl "-proto_dist inet6_tcp" --sname my_remote --cookie ${COOKIE} -e "IO.inspect(Node.connect(:'${FULL_NODE_NAME}'), label: \"Node Connected?\"); IO.inspect(Node.list(), label: \"Connected Nodes\"); :observer.start"
```
Versus without the `--erl` option:
```
iex --sname my_remote --cookie ${COOKIE} -e "IO.inspect(Node.connect(:'${FULL_NODE_NAME}'), label: \"Node Connected?\"); IO.inspect(Node.list(), label: \"Connected Nodes\"); :observer.start"
```
### Manual Script Steps
If you encounter issues, this can help you diagnose what's going on. The script automates 4 things.
1. Getting the cookie value from the ENV - make sure the correct cookie value is either available in the ENV or explicitly set in the script.
2. Uses the `fly status` command to get the app name, Docker image ref, and the first private IP. This is used to build the fully qualified node name. You only need one IP address. Once you join to any node you are introduced and connected to all of them.
3. Set up a local node and executes multiple commands.
4. It runs a command like `Node.connect(:'APP_NAME-IMAGE_REF@IPv6_ADDRESS')` to connect to the remote node. It returns `true` when it succeeds or `false` when it fails. The app name and the IP address used to make up the node's name are assembled from the previous steps.
5. Launch observer with the command `:observer.start`. If this fails, check the other tip for WxWidgets.
To do it manually, once you get the IP address, you can customize the following command to launch Observer.
```
iex --erl "-proto_dist inet6_tcp" --sname my_remote --cookie ${YOUR-COOKIE-VALUE} -e "IO.inspect(Node.connect(:'${FULL_NODE_NAME}'), label: \"Node Connected?\"); :observer.start"
```
You need to substitute in your `YOUR-COOKIE-VALUE` value, the `FULL_NODE_NAME`.
### WxWidgets Support
If you are using [asdf-vm](https://asdf-vm.com/) for managing your Elixir and Erlang versions, check out the [Erlang plugin's documentation](https://github.com/asdf-vm/asdf-erlang) for getting WxWidget support in your Erlang environment. This is required for using Observer.
To test if WxWidgets and observer is working correctly on you local machine, try the following.
Start IEx:
```
iex
```
Start observer:
```
:observer.start
```
It should bring Observer running on your machine. It won't be connected to
anything else, but it verifies that the libraries and dependencies are working.
# GitHub Actions for Elixir CI/CD
Source: https://docs.fly.io/elixir/advanced-guides/github-actions-elixir-ci-cd
Set up a GitHub Actions CI workflow for an Elixir project with a Postgres service, dependency caching, and retries, then auto-deploy it to Fly.io.
You want the benefits of modern CI/CD for your Elixir projects? You're in the
right place!
## Continuous Integration (CI)
Continuous integration is a software development practice where developers
frequently merge code changes into a central repository. Automated builds and
tests are run to assert the new code's correctness before integrating the
changes into the main development branch. The goal is to find and correct bugs
faster, improve software quality and enable software releases to happen
faster.
To get started with GitHub Actions in your project, let's create a "test"
workflow. To do this, create this path and file in the root of your project:
`.github/workflows/elixir.yaml`
Let's look at a sample file. Comments are included to explain and document what
we're doing and why.
```yaml theme={null}
name: Elixir CI
# Define workflow that runs when changes are pushed to the
# `main` branch or pushed to a PR branch that targets the `main`
# branch. Change the branch name if your project uses a
# different name for the main branch like "master" or "production".
on:
push:
branches: [ "main" ] # adapt branch for project
pull_request:
branches: [ "main" ] # adapt branch for project
# Sets the ENV `MIX_ENV` to `test` for running tests
env:
MIX_ENV: test
permissions:
contents: read
jobs:
test:
# Set up a Postgres DB service. By default, Phoenix applications
# use Postgres. This creates a database for running tests.
# Additional services can be defined here if required.
services:
db:
image: postgres:12
ports: ['5432:5432']
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
runs-on: ubuntu-latest
name: Test on OTP ${{matrix.otp}} / Elixir ${{matrix.elixir}}
strategy:
# Specify the OTP and Elixir versions to use when building
# and running the workflow steps.
matrix:
otp: ['25.0.4'] # Define the OTP version [required]
elixir: ['1.14.1'] # Define the elixir version [required]
steps:
# Step: Setup Elixir + Erlang image as the base.
- name: Set up Elixir
uses: erlef/setup-beam@v1
with:
otp-version: ${{matrix.otp}}
elixir-version: ${{matrix.elixir}}
# Step: Check out the code.
- name: Checkout code
uses: actions/checkout@v3
# Step: Define how to cache deps. Restores existing cache if present.
- name: Cache deps
id: cache-deps
uses: actions/cache@v3
env:
cache-name: cache-elixir-deps
with:
path: deps
key: ${{ runner.os }}-mix-${{ env.cache-name }}-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ runner.os }}-mix-${{ env.cache-name }}-
# Step: Define how to cache the `_build` directory. After the first run,
# this speeds up tests runs a lot. This includes not re-compiling our
# project's downloaded deps every run.
- name: Cache compiled build
id: cache-build
uses: actions/cache@v3
env:
cache-name: cache-compiled-build
with:
path: _build
key: ${{ runner.os }}-mix-${{ env.cache-name }}-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ runner.os }}-mix-${{ env.cache-name }}-
${{ runner.os }}-mix-
# Step: Conditionally bust the cache when job is re-run.
# Sometimes, we may have issues with incremental builds that are fixed by
# doing a full recompile. In order to not waste dev time on such trivial
# issues (while also reaping the time savings of incremental builds for
# *most* day-to-day development), force a full recompile only on builds
# that are retried.
- name: Clean to rule out incremental build as a source of flakiness
if: github.run_attempt != '1'
run: |
mix deps.clean --all
mix clean
shell: sh
# Step: Download project dependencies. If unchanged, uses
# the cached version.
- name: Install dependencies
run: mix deps.get
# Step: Compile the project treating any warnings as errors.
# Customize this step if a different behavior is desired.
- name: Compiles without warnings
run: mix compile --warnings-as-errors
# Step: Check that the checked in code has already been formatted.
# This step fails if something was found unformatted.
# Customize this step as desired.
- name: Check Formatting
run: mix format --check-formatted
# Step: Execute the tests.
- name: Run tests
run: mix test
```
### When the Workflow Runs...
When code is pushed to the `main` branch, this workflow is run. This happens
either from directly pushing to the `main` branch or after merging a PR into the
`main` branch.
This workflow is also configured to run checks on PR branches that target the
`main` branch. This is where it is most helpful. We can work on a fix or a new
feature in a branch and as we work and push our code up, it automatically runs
the full gamut of checks we want.
When all steps in the workflow succeed, the workflow "passes" and the automated
checks say it can be merged. When a step fails, the workflow halts at that point
and "fails", potentially blocking a merge.
### Customizing the Workflow Steps
This workflow is a starting point for a team. Every project is unique and every
team values different things. Is this missing something your team wants? Check
out some additional steps that can be added to your workflow.
* Add a step to run [Credo](https://github.com/rrrene/credo) checks.
* Add a step to run [Sobelow](https://github.com/nccgroup/sobelow) for security
focused static analysis.
* Add a step to run [dialyxir](https://github.com/jeremyjh/dialyxir). This runs
[Dialyzer](https://www.erlang.org/doc/man/dialyzer.html) static code analysis
on the project. Refer to the project for tips on caching the PLT.
* Customize the `mix test` command to include [code coverage](https://hexdocs.pm/mix/main/Mix.Tasks.Test.html#module-coverage) checks.
* Add Node setup and caching if `npm` assets are part of the project's test suite.
* Add a step to run [mix\_audit](https://github.com/mirego/mix_audit). This provides
a `mix deps.audit` task to scan a project's Mix dependencies for known Elixir
security vulnerabilities
* Add a step to run [`mix hex.audit`](https://hexdocs.pm/hex/Mix.Tasks.Hex.Audit.html).
This shows all Hex dependencies that have been marked as retired, which the
package maintainers no longer recommended using.
* Add a set to run [`mix deps.unlock --check-unused`](https://hexdocs.pm/mix/Mix.Tasks.Deps.Unlock.html).
This checks that the `mix.lock` file has no unused dependencies. This is useful
if you want to reject contributions with extra dependencies.
### Benefits of Caching
It's worth spending time tweaking your caches. Why? A lot of effort has been put
into speeding up Elixir build times. If we don't cache the build artifacts, then
we don't reap any of those benefits!
Of course, faster build times means you spend less money running your CI
workflow. But that's not the reason to do it! Better caches mean the checks are
performed faster and that means faster feedback. Faster feedback means that, as
a team, you save time and can move faster. No waiting 20 minutes for the checks
to complete so a PR can be merged. (Yes, I have felt that pain!)
This starting template builds in two caching steps. If `node_modules` factor
into your project's tests, then caching there makes a lot of sense too.
Just keep in mind that the reason we cache is to reduce drag on the speed of our
team.
If our caches ever cause a problem, and sometimes they can, it's good to know
how to clear them. In our project, go to Actions > (Sidebar) Management >
Caches. This is the list of caches saved for the project. We can use our naming
format to identify which cache file is for what.
### Retrying Failed Builds
We cache aggressively because we want the productivity benefits that incremental
compiles bring the team. However, on occasion, incremental builds can fail. The
quick fix is to perform a full recompile.
The following step from our CI pipeline does this for us.
```yaml theme={null}
# Step: Conditionally bust the cache when job is re-run.
# Sometimes, we may have issues with incremental builds that are fixed by
# doing a full recompile. In order to not waste dev time on such trivial
# issues (while also reaping the time savings of incremental builds for
# *most* day-to-day development), force a full recompile only on builds
# that are retried.
- name: Clean to rule out incremental build as a source of flakiness
if: github.run_attempt != '1'
run: |
mix deps.clean --all
mix clean
shell: sh
```
If this is not the first run attempt, meaning we are re-running the job, then it
busts our cache by cleaning out the `deps` and `_build` directories.
This compromise works well. Most of the time, we get fast checks and, where
there is a problem, we only need to retry the failed task to force a clean
build.
### Additional Resources
* [Documentation: Caching dependencies to speed up workflows](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows)
* [Managing Caches](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#managing-caches)
* [erlef/setup-beam](https://github.com/erlef/setup-beam) - Set up your BEAM-based GitHub Actions workflow (Erlang, Elixir, and more)
* See project for settings and ENV options
* [Elixir setup action - Archived](https://github.com/actions/setup-elixir#phoenix-example) - Includes helpful examples like setting up the Postgres service database.
* [Felt blog: Taking Hashrocket's "Ultimate Elixir CI" to the Next Level](https://felt.com/blog/hashrocket-ultimate-elixir-to-the-next-level)
* [felt/ultimate-elixir-ci](https://github.com/felt/ultimate-elixir-ci) - Good examples on CI setup with explanations.
* [Source for cleaning out incremental builds on retry](https://github.com/felt/ultimate-elixir-ci/blob/main/.github/actions/elixir-setup/action.yml#L97)
## Continuous Deployment (CD)
Continuous deployment is a software development practice that automatically
builds and releases the project into production when new changes pass automated
checks and are merged to the main branch. Using this practice, it is common to
release new versions of the applications multiple times a day, ideally, without
users even being aware of the process.
### Auto-Deploying our Elixir Applications
We can easily auto-deploy our Elixir applications on Fly.io. There is nothing
specific to Elixir about this process, so please refer to this guide for the details.
[Continuous Deployment with Fly.io and GitHub Actions](/launch/continuous-deployment-with-github-actions)
# Explore Your Production App with Livebook
Source: https://docs.fly.io/elixir/advanced-guides/interesting-things-with-livebook
Use Livebook connected to your Elixir app on Fly.io for business intelligence, querying production data and charting it with VegaLite and Kino.
[Livebook](https://livebook.dev/) is an exciting advancement in the Elixir community. It was created for the Elixir machine learning library [nx](https://github.com/elixir-nx/nx). You can think of it as Elixir's version of [Jupyter Notebooks](https://jupyter.org/).
Livebook ends up being very flexible and powerful. Because Elixir nodes can easily be clustered together, you can run Livebook on your local machine, connect it to your Elixir servers, and do some really interesting things.
Once you have [Livebook connected to your app on Fly](/elixir/advanced-guides/connect-livebook-to-your-app), you are ready to start playing!
## Getting Started with Livebook
Once you have Livebook running locally, **visit the "Learn" section** for examples of things you can do. Also, check out the [Livebook News](https://news.livebook.dev/) to find interesting developments and to spark new ideas.
Here are a few ideas of things you can do when Livebook is connected to your server.
* Query your users grouping by an interesting attribute and graph it.
* Chart user sign-ups over time.
* Analyze customer orders in a table.
* Create scripts in the form of notebooks to run common administrative commands.
## Business Intelligence Through Livebook
Let's walk through using Livebook to get Business Intelligence (BI) data from your Elixir app.
### Visualization Setup
With [Livebook v0.2](https://github.com/livebook-dev/livebook/blob/main/CHANGELOG.md), support was added for visualizations through the Elixir packages [VegaLite](https://github.com/livebook-dev/vega_lite) and [Kino](https://github.com/livebook-dev/kino). In this example, we're connecting a locally running Livebook to our app running on Fly. In order for the visualization tools to work, these libraries need to be installed on our server.
You may have seen Livebook examples showing how we can `Mix.install` the needed graphics libraries. That doesn't work when you are attaching to a remote mix project. It's actually a good thing that we can't just `Mix.install` like that on a remote project. You wouldn't want anyone with access to attach to your live system and install random packages!
That just means we need to install the dependencies first. Add the `vega_lite` and `kino` libraries to your project's `mix.exs` file and deploy your app.
```elixir theme={null}
defp deps do
[
{:phoenix, "~> 1.5.6"},
# ...
# Graphing/charting - for Livebook or analysis
{:vega_lite, "~> 0.1.0"},
{:kino, "~> 0.1.0"}
]
end
```
With the updated app deployed to Fly, Livebook can connect and render charts and visualizations!
### Starting Your Analysis Notebook
While you are figuring out **what** you want to examine in your app, you can explore it locally on your dev machine. Once you start, you may find you want to update your project code. Also, while playing locally, you reduce the chance of a badly structured query negatively impacting your live system.
I like [starting Livebook using the EScript method](https://github.com/livebook-dev/livebook#escript). Once installed, in a terminal you just go to the directory where your project's source code lives and start Livebook from there.
```bash theme={null}
livebook server
```
#### Create a Notebook
Open Livebook in a browser and create a "New notebook". For the first notebook, let's keep it really generic because I don't know what makes sense for your project.
Name the notebook something like "Exploring Data". Create a section for "Setup" and one for "Analysis".
Save the notebook with your project. I suggest creating a "notebook" folder to hold them so they can be checked in with your project and shared with the team.
#### Connect to Your Project
At this point, we haven't connected Livebook to our project so we can't access the project's code yet. To do this, start the project with something similar to the following command:
```
iex --sname my_app --cookie secret -S mix phx.server
```
This starts our local project with a short name of "my\_app" and a cookie of "secret".
Back in Livebook, click the "Runtime settings" button.
From here, click "Configure" then "Attached Node". Now, enter the name "my\_app" and the cookie "secret" (or whatever values were used when starting the server) and click "Connect". If everything went well, Livebook is connected to your locally running application!
We can verify it's connected by typing a module from the project into a code cell in the notebook and getting auto-complete help that's specific to our project.
#### Your First Analysis
This is where everything gets specific to the project. I'll walk through an example so we can see how to pull data, aggregate it, and visualize it. Then we'll turn you loose on your project!
My project contains survey-style questions. Some questions use a "personality" type field. My first goal is just to get the data and some counts to see how it's distributed.
Under **Setup**, I add an Elixir cell and the following code:
```elixir theme={null}
alias VegaLite, as: Vl
alias Core.Schemas.Question
alias Core.Repo
import Ecto.Query
```
The important one for visualizations is:
```elixir theme={null}
alias VegaLite, as: Vl
```
My first query groups the questions by the personality type value and returns the count.
```elixir theme={null}
query =
from(q in Question,
group_by: q.personality,
select: {q.personality, count(q.id)}
)
personality_counts =
query
|> Repo.all()
|> Enum.into(%{})
```
The results look like this:
```elixir theme={null}
%{any: 77, extrovert: 10, introvert: 13}
```
That's helpful already, but it's still just data. I'd like to see what that looks like.
I add another Elixir cell and write the following code:
```elixir theme={null}
# Convert the raw data into a Pie Chart friendly format
data =
Enum.map(personality_counts, fn {type, count} ->
%{"personality" => Atom.to_string(type), "value" => count}
end)
Vl.new()
|> Vl.data_from_values(data)
|> Vl.mark(:arc)
|> Vl.encode_field(:theta, "value", type: :quantitative)
|> Vl.encode_field(:color, "personality", type: :nominal)
|> Vl.config(view: [stroke: nil])
```
The `Vl` is the VegaLite alias. Here's the result:
Nice! I get a better sense for the personality type distribution in the question data.
What if I want to render the data as bar charts?
```elixir theme={null}
# Convert the raw data into a Bar Chart friendly format
data =
Enum.map(personality_counts, fn {type, count} ->
%{"Personality" => Atom.to_string(type), "Question Count" => count}
end)
# Sort the data by counts to order the results
data = Enum.sort_by(data, & &1["Question Count"], :desc)
Vl.new(width: 300, height: 300)
|> Vl.data_from_values(data)
|> Vl.mark(:bar)
|> Vl.encode_field(:x, "Personality",
type: :nominal,
axis: [label_angle: 0],
sort: [field: "Question Count", order: "descending"]
)
|> Vl.encode_field(:y, "Question Count", type: :quantitative)
```
Here's the result:
### Getting Started with VegaLite
If the VegaLite code looks scary, don't worry! I'm actually a total VegaLite noob! Livebook includes some built-in notebooks you should check out for code examples on how to structure data and configure VegaLite for the visualization you want.
Then I customized a little using the [VegaLite docs](https://vega.github.io/vega-lite/docs/) and [examples](https://vega.github.io/vega-lite/examples/).
### Summary
Livebook is an awesome tool for BI on your Elixir applications. Because it's going directly to your **code** and not just your **data**, you can use your code to query integrated services like Stripe and do awesome things!
# Elixir: SQLite3
Source: https://docs.fly.io/elixir/advanced-guides/sqlite3
Run a Phoenix app with SQLite3 on Fly.io by storing the database on a volume, running migrations on startup, and copying data over from Postgres or MySQL.
While Elixir applications on [Fly.io](https://fly.io) normally run on Postgres databases, you can
choose to run them on [SQLite3](https://www.sqlite.org/index.html). This guide will assume you have setup
and configured Phoenix Application using [ecto\_sqlite3](https://github.com/elixir-sqlite/ecto_sqlite3) running locally.
To make this work, you will need to place your databases on persistent [Volumes](https://www.sqlite.org/index.html)
as your deployment image will get overwritten the next time you deploy.
Volumes are limited to one host, this currently means that fly.io hosted Elixir applications that use
SQLite3 for their database can't be deployed to multiple regions.
But if you are okay using beta software, [LiteFS](/litefs) could work for multi-region sync, check it out! But this guide is going to assume you have one node and one volume.
This guide is heavily influenced by the [guide from](https://gist.github.com/mcrumm/98059439c673be7e0484589162a54a01) Phoenix Core Team Member, Michael Crumm.
Following are the steps required to make this work:
## Create volume
```bash theme={null}
fly volumes create name
```
Replace `name` with your desired volume name. Only alphanumeric characters and underscores are allowed in names.
Optionally, you may specify the size of the volume, in gigabytes, by adding a `--size int` argument.
The default volume size is 3 gigabytes.
## Mount and Prep for Deployment
Add the following to your `fly.toml`, once again replacing the name with what you selected, this
time in two places:
```toml theme={null}
[mounts]
source="name"
destination="/mnt/name"
```
Next remove the release step from the `[deploy]` section of your `fly.toml` and add a DATABASE\_PATH variable:
```diff theme={null}
[env]
+ DATABASE_PATH = "/mnt/name/name.db"
-[deploy]
- release_command = "/app/bin/migrate"
```
This step is required because a volume may not be ready once your application release runs, so to fix this we need to run migrations on
application start. This can be accomplished by adding the following line to your `lib/name/application.ex`:
```diff theme={null}
@impl true
def start(_type, _args) do
+ Name.Release.migrate()
children = [
```
## Deploy
These changes can be deployed:
```bash theme={null}
fly deploy
```
## Converting to SQLite3
Creating a new project with SQLite3 as the default is as simple as
```shell theme={null}
mix phx.new my_app --database sqlite3
```
But if you're working with existing code, here are the steps you'll need to make on a Phoenix 1.6+ application.
Append the following your `.gitignore` to ignore SQLite database files:
```.gitignore theme={null}
# Database files
*.db
*.db-*
```
Replace the `:postgrex` dep from your `mix.exs` with `ecto_sqlite`.
```elixir theme={null}
def deps do
[
{:ecto_sqlite3, "~> 0.9.1"}, # Latest version at time of publication.
# deps..
]
end
```
Update the repo configuration in `config/dev.exs`:
```elixir theme={null}
config :name, Name.Repo,
database: Path.expand("../name_dev.db", Path.dirname(__ENV__.file)),
pool_size: 5,
show_sensitive_data_on_connection_error: true
```
Update the repo configuration in `config/test.exs`:
```elixir theme={null}
config :name, Name.Repo,
database: Path.expand("../name_test.db", Path.dirname(__ENV__.file)),
pool_size: 5,
pool: Ecto.Adapters.SQL.Sandbox
```
And replace the `DATABASE_URL` in `config/runtime.exs` for production:
```elixir theme={null}
if config_env() == :prod do
database_path =
System.get_env("DATABASE_PATH") ||
raise """
environment variable DATABASE_PATH is missing.
For example: /data/name/name.db
"""
config :name, Name.Repo,
database: database_path,
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5")
```
Finally, update your Repo to use the SQLite adapter in `lib/name/repo.ex`:
```elixir theme={null}
defmodule Name.Repo do
use Ecto.Repo,
otp_app: :name,
adapter: Ecto.Adapters.SQLite3
end
```
## SQLite3 Tips
### Limitations
The `ecto_sqlite3` documentation includes a [good guide](https://hexdocs.pm/ecto_sqlite3/Ecto.Adapters.SQLite3.html#module-limitations-and-caveats) on the limits of using Ecto with SQLite3. We recommend at least skimming this before putting it into production.
### Transferring Data from Postgres/MySQL to SQLite3
> [!NOTE]
>
> This section is something to give you a starting point on how to get from X to SQLite. Make sure to back up your data and be vigilant, because this kind of thing is fraught at best.
There is really no very easy way to do this since the data types between the databases are different...
But here is one way that has worked for me in the past and might work for you! The Ruby [Sequel](https://sequel.jeremyevans.net/) project comes with a [command line tool for copying databases](https://sequel.jeremyevans.net/rdoc/files/doc/bin_sequel_rdoc.html#label-Copy+Databases). This method *should* help when transferring between ADO, Amalgalite, IBM\_DB, JDBC, MySQL, Mysql2, ODBC, Oracle, PostgreSQL, SQLAnywhere, and TinyTDS to SQLite3. That said it has limits! This is directly from the documentation:
> [!NOTE]
>
> This copies the table structure, table data, indexes, and foreign keys from the MySQL database to the PostgreSQL database.
>
> Note that the support for copying is fairly limited. It doesn’t handle database views, functions, triggers, schemas, partial indexes, functional indexes, and many other things. Also, the data type conversion may not be exactly what you want. It is best designed for quick conversions and testing. For serious production use, use the database’s tools to copy databases for the same database type, and for different database types, use the Sequel API.
So YMMV on how useful this is for you. If you have a relatively simple database it might work great!
Install Ruby if you don't already have it on your machine.
#### macOS
```bash theme={null}
brew install ruby
```
or
#### Ubuntu
```bash theme={null}
apt-get install ruby-full
```
The Ruby database adapter Sequel comes with a really slick command line tool for copying databases; let's install that
```sh theme={null}
gem install sequel
```
Depending on your database, you may need to install an extra adapter. In this example we use Postgres, so let's install that:
```bash theme={null}
gem install pg
```
Next all we need to do is run the Sequel Copy command, where the first database is the source and the second database is the destination:
```bash theme={null}
sequel -C postgres://localhost/database sqlite://name.db
```
And that's it! If you open up your SQLite database using the command line it should have your tables and data all moved over. If you get an error here about connecting you will need to figure out which ruby gem handles your adapter and install it like we did for pg.
```bash theme={null}
sqlite3 name.db
```
#### Note on types
SQLite doesn't have official support for Postgres Arrays or Hstores, and most special datatypes! These will be copied into strings in the resulting SQLite table, so for Arrays the data will look like `{item1, item2, item3}` if you want to still use this as an array you need to use [string manipulation](https://www.sqlitetutorial.net/sqlite-replace-function/) to convert them to json and then it should work just fine. This is just one example, but the positive is that it's all just strings in SQLite, so if you can make the string look like json you are set!
### Copying an existing database to a fly volume
If you've exported or copied your database to SQLite you will need to get your database file up to Fly.io. To do this we will use the [fly sftp](/flyctl/cmd/fly_sftp) command.
First open an sftp shell
```bash cmd theme={null}
fly sftp shell
```
```output output theme={null}
»
```
And use the `put` command to transfer your file to the volume path.
**NOTE**: Because our server is running, we first need to give the database a new name. Do **not** try to put this file in the same place as your current `DATABASE_PATH`.
```
» put ./name.db mnt/volume_name/name-prod.db
```
Where `volume_name` corresponds to the name you used when [creating your volume](#create-volume).
Check that it's there:
```sh theme={null}
» ls /mnt
```
```sh theme={null}
name.db
name-prod.db
.... other files
```
Then `ctrl-c` to exit.
Finally, update your env in `fly.toml`:
```diff theme={null}
[env]
- DATABASE_PATH = "/mnt/name.db"
+ DATABASE_PATH = "/mnt/name-prod.db"
```
And to persist that change:
```bash theme={null}
fly deploy
```
Next time it boots it should use your new database!
### Download a local copy of your production database
You can use the [fly sftp get](/flyctl/cmd/fly_sftp_get) command to download your database locally.
```elixir theme={null}
fly sftp get /mnt/name/name-prod.db prod.db
```
Replace `/mnt/name/name-prod.db` with the path to your database (DATABASE\_PATH).
# Deploy an Existing Elixir App
Source: https://docs.fly.io/elixir/getting-started/existing
Move an existing Elixir or Phoenix app to Fly.io: launch with fly launch, attach Postgres, deploy, and troubleshoot your first release.
If you have an existing Elixir app that you want to move over to Fly, this guide walks you through the initial deployment process and shows you techniques you can use to troubleshoot issues you may encounter in a new environment.
### Provision Elixir and Postgres Servers
To configure and launch your Elixir app, you can use `fly launch` and follow the wizard.
When you run `fly launch` from the newly-created project directory, the launcher provides some defaults for your new app, and gives you the option to tweak the settings.
Run:
```sh theme={null}
cd my-app-name
fly launch
```
You'll get a summary of the defaults chosen for your app:
```output theme={null}
Organization: MyOrgName (fly launch defaults to the personal org)
Name: my-app-name (derived from your directory name)
Region: Secaucus, NJ (US) (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM (most apps need about 1GB of RAM)
Postgres: (not requested)
Redis: (not requested)
? Do you want to tweak these settings before proceeding? Yes
Opening https://fly.io/cli/launch/bea626e2d179a083a3ba622a367e24ec ...
```
Type `y` at the prompt to open the Fly Launch page, and make the following changes to your app config:
* Change the default app name and region, if needed.
* For Databases, select Fly Postgres, give the Postgres database app a name (for example, your app name with `-db` appended) and choose a configuration.
Once you confirm your settings, you can return to the terminal, where the launcher will:
* Run the deployment setup task
* Build the image
* Set secrets required by (`SECRET_KEY_BASE`, for example)
* Deploy the application in your selected region
```output theme={null}
Waiting for launch data... Done
Created app 'my-app-name' in organization 'personal'
Admin URL: https://fly.io/apps/my-app-name
Hostname: my-app-name.fly.dev
Set secrets on my-app-name: SECRET_KEY_BASE
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app my-app-name-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:44b698752cf113110f2fa72443d7fe452b48228aafbb0d93045ef1e3282360a6
Waiting for machine to start...
Machine 2865550c7e96d8 is created
==> Monitoring health checks
Waiting for 2865550c7e96d8 to become healthy (started, 3/3)
Postgres cluster my-app-name-db created
Username: postgres
Password: EChe3BrhCjsPQEI
Hostname: my-app-name-db.internal
Flycast: fdaa:2:45b:0:1::1d
Proxy port: 5432
Postgres port: 5433
Connection string: postgres://postgres:EChe3BrhCjsPQEI@my-app-name-db.flycast:5432
Save your credentials in a secure place -- you won't be able to see them again!
Connect to postgres
Any app within the MyOrgName organization can connect to this Postgres using the above connection string
Now that you've set up Postgres, here's what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user
Postgres cluster my-app-name-db is now attached to my-app-name
The following secret was added to my-app-name:
DATABASE_URL=postgres://aa_hello_elixir2:Er6pLzUBuhKcbBl@my-app-name-db.flycast:5432/aa_hello_elixir2?sslmode=disable
Postgres cluster my-app-name-db is now attached to my-app-name
Generating rel/env.sh.eex for distributed Elixir support
Preparing system for Elixir builds
Installing application dependencies
Running Docker release generator
Wrote config file fly.toml
Validating /Users/anderson/test-elixir-gs/hello_elixir2/fly.toml
✓ Configuration is valid
==> Building image
Remote builder fly-builder-black-pine-7645 ready
Remote builder fly-builder-black-pine-7645 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
...
--> Pushing image done
image: registry.fly.io/my-app-name:deployment-01HPMGHTG8XSYH3ZCV82SF5CEZ
image size: 126 MB
Watch your deployment at https://fly.io/apps/my-app-name/monitoring
Provisioning ips for my-app-name
Dedicated ipv6: 2a09:8280:1::2a:bc0b:0
Shared ipv4: 66.241.124.79
Add a dedicated ipv4 with: fly ips allocate-v4
Running my-app-name release_command: /app/bin/migrate
-------
✔ release_command 784eee4c294298 completed successfully
-------
This deployment will:
* create 2 "app" machines
No machines in group app, launching a new machine
Creating a second machine to increase service availability
Finished launching new machines
-------
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for my-app-name.fly.dev
Visit your newly deployed app at https://my-app-name.fly.dev/
```
> [!NOTE]
>
> Make sure to note your Postgres credentials from the output.
That's it! Run `fly apps open` to see your deployed app in action.
Try a few other commands:
* `fly logs` - Tail your application logs
* `fly status` - App deployment details
* `fly status -a postgres-database-app-name` - Database deployment details
* `fly deploy` - Deploy the application after making changes
## Troubleshooting your initial deployment
Since this is an existing Elixir app, its highly likely it might not boot because you probably need to configure secrets or other service dependencies. Let's walk through how to troubleshoot these issues so you can get your app running.
### View log files
If your application didn't boot on the first deploy, run `fly logs` to see what's going on.
```bash theme={null}
fly logs
```
This shows the past few log file entries and tails your production log files.
### Open an IEx Session
It can be helpful to open a IEx Session to run commands and diagnose production issues.
To do this, we will [login with SSH](/flyctl/cmd/fly_ssh) to our
application VM. There is a one-time setup task for using SSH. Follow the instructions.
```
fly ssh issue --agent
```
With SSH configured, let's open a console.
```bash cmd theme={null}
fly ssh console --pty -C "/app/bin/hello_elixir remote"
```
```output output theme={null}
Connecting to hello_elixir.internal... complete
Erlang/OTP 23 [erts-11.2.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]
Interactive Elixir (1.11.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(hello_elixir@fdaa:0:1da8:a7b:ac4:b204:7e29:2)1>
```
You have a live IEx shell into your application!
## Deploy your application after changes
Deploying your application is done with the following command:
```bash theme={null}
fly deploy
```
This will take a few seconds as it uploads your application, builds a machine image,
deploys the images, and then monitors to ensure it starts successfully. Once complete
visit your app with the following command:
```bash theme={null}
fly apps open
```
If all went well, you'll see your Elixir application homepage.
# Deploy a Phoenix App
Source: https://docs.fly.io/elixir/getting-started/index
Deploy an Elixir Phoenix app to Fly.io with Postgres. Generate the app, configure it with fly launch, and ship it with fly deploy.
In this guide we’ll learn how to deploy an [Elixir Phoenix](https://www.phoenixframework.org/) application on Fly.io and connect it to a PostgreSQL database.
> [!NOTE]
>
> **NOTE:** This guide is for apps generated on Phoenix 1.6.3 or later, where deployment is streamlined significantly.
> If you're on an earlier version and can't upgrade, check out our **[legacy deployment guide](/elixir/getting-started/legacy)**.
We'll be using the standard web application generated by the Elixir [Phoenix Framework](https://www.phoenixframework.org/).
## *Preparation*
If you don't already have Elixir and Phoenix installed, get them set up before starting this guide. When you [install Elixir](https://elixir-lang.org/install.html), the [Mix](https://hexdocs.pm/mix/Mix.html) build tool ships with it.
You can [install](https://hexdocs.pm/phoenix/installation.html#phoenix) the Phoenix project generator `phx.new` from a [Hex](https://hex.pm/docs/usage) package as follows:
```bash theme={null}
mix archive.install hex phx_new
```
## *Generate the app and deploy with Postgres*
If you just want to see how Fly.io deployment works, this is the section to focus on. Here we'll bootstrap the app and deploy it with a Postgres database.
First, [install flyctl](/flyctl/install), the Fly.io CLI, and [sign up to Fly.io](/getting-started/sign-up-sign-in#no-account-sign-up-for-fly-io) if you haven't already.
Now let's generate a shiny, new Phoenix app:
```bash theme={null}
mix phx.new hello_elixir
```
The output ends with instructions for configuring the database and starting the app manually. The Fly.io launcher is going to take care of this, so we can ignore them.
When you run `fly launch` from the newly-created project directory, the launcher provides some defaults for your new app, and gives you the option to tweak the settings.
Run:
```sh theme={null}
cd hello_elixir
fly launch
```
You'll get a summary of the defaults chosen for your app:
```output theme={null}
Organization: MyOrgName (fly launch defaults to the personal org)
Name: hello_elixir (derived from your directory name)
Region: Secaucus, NJ (US) (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM (most apps need about 1GB of RAM)
Postgres: (not requested)
Redis: (not requested)
? Do you want to tweak these settings before proceeding? Yes
Opening https://fly.io/cli/launch/bea626e2d179a083a3ba622a367e24ec ...
```
Type `y` at the prompt to open the Fly Launch page, and make the following changes to your app config:
* Change the default app name and region, if needed.
* For Databases, select Fly Postgres, give the Postgres database app a name (for example, your app name with `-db` appended) and choose a configuration.
Once you confirm your settings, you can return to the terminal, where the launcher will:
* Run the deployment setup task
* Build the image
* Set secrets required by (`SECRET_KEY_BASE`, for example)
* Deploy the application in your selected region
```output theme={null}
Waiting for launch data... Done
Created app 'hello_elixir' in organization 'personal'
Admin URL: https://fly.io/apps/hello_elixir
Hostname: hello_elixir.fly.dev
Set secrets on hello_elixir: SECRET_KEY_BASE
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app hello_elixir-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:44b698752cf113110f2fa72443d7fe452b48228aafbb0d93045ef1e3282360a6
Waiting for machine to start...
Machine 2865550c7e96d8 is created
==> Monitoring health checks
Waiting for 2865550c7e96d8 to become healthy (started, 3/3)
Postgres cluster hello_elixir-db created
Username: postgres
Password: EChe3BrhCjsPQEI
Hostname: hello_elixir-db.internal
Flycast: fdaa:2:45b:0:1::1d
Proxy port: 5432
Postgres port: 5433
Connection string: postgres://postgres:EChe3BrhCjsPQEI@hello_elixir-db.flycast:5432
Save your credentials in a secure place -- you won't be able to see them again!
Connect to postgres
Any app within the MyOrgName organization can connect to this Postgres using the above connection string
Now that you've set up Postgres, here's what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user
Postgres cluster hello_elixir-db is now attached to hello_elixir
The following secret was added to hello_elixir:
DATABASE_URL=postgres://aa_hello_elixir2:Er6pLzUBuhKcbBl@hello_elixir-db.flycast:5432/aa_hello_elixir2?sslmode=disable
Postgres cluster hello_elixir-db is now attached to hello_elixir
Generating rel/env.sh.eex for distributed Elixir support
Preparing system for Elixir builds
Installing application dependencies
Running Docker release generator
Wrote config file fly.toml
Validating /Users/anderson/test-elixir-gs/hello_elixir2/fly.toml
✓ Configuration is valid
==> Building image
Remote builder fly-builder-black-pine-7645 ready
Remote builder fly-builder-black-pine-7645 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
...
--> Pushing image done
image: registry.fly.io/hello_elixir:deployment-01HPMGHTG8XSYH3ZCV82SF5CEZ
image size: 126 MB
Watch your deployment at https://fly.io/apps/hello_elixir/monitoring
Provisioning ips for hello_elixir
Dedicated ipv6: 2a09:8280:1::2a:bc0b:0
Shared ipv4: 66.241.124.79
Add a dedicated ipv4 with: fly ips allocate-v4
Running hello_elixir release_command: /app/bin/migrate
-------
✔ release_command 784eee4c294298 completed successfully
-------
This deployment will:
* create 2 "app" machines
No machines in group app, launching a new machine
Creating a second machine to increase service availability
Finished launching new machines
-------
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for hello_elixir.fly.dev
Visit your newly deployed app at https://hello_elixir.fly.dev/
```
> [!NOTE]
>
> Make sure to note your Postgres credentials from the output.
That's it! Run `fly apps open` to see your deployed app in action.
Try a few other commands:
* `fly logs` - Tail your application logs
* `fly status` - App deployment details
* `fly status -a postgres-database-app-name` - Database deployment details
* `fly deploy` - Deploy the application after making changes
To recap, the `fly launch` command detected that we are using Phoenix, set up a Fly app, created a Fly Postgres app and configured it for us, updated our Dockerfile, and created special modules and scripts to run ecto migrations for us!
### Storing secrets on Fly.io
You may also have some secrets you'd like set on your app.
Use [`fly secrets`](/apps/secrets) to configure those.
```bash theme={null}
fly secrets set MY_SECRET_KEY=my_secret_value
```
### Deploying again
When you want to deploy changes to your application, use `fly deploy`.
```bash theme={null}
fly deploy
```
The `fly deploy` command uses a Fly.io remote builder app to build the image.
You can always check on the status of a deploy:
```bash theme={null}
fly status
```
Check your app logs:
```bash theme={null}
fly logs
```
If everything looks good, open your app on Fly.io!
```bash theme={null}
fly apps open
```
### Important IPv6 settings
The `flyctl` command attempts to modify your project's Dockerfile and append the following lines:
```Dockerfile theme={null}
# Appended by flyctl
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"
```
If you customized your Dockerfile or launched without the Dockerfile, this setting may not have been set for you. These values are important and enable your Elixir app to work smoothly in Fly's private IPv6 network.
Check for this if you encounter network-related errors like the following:
```
Could not contact remote node my-app@fdaa:0:31d4:a5b:9d36:7c1e:f284:2, reason: :nodedown. Aborting...
```
## What's Next?
Next up, [IEx into Your Running App](/elixir/the-basics/iex-into-running-app)!
# Legacy pre v1.6.3 Phoenix App
Source: https://docs.fly.io/elixir/getting-started/legacy
Deploy a Phoenix app generated before v1.6.3 to Fly.io: configure releases and IPv6, add a Dockerfile, set secrets, and attach a Fly Postgres database.
Getting an application running on Fly.io is essentially working out how to package it as a deployable image and attach it to a database. Once packaged, it can be deployed to the Fly.io platform.
In this guide we’ll learn how to deploy an [Elixir Phoenix](https://www.phoenixframework.org/) application on Fly.io and connect it to a PostgreSQL database.
**This guide is for applications generated on Phoenix versions older than v1.6.3.**
If you're on 1.6.3 or higher, deployment setup is baked into the framework! Check out our [guide for
launching newer Elixir apps](/elixir/getting-started).
## *Running the Application Locally*
Let's walk through working with Phoenix locally and preparing it for deployment. We can start with a brand new Phoenix application.
```bash theme={null}
mix phx.new hello_elixir
```
**Tip:** Make sure you have a local [PostgreSQL server installed](https://wiki.postgresql.org/wiki/Detailed_installation_guides) and running.
Now, we setup the database and start the application.
```bash cmd theme={null}
mix ecto.setup
```
```output output theme={null}
The database for HelloElixir.Repo has been created
```
```bash cmd theme={null}
mix phx.server
```
```output output theme={null}
[info] Running HelloElixirWeb.Endpoint with cowboy 2.8.0 at 0.0.0.0:4000 (http)
[info] Access HelloElixirWeb.Endpoint at http://localhost:4000
```
Connect to localhost:4000 to confirm that you have a working Elixir application.
### *Generate Release Config Files*
We use the `mix release.init` command to create some sample files in the `./rel` directory.
```bash cmd theme={null}
mix release.init
```
```output output theme={null}
* creating rel/vm.args.eex
* creating rel/remote.vm.args.eex
* creating rel/env.sh.eex
* creating rel/env.bat.eex
```
## *Special Note on Fly Networking*
Internally Fly uses IPv6 networking. This enables some cool features, but legacy Elixir applications need to be configured to work smoothly with it.
These config changes tell Elixir, Phoenix, and the BEAM that we are using IPv6 addresses. The options look like `inet6` and `inet6_tcp`.
The next steps are how we configure that in our application.
We only need to configure `rel/env.sh.eex`. This file gets turned into a shell script that the release uses to set ENV values used when we run
any release commands. Here's the important parts.
```
#!/bin/sh
ip=$(grep fly-local-6pn /etc/hosts | cut -f 1)
export RELEASE_DISTRIBUTION=name
export ELIXIR_ERL_OPTIONS="-proto_dist inet6_tcp"
```
We configure the node to use a full node name when it runs. We get the Fly assigned IPv6 address and use that to name our node. Finally, we configure `inet6_tcp` for the BEAM as well.
Even if you don't care to cluster your nodes together, you still want to do this because it enables running an IEx shell in a running node.
## *Docker Setup*
### *Dockerfile*
You can find a good Dockerfile ready to build your application in the [hello\_elixir GitHub repo](https://github.com/fly-apps/hello_elixir/blob/main/Dockerfile). It is important to note that it uses a Debian-slim base image to build. This avoids DNS issues in Alpine images. The [base elixir image is maintained by the hexpm](https://hub.docker.com/r/hexpm/elixir/tags?page=1\&ordering=last_updated) org and is kept up to date. [Hex.pm](https://hex.pm/) is the official package repository for Elixir. There are other options instead of Debian there as well if you prefer.
The Dockerfile uses a two-stage approach. There are two `FROM` commands. The first stage pulls in the source and builds the release. The second stage takes the prepared release and sets it up in a minimal Docker image. The final deployed image contains only our release.
### *Docker Ignore File*
We add the file `.dockerignore` to the project with the following contents. Depending on how you `COPY` things into the Dockerfile, you may or may not need to configure this.
```
assets/node_modules/
deps/
```
This ensures we keep any natively compiled Elixir or Node packages from our development environments from causing a problem in the Linux container.
## *Launch the App on Fly*
To launch an app on fly, run `fly launch` in the directory with your source code. This creates and configures a fly app for you by inspecting your source code, then prompts you to deploy.
```bash cmd theme={null}
fly launch
```
```output output theme={null}
$ fly launch
Scanning source code
Detected Dockerfile app
? Select region: sea (Seattle, Washington (US))
Created app fly-elixir in organization personal
Wrote config file fly.toml
? Would you like to deploy now? No
```
Don't deploy it just yet. We're going to adjust the generated `fly.toml` file first.
The `fly launch` command scans your source code to determine how to build a deployment image as well as identify any other configuration your app needs, such as secrets and exposed ports.
After your source code is scanned and the results are printed, you'll be prompted for an organization. Organizations are a way of sharing applications and resources between Fly users. Every Fly account has a personal organization, called `personal`, which is only visible to your account. Let's select that for this guide.
Next, you'll be prompted to select a region to deploy in. The closest region to you is selected by default. You can use this or change to another region. You can find the [list of supported regions here](/reference/regions).
At this point, `flyctl` created an app for you and wrote your configuration to a `fly.toml` file. You'll then be prompted to build and deploy your app. Once complete, your app will be running on fly.
## *Inside `fly.toml`*
The `fly.toml` file now contains a default configuration for deploying your app. In the process of creating that file, `flyctl` has also generated a Fly-side application slot with a new name. In this case, it is `fly-elixir`. If we look at the `fly.toml` file we can see the name in there:
```toml theme={null}
app = "fly-elixir"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
```
The `flyctl` command will always refer to this file in the current directory if it exists, specifically for the `app` name/value at the start. That name is used to identify the application to the Fly platform. The rest of the file contains settings to be applied to the application when it deploys.
We'll have more details about these properties as we progress, but for now, it's enough to say that they mostly configure which ports the application will be visible on.
## *Customizing `fly.toml`*
Elixir applications need a little customization to the generated `fly.toml` file.
```toml theme={null}
app = "fly-elixir"
kill_signal = "SIGTERM"
kill_timeout = 5
processes = []
[deploy]
release_command = "/app/entry eval HelloElixir.Release.migrate"
[env]
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
http_checks = []
internal_port = 4000
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "30s" # allow some time for startup
interval = "15s"
restart_limit = 0
timeout = "2s"
```
There are two important changes here:
* We added the `[deploy]` setting. This tells Fly that on a new deploy, **run our database migrations**. The Dockerfile we're using creates a symlink to your app name called `entry`. This uses that symlink run the migrate command.
* The `kill_signal` is set to `SIGTERM`. An Elixir node does a clean shutdown when it receives a `SIGTERM` from the OS.
* The `internal_port` is set to 4000 for our Elixir application. The port value should match your application.
Some other values were tweaked as well.
## *Preparing to Deploy*
We're almost there! Before we can deploy our new app, we need to setup a few things in our Fly account first. Namely, we want to provide the needed secrets and we need a database!
### *Setting our Secrets on Fly*
Elixir has a mix task that can generate a new Phoenix key base secret. Let's use that.
```bash theme={null}
mix phx.gen.secret
```
It generates a long string of random text. Let's store that as a secret for our app. When we run this command in our project folder, `flyctl` uses the `fly.toml` file to know which app we are setting the value on.
```
fly secrets set SECRET_KEY_BASE=
```
### *Creating our Fly Postgres Database*
```bash cmd theme={null}
fly postgres create
```
```output output theme={null}
? App name: hello-elixir-db
Automatically selected personal organization: Mark Ericksen
? Select region: sea (Seattle, Washington (US))
? Select VM size: shared-cpu-1x - 256
? Volume size (GB): 10
Creating postgres cluster hello-elixir-db in organization personal
Postgres cluster hello-elixir-db created
Username:
Password:
Hostname: hello-elixir-db.internal
Proxy Port: 5432
PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!
Monitoring Deployment
2 desired, 2 placed, 2 healthy, 0 unhealthy [health checks: 6 total, 6 passing]
--> v0 deployed successfully
Connect to postgres
Any app within the personal organization can connect to postgres using the above credentials and the hostname "hello-elixir-db.internal."
For example: postgres://:@hello-elixir-db.internal:5432
See the postgres docs for more information on next steps, managing postgres, connecting from outside fly: https://fly.io/docs/reference/postgres/
```
We can take the defaults which select the lowest values for CPU, size, etc. This is perfect for getting started.
### *Attach our App to the Database*
We use `flyctl` to attach our app to the database which also sets our needed `DATABASE_URL` ENV value.
```bash cmd theme={null}
fly postgres attach hello-elixir-db
```
```output output theme={null}
Postgres cluster hello-elixir-db is now attached to fly-elixir
The following secret was added to fly-elixir:
DATABASE_URL=postgres://:@hello-elixir-db.internal:5432/fly_elixir?sslmode=disable
```
We can see the secrets that Fly is using for our app like this.
```bash cmd theme={null}
fly secrets list
```
```output output theme={null}
NAME DIGEST DATE
DATABASE_URL 830d8769ff33cba6c8b29d1cd6a6fbac 1m10s ago
SECRET_KEY_BASE 84c992ac7ef334c21f2aaecd41c43666 9m20s ago
```
Looks like we're ready to deploy!
## *Deploying to Fly*
To deploy your app, just run just run:
```bash theme={null}
fly deploy
```
First, `flyctl` builds our Dockerfile and pushes it to a Fly container registry.
This will lookup our `fly.toml` file, and get the app name `fly-elixir` from there. Then `flyctl` will start the process of deploying our application to the Fly platform. Flyctl returns you to the command line when it's done.
## *Viewing the Deployed App*
Now that the application has been deployed, let's find out more about its deployment. The command `fly status` will give you all the essential details.
```bash cmd theme={null}
fly status
```
```output output theme={null}
App
Name = fly-elixir
Owner = personal
Version = 3
Status = running
Hostname = fly-elixir.fly.dev
Deployment Status
ID = 9762642f-baa4-e4df-c683-13f2ce26a6bc
Version = v3
Status = successful
Description = Deployment completed successfully
Instances = 1 desired, 1 placed, 1 healthy, 0 unhealthy
Instances
ID VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
f617e72a 3 sea run running 1 total, 1 passing 0 1m34s ago
```
## *Connecting to the App*
The quickest way to browse your newly deployed application is with the `fly apps open` command.
```bash cmd theme={null}
fly apps open
```
```output output theme={null}
Opening https://fly-elixir.fly.dev/
```
Your browser will be sent to the displayed URL. Fly will auto-upgrade this URL to a HTTPS secured URL.
### *Special Note on Clustering*
To make clustering your Elixir applications easier on Fly, in the `env.sh.eex` file, the `RELEASE_NODE` is named using the `$FLY_APP_NAME` and
the IPv6 address. It will look something like this in practice.
```
fly-elixir@fdaa:0:1da8:a7b:ac2:216b:da3f:2
```
### *Runtime Config*
When we created the app, the file `config/runtime.exs` was created for us. Now we need to update it.
```elixir theme={null}
import Config
if config_env() == :prod do
database_url =
System.get_env("DATABASE_URL") ||
raise """
environment variable DATABASE_URL is missing.
For example: ecto://USER:PASS@HOST/DATABASE
"""
config :hello_elixir, HelloElixir.Repo,
# IMPORTANT: Or it won't find the DB server
socket_options: [:inet6],
url: database_url,
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
secret_key_base =
System.get_env("SECRET_KEY_BASE") ||
raise """
environment variable SECRET_KEY_BASE is missing.
You can generate one by calling: mix phx.gen.secret
"""
# IMPORTANT: Get the app_name we're using
app_name =
System.get_env("FLY_APP_NAME") ||
raise "FLY_APP_NAME not available"
config :hello_elixir, HelloElixirWeb.Endpoint,
# IMPORTANT: tell our app about the host name to use when generating URLs
url: [host: "#{app_name}.fly.dev", port: 80],
http: [
ip: {0, 0, 0, 0, 0, 0, 0, 0},
port: String.to_integer(System.get_env("PORT") || "4000")
],
secret_key_base: secret_key_base
# IMPORTANT: Enable the endpoint for releases
config :hello_elixir, HelloElixirWeb.Endpoint, server: true
end
```
This code expects to receive some ENV values at runtime. We're expecting `SECRET_KEY_BASE` with our Phoenix secret, our `FLY_APP_NAME` from Fly, and the `DATABASE_URL` for connecting to a Fly hosted Postgres database.
Also, you don't need to turn on TLS for connecting to the Postgres instance. Fly private networks operate over an encrypted WireGuard mesh, so traffic between application servers and PostgreSQL is already encrypted and there's no need to TLS.
## *Bonus Sections*
With your application up and running, there are some additional things you can do to go further. Using some `flyctl` commands, we can easily do some powerful things with our application.
These bonus tips cover:
* Getting an IEx shell into your running node. This helps you manage and work with your running system.
* Clustering multiple Elixir nodes together. Say "Hello!" to the power of Distributed Computing!
* Scaling your application out to more machines and even distant regions (with or without clustering).
### *What is the IP Address?*
If you want to know what IP addresses the app is using, try `fly ips list`:
```bash cmd theme={null}
fly ips list
```
```output output theme={null}
TYPE ADDRESS CREATED AT
v4 213.188.199.124 24m5s ago
v6 2a09:8280:1:ce56:c80f:5071:6e94:6688 24m5s ago
```
# Migrate an Elixir App from Heroku
Source: https://docs.fly.io/elixir/getting-started/migrate-from-heroku
Move a Phoenix app with Postgres, Oban and a custom domain from Heroku to Fly.io: fly launch, import secrets, copy the database, and a command cheat sheet.
This guide runs you through how to migrate a basic Elixir application off of Heroku and onto Fly.io. It assumes you're running the following services on Heroku:
* Postgres database
* Custom domain
* Background worker, like Oban
If your application is running with more services, additional work may be needed to migrate your application off Heroku.
## Migrating your app
The steps below run you through the process of migrating your Phoenix app from Heroku to Fly.
### Provision and deploy Phoenix app to Fly
From the root of the Elixir app you're running on Heroku, run `fly launch` and select the options to provision a new Postgres database.
When you run `fly launch` from the newly-created project directory, the launcher provides some defaults for your new app, and gives you the option to tweak the settings.
Run:
```sh theme={null}
cd my_app_name
fly launch
```
You'll get a summary of the defaults chosen for your app:
```output theme={null}
Organization: MyOrgName (fly launch defaults to the personal org)
Name: my_app_name (derived from your directory name)
Region: Secaucus, NJ (US) (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM (most apps need about 1GB of RAM)
Postgres: (not requested)
Redis: (not requested)
? Do you want to tweak these settings before proceeding? Yes
Opening https://fly.io/cli/launch/bea626e2d179a083a3ba622a367e24ec ...
```
Type `y` at the prompt to open the Fly Launch page, and make the following changes to your app config:
* Change the default app name and region, if needed.
* For Databases, select Fly Postgres, give the Postgres database app a name (for example, your app name with `-db` appended) and choose a configuration.
Once you confirm your settings, you can return to the terminal, where the launcher will:
* Run the deployment setup task
* Build the image
* Set secrets required by (`SECRET_KEY_BASE`, for example)
* Deploy the application in your selected region
```output theme={null}
Waiting for launch data... Done
Created app 'my_app_name' in organization 'personal'
Admin URL: https://fly.io/apps/my_app_name
Hostname: my_app_name.fly.dev
Set secrets on my_app_name: SECRET_KEY_BASE
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app my_app_name-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:44b698752cf113110f2fa72443d7fe452b48228aafbb0d93045ef1e3282360a6
Waiting for machine to start...
Machine 2865550c7e96d8 is created
==> Monitoring health checks
Waiting for 2865550c7e96d8 to become healthy (started, 3/3)
Postgres cluster my_app_name-db created
Username: postgres
Password: EChe3BrhCjsPQEI
Hostname: my_app_name-db.internal
Flycast: fdaa:2:45b:0:1::1d
Proxy port: 5432
Postgres port: 5433
Connection string: postgres://postgres:EChe3BrhCjsPQEI@my_app_name-db.flycast:5432
Save your credentials in a secure place -- you won't be able to see them again!
Connect to postgres
Any app within the MyOrgName organization can connect to this Postgres using the above connection string
Now that you've set up Postgres, here's what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user
Postgres cluster my_app_name-db is now attached to my_app_name
The following secret was added to my_app_name:
DATABASE_URL=postgres://aa_hello_elixir2:Er6pLzUBuhKcbBl@my_app_name-db.flycast:5432/aa_hello_elixir2?sslmode=disable
Postgres cluster my_app_name-db is now attached to my_app_name
Generating rel/env.sh.eex for distributed Elixir support
Preparing system for Elixir builds
Installing application dependencies
Running Docker release generator
Wrote config file fly.toml
Validating /Users/anderson/test-elixir-gs/hello_elixir2/fly.toml
✓ Configuration is valid
==> Building image
Remote builder fly-builder-black-pine-7645 ready
Remote builder fly-builder-black-pine-7645 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
...
--> Pushing image done
image: registry.fly.io/my_app_name:deployment-01HPMGHTG8XSYH3ZCV82SF5CEZ
image size: 126 MB
Watch your deployment at https://fly.io/apps/my_app_name/monitoring
Provisioning ips for my_app_name
Dedicated ipv6: 2a09:8280:1::2a:bc0b:0
Shared ipv4: 66.241.124.79
Add a dedicated ipv4 with: fly ips allocate-v4
Running my_app_name release_command: /app/bin/migrate
-------
✔ release_command 784eee4c294298 completed successfully
-------
This deployment will:
* create 2 "app" machines
No machines in group app, launching a new machine
Creating a second machine to increase service availability
Finished launching new machines
-------
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for my_app_name.fly.dev
Visit your newly deployed app at https://my_app_name.fly.dev/
```
> [!NOTE]
>
> Make sure to note your Postgres credentials from the output.
That's it! Run `fly apps open` to see your deployed app in action.
Try a few other commands:
* `fly logs` - Tail your application logs
* `fly status` - App deployment details
* `fly status -a postgres-database-app-name` - Database deployment details
* `fly deploy` - Deploy the application after making changes
There's still work to be done to move more Heroku stuff over, so don't worry if the app doesn't boot right away. Also check out:
* [Connect via IEx](/elixir/the-basics/iex-into-running-app)
### Transfer Heroku secrets
To see all of your Heroku env vars and secrets, run:
```bash theme={null}
heroku config -s | grep -v -e "DATABASE_URL" | fly secrets import
```
This command exports the Heroku secrets, excludes `DATABASE_URL` and imports them into Fly.
Verify your Heroku secrets are in Fly.
```bash theme={null}
fly secrets list
NAME DIGEST CREATED AT
DATABASE_URL 24e455edbfcf1247a642cdae30e14872 14m29s ago
LANG 95a7bb7a8d0ee402edde95bb78ef95c7 1m24s ago
MIX_ENV fd89784e59c72499525556f80289b2c7 1m26s ago
SECRET_KEY_BASE 5afb43c2ddbba6c02ffa7e2834689692 1m22s ago
```
### Transfer the Database
> [!NOTE]
>
> Any new data created by your Heroku app during this database migration won't be moved over to Fly.io. Consider taking your Heroku application offline or place in read-only mode if you want to be confident that this migration will move over 100% of your Heroku data to Fly.io.
Set the `HEROKU_DATABASE_URL` variable in your Fly.io environment.
```bash theme={null}
fly secrets set HEROKU_DATABASE_URL=$(heroku config:get DATABASE_URL)
```
Alright, lets start the transfer remotely on the Fly.io instance.
```bash theme={null}
fly ssh console
```
Then from the remote Fly SSH console transfer the database.
```bash theme={null}
pg_dump -Fc --no-acl --no-owner -d $HEROKU_DATABASE_URL | pg_restore --verbose --clean --no-acl --no-owner -d $DATABASE_URL
```
You may need to upgrade your Heroku database to match the version of the source Fly.io database. Refer to Heroku's [Upgrading the Version of a Heroku Postgres Database](https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases) for instructions on how to upgrade, then try the command above again.
After the database transfers unset the `HEROKU_DATABASE_URL` variable.
```bash theme={null}
fly secrets unset HEROKU_DATABASE_URL
```
Then launch your Heroku app to see if its running.
```
fly apps open
```
If you have a Redis server, there's a good chance you need to set that up.
### Custom Domain & SSL Certificates
After you finish deploying your application to Fly.io and have tested it extensively, [read through the Custom Domain docs](/networking/custom-domain) and point your domain at it.
In addition to supporting [`CNAME` DNS records](/networking/custom-domain#cname-records), Fly.io also supports [`A` and `AAAA` records](/networking/custom-domain#a-and-aaaa-records-recommended) for those who want to point `example.com` (without the `www.example.com`) directly at their App on Fly.io.
## Cheat Sheet
Old habits die hard, especially good habits like deploying frequently to production. Below is a quick overview of the differences you'll notice initially between Fly.io and Heroku.
### Commands
flyctl commands are a bit different than Heroku, but you'll get use to them after a few days.
| Task | Heroku | Fly.io |
| - | - | - |
| Deployments | `git push heroku` | `fly deploy` |
| IEx console | `heroku console` | `fly ssh console --pty -C "/app/bin/app remote"` |
| Database migration | `heroku rake db:migrate` | `fly ssh console -C "/app/bin/migrate"` |
| Postgres console | `heroku psql` | `fly postgres connect -a ` |
| Tail log files | `heroku logs` | `fly logs` |
| View configuration | `heroku config` | `fly ssh console -C "printenv"` |
| View releases | `heroku releases` | `fly releases` |
| Help | `heroku help` | `fly help` |
Check out the [flyctl docs](/flyctl) for a more extensive inventory of flyctl commands.
### Deployments
By default Heroku deployments are kicked off via the `git push heroku` command. Fly.io works a bit differently by kicking of deployments via `fly deploy`—git isn't needed to deploy to Fly.io. The advantage to this approach is your git history will be clean and not full of commits like `git push heroku -am "make app work"` or `git push heroku -m "ok it will really work this time"`.
To achieve the desired `git push` behavior, we recommend setting up `fly deploy` as the final command in your continuous integration pipeline, as outlined for GitHub in the [Continuous Deployment with Fly.io and GitHub Actions](/launch/continuous-deployment-with-github-actions) docs.
#### Deploy via git
Heroku's default deployment technique is via `git push heroku`. Fly.io doesn't require a git commit, just run `fly deploy` and the files on your local workstation will be deployed.
Fly.io can be configured to deploy on git commits with the following techniques with a [GitHub Action](/launch/continuous-deployment-with-github-actions).
### Databases
Fly.io and Heroku have different Postgres database offerings. The most important distinction to understand about using Fly.io is that it automates provisioning, maintenance, and snapshot tasks for your Postgres database, but it does not manage it. If you run out of disk space, RAM, or other resources on your Fly Postgres instances, you'll have to scale those virtual machines from the [Fly CLI](/unmanaged-postgres).
Contrast that with Heroku, which fully manages your database and includes an extensive suite of tools to provision, backup, snapshot, fork, patch, upgrade, and scale up/down your database resources.
The good news for people who want a highly managed Postgres database is they can continue hosting it at Heroku and point their Fly.io instances to it!
#### Heroku's managed database
One command is all it takes to point Fly Apps at your Heroku managed database.
```bash theme={null}
fly secrets set DATABASE_URL=$(heroku config:get DATABASE_URL)
```
This is a great way to get comfortable with Fly.io if you prefer a managed database provider. In the future if you decide you want to migrate your data to Fly, you can do so [pretty easily with a few commands](#transfer-the-database).
#### Fly Postgres
The most important thing you'll want to be comfortable with using Fly.io's database offering is [backing up and restoring your database](/unmanaged-postgres/managing/backup-and-restore).
As your application grows, you'll probably first [scale disk and RAM resources](/unmanaged-postgres/managing/scaling), then [scale out with multiple replicas](/unmanaged-postgres/managing/horizontal-scaling). Common maintenance tasks will include [upgrading Postgres](/unmanaged-postgres/managing/upgrades) as new versions are released with new features and security updates.
[See here](/unmanaged-postgres) for a more comprehensive guide for what's required when running your Postgres databases on Fly.io.
### Pricing
Heroku and Fly.io have very different pricing structures. You'll want to read through the details on [Fly.io's pricing page](/about/pricing) before launching to production. The sections below serve as a rough comparison between Heroku's and Fly.io's plans as of August 2022.
> [!NOTE]
>
> Please do your own comparison of plans before switching from Heroku to Fly.io. The examples below are illustrative estimates between two very different offerings, which focuses on the costs of app & database servers. It does not represent the final costs of each plan. Also, the prices below may not be immediately updated if Fly.io or Heroku change prices.
#### Free Plans
Heroku will not offer free plans as of November 28, 2022.
Fly.io offers free usage for up to 3 full time VMs with 256MB of RAM, which is enough to run a Elixir app and Postgres database to get a feel for how Fly.io works.
#### Plans for Small Elixir Apps
Heroku's Hobby tier is limited to 10,000 rows of data, which gets exceeded pretty quickly requiring the purchase of additional rows of data.
| Heroku Resource | Specs | Price |
| - | - | - |
| App Dyno | 512MB RAM | \$7/mo |
| Database | 10,000,000 rows | \$9/mo |
| **Estimated cost** | | \$16/mo |
Fly.io pricing is [metered for the resources](/about/pricing) you use. Database is billed by the amount of RAM and disk space used, not by rows. The closest equivalent to the Heroku Hobby tier on Fly.io looks like this:
| Fly.io Resource | Specs | Price |
| - | - | - |
| App Server | 1GB RAM | \~\$6.70/mo |
| Database Server | 256MB RAM / 10Gb disk | \~\$3.44/mo |
| **Estimated cost** | | \~\$9.14/mo |
#### Plans for Medium to Elixir Apps
There's too many variables to compare Fly.io's and Heroku's pricing for larger Elixir applications depending on your needs, so you'll definitely want to do your homework before migrating everything to Fly.io. This comparison focuses narrowly on the costs of app & database resources, and excludes other factors such as bandwidth costs, bundled support, etc.
| Heroku Resource | Specs | Price | Quantity | Total |
| - | - | - | - | - |
| App Dyno | 2.5GB RAM | \$250/mo | 8 | \$2,000/mo |
| Database | 61GB RAM / 1TB disk | \$2,500/mo | 1 | \$2,500/mo |
| **Estimated cost** | | | | \$4,500/mo |
Here's roughly the equivalent resources on Fly:
| Fly.io Resource | Specs | Price | Quantity | Total |
| - | - | - | - | - |
| App Server | 4GB RAM / 2X CPU | \~\$66.00/mo | 8 | \~\$528/mo |
| Database Server | 64GB RAM / 500GB disk | \~\$633/mo | 2 | \~\$1,266/mo |
| **Estimated cost** | | | | \~\$1,762/mo |
Again, the comparison isn't realistic because it focuses only on application and database servers, but it does give you an idea of how the different cost structures scale on each platform. For example, Heroku's database offering at this level is redundant, whereas Fly.io offers 2 database instances to achieve similar levels of redundancy.
# Elixir on Fly.io
Source: https://docs.fly.io/elixir/index
Deploy Phoenix and LiveView apps to Fly.io and cluster Elixir nodes across regions. Jump to getting started, the basics, or advanced BEAM guides.
Fly.io makes it easier to run an Elixir cluster on multiple servers around the world and deploy to those regions with a single command. This comprehensive guide walks you through how to launch your first Phoenix app on Fly.io and includes a few additional guides to walk you through more advanced topics.
When you deploy your clustered application around the world, you give your users a fast, snappy, low-latency experience. This becomes particularly important if you're deploying a Phoenix LiveView app.
## [Getting Started](./getting-started)
Run through the [Elixir Getting Started](./getting-started/) guide to get a feel for what it's like to deploy to Fly.io. This is the best jumping-off point for getting your app up and running.
## [The Basics](./the-basics)
Go from running an app to clustering your app globally and administering with Fly.io.
## [Advanced Guides](./advanced-guides)
A collection of guides for doing some extra nifty things with your app on Fly.io.
# Elixir: The Basics
Source: https://docs.fly.io/elixir/the-basics
Elixir basics on Fly.io: open an IEx shell into a running production app, cluster Erlang nodes into one distributed network, and troubleshoot deploys.
Most Elixir applications require additional setup beyond provisioning a Postgres database. These guides will help you get through the basics of setting up your Elixir application with pieces of infrastructure commonly found in medium-to-high complexity Elixir applications.
## IEx into Your Running App
Get an IEx shell into your running production app on Fly!
## Clustering Your Application
Cluster your erlang nodes into a single distributed network!
## Troubleshooting
Common troubleshooting issues and resources for getting unstuck.
# Clustering Your Application
Source: https://docs.fly.io/elixir/the-basics/clustering
Cluster Elixir nodes on Fly.io with dns_cluster and node names built from the app name and private IPv6 address, then scale across regions with a static cookie.
Elixir and the BEAM have the incredible ability to be clustered together and
processes can pass messages seamlessly to each other between nodes. Fly.io makes
clustering easy! This extra (and totally optional) portion of the guide walks
you through clustering your Elixir application.
There are 3 parts to getting clustering quickly setup on Fly.io.
* Naming the Node(s)
* Installing and using `dns_cluster`
* Scaling our application to multiple VMs
## Naming the Node(s)
To make clustering easier, we want our Elixir nodes to be named using our Fly application name and the IPv6 address assigned to the node. Then later, our cluster can form up using DNS settings based on this naming scheme.
If you've run `fly launch` already then you should have a `rel` folder with a `rel/env.sh.eex` file already.
If not, run this command from your Elixir application:
```bash theme={null}
mix release.init
```
Then check the generated `rel/env.sh.eex` file and add ensure the lines looks similar:
```shell theme={null}
# configure node for distributed erlang with IPV6 support
export ERL_AFLAGS="-proto_dist inet6_tcp"
export ECTO_IPV6="true"
export DNS_CLUSTER_QUERY="${FLY_APP_NAME}.internal"
export RELEASE_DISTRIBUTION="name"
export RELEASE_NODE="${FLY_APP_NAME}-${FLY_IMAGE_REF##*-}@${FLY_PRIVATE_IP}"
```
This names our Elixir node's name (also known as RELEASE\_NODE) using the Fly application name, the Docker image reference value, and the internal IPv6 address. Make sure to deploy after making this change!
```bash theme={null}
fly deploy
```
## Adding `dns_cluster`
The Phoenix library [dns\_cluster](https://github.com/phoenixframework/dns_cluster) helps here.
The `dns_cluster` library lets you easily setup Erlang Clustering using DNS, and Fly.io has built in DNS support!
After installing `dns_cluster`, add it to the application like this:
```elixir theme={null}
defmodule HelloElixir.Application do
use Application
def start(_type, _args) do
children = [
# ...
# setup for clustering
{Phoenix.PubSub, ...},
{DNSCluster, query: Application.get_env(:hello_elixir, :dns_cluster_query) || :ignore},
HelloElixirWeb.Endpoint
]
# ...
end
# ...
end
```
Our next step is to add the `dns_cluster_query` configuration to the file `config/runtime.exs`.
```elixir theme={null}
config :hello_elixir, dns_cluster_query: System.get_env("DNS_CLUSTER_QUERY")
```
And finally add a key to our `fly.toml` file:
```toml theme={null}
[env]
DNS_CLUSTER_QUERY = "hello-elixir.internal"
```
**REMEMBER:** Deploy your updated app so the clustering code is available, with `fly deploy`.
This configures `dns_cluster` to look for other deployed apps using the same `$FLY_APP_NAME` on the `.internal` private network.
This assumes that your `rel/env.sh.eex` file is configured to name your Elixir node using the `$FLY_APP_NAME`. We did this earlier in the "Naming Your Elixir Node" section.
Before this app can be clustered, we need more than one VM. We'll do that next!
## Running multiple VMs
There are two ways to run multiple VMs.
1. Scale our application to have multiple Fly Machines in one region.
2. Add a Machine to another region (multiple regions).
Both approaches are valid and our Elixir application doesn't change at all for the approach you choose!
Let's first start with a baseline of our single deployment.
```bash cmd theme={null}
fly status
```
```output output theme={null}
...
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app 6e82dd00f75687 20 sea started 1 total, 1 passing 2023-03-16T22:01:45Z
```
### Scaling in a single region
Let's scale up to 2 VMs in our current region.
```bash theme={null}
fly scale count 2
```
Checking on the status we can see what happened.
```bash cmd theme={null}
fly status
```
```output output theme={null}
...
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app 5683d474b4658e 20 sea started 1 total, 1 passing 2023-06-16T01:49:36Z
app 6e82dd00f75687 20 sea started 1 total, 1 passing 2023-03-16T22:01:45Z
```
We now have two VMs in the same region! That was easy.
Let's make sure they are clustered together. We can check the logs:
```bash cmd theme={null}
fly logs
```
```output output theme={null}
...
app[5683d474b4658e] sea [info] 21:50:21.924 [info] [libcluster:fly6pn] connected to :"fly-elixir@fdaa:0:1da8:a7b:ac2:f901:4bf7:2"
...
```
But that's not as rewarding as seeing it from inside a node. From an IEx shell, we can ask the node we're connected to, what other nodes it can see.
```bash theme={null}
fly ssh console --pty -C "/app/bin/hello_elixir remote"
```
```elixir theme={null}
iex(fly-elixir@fdaa:0:1da8:a7b:ac2:f901:4bf7:2)1> Node.list
[:"fly-elixir@fdaa:0:1da8:a7b:ac4:eb41:19d3:2"]
```
I included the IEx prompt because it shows the IP address of the node I'm connected to. Then getting the `Node.list` returns the other node. Our two VMs are connected and clustered!
### Scaling to multiple regions
Fly.io makes it super easy to run VMs of your applications physically closer to your users. Through the magic of DNS, users are directed to the nearest [region](/reference/regions) where your application is located.
Starting back from our baseline of a single VM running in `sea` which is Seattle, Washington (US), I'll add the region `ewr` which is NJ (US). I can do this by cloning the existing Fly Machine into my desired region:
```bash theme={null}
fly machine clone 6e82dd00f75687 --region ewr
```
Now our status shows we have two Machines spread across 2 regions! This puts a VM on both coasts of the US.
```bash cmd theme={null}
fly status
```
```output output theme={null}
...
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app 0e2869ea63d486 20 ewr started 1 total, 1 passing 2023-06-16T01:56:19Z
app 6e82dd00f75687 20 sea started 1 total, 1 passing 2023-03-16T22:01:45Z
```
Let's ensure they are clustered together.
```bash theme={null}
fly ssh console --pty -C "/app/bin/hello_elixir remote"
```
```elixir theme={null}
iex(fly-elixir@fdaa:0:1da8:a7b:ac2:cdf6:c422:2)1> Node.list
[:"fly-elixir@fdaa:0:1da8:a7b:ab2:a8e:6666:2"]
```
We have two VMs of our application deployed to the West and East coasts of the North American continent and they are clustered together! Our users will automatically be directed to the server nearest them. That is so cool!
## The cookie situation
Before two Elixir nodes **can** cluster together, they must share a secret cookie. The cookie itself isn't meant to be a super secret encryption key or anything like that, it's designed to let us create multiple sets of small clusters on the same network that don't all just connect together. Different cookies means different clusters. For instance, only the nodes that all use the cookie `abc` will connect together.
For us, this means that in order for `my_remote` node to connect to the cluster on Fly, we need to share the same cookie value used in production.
### The cookie problem
When we build a `mix release`, it generates a long random string for the cookie value. When we **re-run** the `mix release` command, it keeps the same cookie value. That is, when we don't run it in Docker. The Dockerfile we're using is building a fresh release every time we run it. That's kind of the point of a Docker container. So **our cookie value is being randomly generated every time we deploy**. This means after every deploy, we would have to figure out what the new cookie value is so our local node can use it.
### The cookie solution
The easiest solution here is to **specify** the value to use for our cookie. One that we will know outside of the build and that won't keep changing on us.
## Making the cookie changes
If we read the [Mix.Tasks.Release docs](https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-options), in the `:cookie` section we learn that if we provide an ENV named `RELEASE_COOKIE`, it will be used. If that ENV is not found, it falls back to the randomly generated one.
To generate the cookie string we will use this Elixir command:
```elixir theme={null}
Base.url_encode64(:crypto.strong_rand_bytes(40))
```
To provide the ENV named `RELEASE_COOKIE` inside the running app, after generating the cookie, we can either:
* [Put it as a secret](/apps/secrets#set-secrets) inside project settings under the `RELEASE_COOKIE` name, or
* Store it in our `fly.toml` file like this:
```toml theme={null}
[env]
RELEASE_COOKIE = "my-app-cookie"
```
After setting up the ENV and deploying the application, we can verify that the cookie is being used by getting an [IEx shell into our running server](/elixir/the-basics/iex-into-running-app) and issuing the following command:
```elixir theme={null}
Node.get_cookie()
```
This shows the cookie being used at runtime.
We can also check list of connected nodes by running the following command:
```elixir theme={null}
Node.list()
```
An empty list means the node has no connections. If you are sure that there is more then one node running, you could proceed to [Troubleshooting](/elixir/the-basics/troubleshooting) documentation.
With a known and unchanging cookie deployed in our application, we are ready for the next step!
### Important IPv6 settings
The `flyctl` command attempts to modify your project's Dockerfile and append the following lines:
```Dockerfile theme={null}
# Appended by flyctl
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"
```
If you customized your Dockerfile or launched without the Dockerfile, this setting may not have been set for you. These values are important and enable your Elixir app to work smoothly in Fly's private IPv6 network.
Check for this If you encounter network related errors like this:
```
Could not contact remote node my-app@fdaa:0:31d4:a5b:9d36:7c1e:f284:2, reason: :nodedown. Aborting...
```
If you have non-empty list with all of your running nodes - congratulations, you have successfully set up the clustering!
# IEx into Your Running App
Source: https://docs.fly.io/elixir/the-basics/iex-into-running-app
Open an IEx shell into a running Elixir app on Fly.io with fly ssh console and the --pty flag, and wrap the command in a short shell script for quick access.
Elixir supports getting a IEx shell into a running production node. How cool is that?
## SSH Into our App
To do this, we will [login with SSH](/flyctl/cmd/fly_ssh) to our
application VM. There is a one-time setup task for using SSH. Follow the instructions.
```
fly ssh issue --agent
```
With SSH configured, let's open a console.
```bash cmd theme={null}
fly ssh console --pty -C "/app/bin/hello_elixir remote"
```
```output output theme={null}
Connecting to hello_elixir.internal... complete
Erlang/OTP 23 [erts-11.2.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]
Interactive Elixir (1.11.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(hello_elixir@fdaa:0:1da8:a7b:ac4:b204:7e29:2)1>
```
You have a live IEx shell into your application!
> [!NOTE]
>
> **Tip:** One of several ways to [exit the IEx shell](https://hexdocs.pm/iex/IEx.html#module-exiting-the-shell) is to hit `Ctrl+C, Ctrl+C`; to log out of the VM console, use `Ctrl+D` or `exit`.
> [!NOTE]
>
> The `--pty` flag tells the SSH server to run the command in a pseudo-terminal. You will generally need this only when running interactive commands, like IEx.
## Creating a Shell Script for Convenience
In UNIX-based operating system like Linux and Mac OS, we can create a shell script to make connecting to the server with an IEx terminal easy. The same can be done with a Windows PowerShell script or batch file.
However we name the file will be our "command" for opening the IEx shell into the server. Here we'll use `fiex`, short for "Fly IEx".
```shell theme={null}
#!/bin/bash
set -e
fly ssh console --pty --select -C "/app/bin/hello_elixir remote"
```
The `--pty` flag supports our interactive terminal and is needed to support the `--remsh` option used when we call `remote`.
The `--select` flag will prompt for which instance to connect to. This is handy when deployed to multiple regions and we want to be explicit about where we connect.
Remember to make the script executable: `chmod +x fiex`.
Using a terminal from the app directory, type `./fiex` to quickly jump into an IEx shell on the server.
## What's Next?
With another quick update we can prepare our application for clustering by naming our nodes differently.
Next up, [Clustering Your Application](/elixir/the-basics/clustering)!
# Troubleshooting
Source: https://docs.fly.io/elixir/the-basics/troubleshooting
Troubleshoot Elixir deploys on Fly.io: test releases and Docker builds locally, check IPv6 config, tune connection limits, and debug clustering.
Some problems are harder to diagnose because they deal with [Elixir releases](https://hexdocs.pm/mix/master/Mix.Tasks.Release.html) or Docker build problems. Typically, you don't run the application that way locally, so you only encounter those problems when it's time to deploy.
Here are a few tips to help diagnose and identify problems.
* Run `mix release` locally on your project.
* Build the Dockerfile locally to verify it builds correctly. `docker build .`
* Check the `:prod` config in `config/runtime.exs`, which is not used locally. Carefully review it.
* Run `fly logs` to check server logs.
For diagnosing database app issues, refer to the [Postgres Monitoring](/unmanaged-postgres/managing/monitoring) information.
Here's a quick hit list of commands to help:
* Run `fly logs -a ` to check database app's server logs.
* Run `fly checks list -a ` to check the database app's health.
* Run `fly status -a --all` to see if any VMs failed.
* Run `fly vm status -a ` to debug a specific VM.
## *Diagnosis Tip*
Most difficulties center around application config. Applications generated with an older version of Phoenix are configured differently than a newly generated app. If you have problems like connecting to your database, usually an IPv6 configuration update is needed.
The internal networks at Fly.io use a IPv6 addresses. Elixir/OTP needs some config to work smoothly.
One way to identify an issue is to generate a new Elixir application using a current version of Phoenix. Deploy that to Fly.io with a database. With that, you have a local working example to compare against. Don't worry, you can easily [`destroy`](/flyctl/cmd/fly_apps_destroy) the test app when you're ready to.
Suggested files to pay attention to when looking for config differences.
* `config/config.exs`
* `config/prod.exs`
* `config/runtime.exs`
* `Dockerfile`
* `mix.exs`
## *Not Enough Connections*
A common failure mode is the application exhausting the number of free connections, your default `fly.toml` has the following settings:
```toml theme={null}
[services.concurrency]
hard_limit = 50
soft_limit = 25
type = "connections"
```
Setting the `hard_limit` and `soft_limit` closer to your needs will free up the number of live connections per node. A safe starting point could be 1000 for the hard\_limit and 975 for the soft\_limit. The "right" amount depends on how much data is actively stored in the LiveView processes. That value will vary for each application.
## *Clustering*
Here are some troubleshooting tips when working with [Clustering](/elixir/the-basics/clustering).
When using [IEx to remote into a running application](/elixir/the-basics/iex-into-running-app) to diagnose connection issues, note if you see this warning when connecting:
```
warning: the --remsh option will be ignored because IEx is running on limited shell
```
When that warning is present, the Elixir node we are connecting to is not the remote running node. A new node was launched when the remote shell request was ignored. Refer to the [docs here about the `--pty`](/elixir/the-basics/iex-into-running-app) option.
Another indication of this situation is when typing `Node.self()`, the name of the node is returned with a prefix similar to "rem-ea46-".
Example:
```
:"rem-ea46-hello_elixir@fdaa:0:1da8:a7b:115:5641:7e85:2"
```
A working remote shell will return a node name more like this:
```
:"hello_elixir@fdaa:0:1da8:a7b:115:5641:7e85:2"
```
### Testing Node Connectivity
Using `libcluster` is an easy way to [auto-cluster](#clustering) an Elixir application. However, going through the process manually can help diagnose issues.
We can open two terminals locally on our machine. In terminal A, we get an [IEx terminal](/elixir/the-basics/iex-into-running-app) to one node. Then in terminal B, we get an IEx terminal to a different node.
In each terminal, we can ask the node for it's name:
```
Node.self
```
Then, taking the response node in terminal A, we can explicitly try to connect through terminal B (on the other node).
That command might look like this:
```
Node.connect(:"result-from-terminal-A@ipvs-address")
```
If the result is `true` then it either connected to the other node or we entered into the wrong terminal and it says it's connected to itself.
If we received a `true` response, we can check the list of connected nodes using this command:
```
Node.list()
```
An empty list means is has no connections.
During the process, make note of any logged messages that might help explain why the two nodes can't connect.
# fly agent
Source: https://docs.fly.io/flyctl/cmd/fly_agent
The Fly agent is a background process that manages wireguard connections started by flyctl.
Commands such as 'fly ssh' and 'fly proxy' use this agent.
Generally, the agent starts and stops automatically. These commands are useful for debugging.
## Usage
```
fly agent [command] [flags]
```
## Available Commands
* [ping](/flyctl/cmd/fly_agent_ping) - Ping the Fly agent
* [restart](/flyctl/cmd/fly_agent_restart) - Restart the Fly agent
* [run](/flyctl/cmd/fly_agent_run) - Run the Fly agent in the foreground
* [start](/flyctl/cmd/fly_agent_start) - Start the Fly agent
* [stop](/flyctl/cmd/fly_agent_stop) - Stop the Fly agent
## Options
```
-h, --help help for agent
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly agent ping
Source: https://docs.fly.io/flyctl/cmd/fly_agent_ping
Ping the Fly agent
## Usage
```
fly agent ping [flags]
```
## Options
```
-h, --help help for ping
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
# fly agent restart
Source: https://docs.fly.io/flyctl/cmd/fly_agent_restart
Restart the Fly agent
## Usage
```
fly agent restart [flags]
```
## Options
```
-h, --help help for restart
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
# fly agent run
Source: https://docs.fly.io/flyctl/cmd/fly_agent_run
Run the Fly agent in the foreground
## Usage
```
fly agent run [flags]
```
## Options
```
-h, --help help for run
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
# fly agent start
Source: https://docs.fly.io/flyctl/cmd/fly_agent_start
Start the Fly agent
## Usage
```
fly agent start [flags]
```
## Options
```
-h, --help help for start
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
# fly agent stop
Source: https://docs.fly.io/flyctl/cmd/fly_agent_stop
Stop the Fly agent
## Usage
```
fly agent stop [flags]
```
## Options
```
-h, --help help for stop
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
# fly apps
Source: https://docs.fly.io/flyctl/cmd/fly_apps
Manage your Fly applications.
## Usage
```
fly apps [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_apps_create) - Create a new application.
* [destroy](/flyctl/cmd/fly_apps_destroy) - Permanently destroy one or more apps.
* [errors](/flyctl/cmd/fly_apps_errors) - View application errors on Sentry.io
* [list](/flyctl/cmd/fly_apps_list) - List applications.
* [move](/flyctl/cmd/fly_apps_move) - Move an app to another organization.
* [open](/flyctl/cmd/fly_apps_open) - Open browser to current deployed application
* [releases](/flyctl/cmd/fly_apps_releases) - List app releases
* [restart](/flyctl/cmd/fly_apps_restart) - Restart an application.
## Options
```
-h, --help help for apps
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly apps create
Source: https://docs.fly.io/flyctl/cmd/fly_apps_create
Create a new application on the Fly platform.
This command won't generate a fly.toml configuration file, but you can
fetch one with 'fly config save -a \'.
## Usage
```
fly apps create [flags]
```
## Options
```
--generate-name Generate an app name
-h, --help help for create
-j, --json JSON output
--name string The app name to use
--network string Specify custom network id
-o, --org string The target Fly.io organization
--save Save the app name to the config file
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps destroy
Source: https://docs.fly.io/flyctl/cmd/fly_apps_destroy
Delete one or more applications from the Fly platform.
## Usage
```
fly apps destroy [flags]
```
## Options
```
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps errors
Source: https://docs.fly.io/flyctl/cmd/fly_apps_errors
View application errors on Sentry.io
## Usage
```
fly apps errors [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for errors
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps list
Source: https://docs.fly.io/flyctl/cmd/fly_apps_list
List the applications currently
available to this user. The list includes applications
from all the organizations the user is a member of. The list shows
the name, owner (org), status, and date/time of latest deploy for each app.
Apps on a non-default network also show the network name.
## Usage
```
fly apps list [flags]
```
## Options
```
-h, --help help for list
-j, --json JSON output
-o, --org string The target Fly.io organization
-q, --quiet Only list app names
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps move
Source: https://docs.fly.io/flyctl/cmd/fly_apps_move
Move an application to another
organization the current user belongs to.
For details, see [https://fly.io/docs/apps/move-app-org/](https://fly.io/docs/apps/move-app-org/).
## Usage
```
fly apps move [flags]
```
## Options
```
-h, --help help for move
-o, --org string The target Fly.io organization
--skip-health-checks Update machines without waiting for health checks
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps open
Source: https://docs.fly.io/flyctl/cmd/fly_apps_open
Open browser to current deployed application. If an optional relative URI is specified, it is appended
to the root URL of the deployed application.
## Usage
```
fly apps open [RELATIVE_URI] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for open
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps releases
Source: https://docs.fly.io/flyctl/cmd/fly_apps_releases
List all the releases of the application onto the Fly platform,
including type, when, success/fail and which user triggered the release.
## Usage
```
fly apps releases [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for releases
--image Display the Docker image reference of the release
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly apps restart
Source: https://docs.fly.io/flyctl/cmd/fly_apps_restart
Restart an application. Perform a rolling restart against all running Machines.
## Usage
```
fly apps restart [flags]
```
## Options
```
--force-stop Performs a force stop against the target Machine
-h, --help help for restart
--skip-health-checks Restarts app without waiting for health checks
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly apps](/flyctl/cmd/fly_apps) - Manage apps.
# fly auth
Source: https://docs.fly.io/flyctl/cmd/fly_auth
Authenticate with Fly (and logout if you need to).
If you do not have an account, start with the AUTH SIGNUP command.
If you do have an account, begin with the AUTH LOGIN subcommand.
## Usage
```
fly auth [command] [flags]
```
## Available Commands
* [docker](/flyctl/cmd/fly_auth_docker) - Authenticate docker
* [login](/flyctl/cmd/fly_auth_login) - Log in a user
* [logout](/flyctl/cmd/fly_auth_logout) - Logs out the currently logged in user
* [signup](/flyctl/cmd/fly_auth_signup) - Create a new fly account
* [whoami](/flyctl/cmd/fly_auth_whoami) - Displays the users email address/service identity currently
authenticated and in use.
## Options
```
-h, --help help for auth
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly auth docker
Source: https://docs.fly.io/flyctl/cmd/fly_auth_docker
Adds registry.fly.io to the Docker daemon's authenticated
registries. This allows you to push images directly to Fly.io from
the Docker CLI. Note that tokens may expire.
## Usage
```
fly auth docker [flags]
```
## Options
```
-h, --help help for docker
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly auth](/flyctl/cmd/fly_auth) - Manage authentication
# fly auth login
Source: https://docs.fly.io/flyctl/cmd/fly_auth_login
Logs a user into the Fly platform. Supports browser-based,
email/password and one-time-password authentication. Defaults to using
browser-based authentication.
Use --code to finish a browser login that is waiting for approval, with
the one-time code the approval page shows: for example when the browser
is on another machine, or the waiting command was stopped.
## Usage
```
fly auth login [flags]
```
## Options
```
--code string Finish a login that is waiting for approval, with the one-time code its approval page shows
--email string Login email
-h, --help help for login
-i, --interactive Log in with an email and password interactively
--otp string One time password
--password string Login password
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly auth](/flyctl/cmd/fly_auth) - Manage authentication
# fly auth logout
Source: https://docs.fly.io/flyctl/cmd/fly_auth_logout
Log the currently logged-in user out of the Fly platform.
To continue interacting with Fly, the user will need to log in again.
## Usage
```
fly auth logout [flags]
```
## Options
```
--force Remove the saved access token even if it could not be revoked
-h, --help help for logout
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly auth](/flyctl/cmd/fly_auth) - Manage authentication
# fly auth signup
Source: https://docs.fly.io/flyctl/cmd/fly_auth_signup
Creates a new fly account. The command opens the browser
and sends the user to a form to provide appropriate credentials.
## Usage
```
fly auth signup [flags]
```
## Options
```
-h, --help help for signup
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly auth](/flyctl/cmd/fly_auth) - Manage authentication
# fly auth whoami
Source: https://docs.fly.io/flyctl/cmd/fly_auth_whoami
Show the currently authenticated user
## Usage
```
fly auth whoami [flags]
```
## Options
```
-h, --help help for whoami
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly auth](/flyctl/cmd/fly_auth) - Manage authentication
# fly certs
Source: https://docs.fly.io/flyctl/cmd/fly_certs
Manages the certificates associated with a deployed application.
Certificates are created by associating a hostname/domain with the application.
When Fly is then able to validate that hostname/domain, the platform gets
certificates issued for the hostname/domain by Let's Encrypt.
## Usage
```
fly certs [command] [flags]
```
## Available Commands
* [add](/flyctl/cmd/fly_certs_add) - Add a certificate for an app
* [check](/flyctl/cmd/fly_certs_check) - Show certificate and DNS status
* [import](/flyctl/cmd/fly_certs_import) - Import a custom certificate
* [list](/flyctl/cmd/fly_certs_list) - List certificates for an app
* [remove](/flyctl/cmd/fly_certs_remove) - Removes a certificate from an app
* [setup](/flyctl/cmd/fly_certs_setup) - Shows certificate setup instructions
## Options
```
-h, --help help for certs
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly certs add
Source: https://docs.fly.io/flyctl/cmd/fly_certs_add
Add a certificate for an application. Takes a hostname
as a parameter for the certificate.
## Usage
```
fly certs add [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for add
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly certs check
Source: https://docs.fly.io/flyctl/cmd/fly_certs_check
Shows detailed certificate information and checks the DNS configuration
for the specified hostname.
## Usage
```
fly certs check [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for check
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly certs import
Source: https://docs.fly.io/flyctl/cmd/fly_certs_import
Import a custom TLS certificate for a hostname.
Upload your own certificate and private key in PEM format. Requires domain
ownership verification via DNS before the certificate becomes active.
## Usage
```
fly certs import [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
--fullchain string Path to certificate chain file (PEM format)
-h, --help help for import
-j, --json JSON output
--private-key string Path to private key file (PEM format)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly certs list
Source: https://docs.fly.io/flyctl/cmd/fly_certs_list
List the certificates associated with a deployed application.
## Usage
```
fly certs list [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly certs remove
Source: https://docs.fly.io/flyctl/cmd/fly_certs_remove
Removes a certificate from an application. Takes hostname
as a parameter to locate the certificate.
Use --custom to remove only the custom certificate while keeping ACME certificates.
Use --acme to stop ACME certificate issuance while keeping custom certificates.
## Usage
```
fly certs remove [flags]
```
## Options
```
--acme Stop ACME certificate issuance, keeping custom certificates
-a, --app string Application name
-c, --config string Path to application configuration file
--custom Remove only the custom certificate, keeping ACME certificates
-h, --help help for remove
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly certs setup
Source: https://docs.fly.io/flyctl/cmd/fly_certs_setup
Shows setup instructions for configuring DNS records for a certificate.
Takes hostname as a parameter to show the setup instructions for that certificate.
## Usage
```
fly certs setup [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for setup
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly certs](/flyctl/cmd/fly_certs) - Manage certificates
# fly checks
Source: https://docs.fly.io/flyctl/cmd/fly_checks
Manage health checks
## Usage
```
fly checks [command] [flags]
```
## Available Commands
* [list](/flyctl/cmd/fly_checks_list) - List health checks
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for checks
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly checks list
Source: https://docs.fly.io/flyctl/cmd/fly_checks_list
List health checks
## Usage
```
fly checks list [flags]
```
## Options
```
-a, --app string Application name
--check-name string Filter checks by name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly checks](/flyctl/cmd/fly_checks) - Manage health checks
# fly config
Source: https://docs.fly.io/flyctl/cmd/fly_config
The CONFIG commands allow you to work with an application's configuration.
## Usage
```
fly config [command] [flags]
```
## Available Commands
* [env](/flyctl/cmd/fly_config_env) - Display an app's runtime environment variables
* [save](/flyctl/cmd/fly_config_save) - Save an app's config file
* [show](/flyctl/cmd/fly_config_show) - Show an app's configuration
* [validate](/flyctl/cmd/fly_config_validate) - Validate an app's config file
## Options
```
-h, --help help for config
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly config env
Source: https://docs.fly.io/flyctl/cmd/fly_config_env
Display an app's runtime environment variables. It displays a section for
secrets and another for config file defined environment variables.
## Usage
```
fly config env [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for env
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly config](/flyctl/cmd/fly_config) - Manage an app's configuration
# fly config save
Source: https://docs.fly.io/flyctl/cmd/fly_config_save
Save an application's configuration locally. The configuration data is
retrieved from the Fly service and saved in TOML format.
## Usage
```
fly config save [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for save
--json Output the configuration in JSON format
--yaml Output the configuration in YAML format
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly config](/flyctl/cmd/fly_config) - Manage an app's configuration
# fly config show
Source: https://docs.fly.io/flyctl/cmd/fly_config_show
Show an application's configuration. The configuration is presented by default
in JSON format. The configuration data is retrieved from the Fly service.
## Usage
```
fly config show [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for show
--local Parse and show local fly.toml file instead of fetching from the Fly service
--toml Show configuration in TOML format
--yaml Show configuration in YAML format
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly config](/flyctl/cmd/fly_config) - Manage an app's configuration
# fly config validate
Source: https://docs.fly.io/flyctl/cmd/fly_config_validate
Validates an application's config file against the Fly platform to
ensure it is correct and meaningful to the platform.
## Usage
```
fly config validate [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for validate
-s, --strict Enable strict validation to check for unrecognized sections and keys
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly config](/flyctl/cmd/fly_config) - Manage an app's configuration
# fly console
Source: https://docs.fly.io/flyctl/cmd/fly_console
Run a console in a new or existing machine. The console command is
specified by the `console_command` configuration field. By default, a
new machine is created by default using the app's most recently deployed
image. An existing machine can be used instead with --machine.
## Usage
```
fly console [flags]
```
## Options
```
-a, --app string Application name
--build-context-warn-size string Warn when the Docker build context is larger than this. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb). Set to 0 to disable. Also set with FLY_BUILD_CONTEXT_WARN_SIZE.
-C, --command string command to run on SSH session
-c, --config string Path to application configuration file
--container string Container to connect to
--dockerfile string Path to a Dockerfile. Defaults to the Dockerfile in the working directory.
--entrypoint string ENTRYPOINT replacement
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--file-literal stringArray Set of literals to write to the Machine, in the form of /path/inside/machine=VALUE pairs, where VALUE is the base64-encoded raw content. Can be specified multiple times.
--file-local stringArray Set of files to write to the Machine, in the form of /path/inside/machine= pairs. Can be specified multiple times.
--file-secret stringArray Set of secrets to write to the Machine, in the form of /path/inside/machine=SECRET pairs, where SECRET is the name of the secret. The content of the secret must be base64 encoded. Can be specified multiple times.
-h, --help help for console
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
-i, --image string image to use (default: current release)
--machine string Run the console in the existing machine with the specified ID
-p, --port strings Publish ports, format: port[:machinePort][/protocol[:handler[:handler...]]])
i.e.: --port 80/tcp --port 443:80/tcp:http:tls --port 5432/tcp:pg_tls
To remove a port mapping use '-' as handler, i.e.: --port 80/tcp:-
-r, --region string The target region (see 'flyctl platform regions')
-s, --select Select the machine and container on which to execute the console from a list.
-u, --user string Unix username to connect as (default "root")
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
--volume strings Volume to mount, in the form of :/path/inside/machine[:]
--wg Determines whether communication with remote builders are conducted over wireguard or plain internet(https) (default true)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly consul
Source: https://docs.fly.io/flyctl/cmd/fly_consul
Enable and manage Consul clusters
## Usage
```
fly consul [command] [flags]
```
## Available Commands
* [attach](/flyctl/cmd/fly_consul_attach) - Attach Consul cluster to an app
* [detach](/flyctl/cmd/fly_consul_detach) - Detach Consul cluster from an app
## Options
```
-h, --help help for consul
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly consul attach
Source: https://docs.fly.io/flyctl/cmd/fly_consul_attach
Attach Consul cluster to an app, and setting the FLY\_CONSUL\_URL secret
## Usage
```
fly consul attach [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for attach
--variable-name string The environment variable name that will be added to the consuming app. (default "FLY_CONSUL_URL")
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly consul](/flyctl/cmd/fly_consul) - Enable and manage Consul clusters
# fly consul detach
Source: https://docs.fly.io/flyctl/cmd/fly_consul_detach
Detach Consul cluster from an app, and unsetting the FLY\_CONSUL\_URL secret
## Usage
```
fly consul detach [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for detach
--variable-name string The secret name that will be removed from the app. (default "FLY_CONSUL_URL")
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly consul](/flyctl/cmd/fly_consul) - Enable and manage Consul clusters
# fly dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_dashboard
Open web browser on Fly Web UI for this application
## Usage
```
fly dashboard [flags]
```
## Available Commands
* [metrics](/flyctl/cmd/fly_dashboard_metrics) - Open web browser on Fly Web UI for this app's metrics
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly dashboard metrics
Source: https://docs.fly.io/flyctl/cmd/fly_dashboard_metrics
Open web browser on Fly Web UI for this application's metrics
## Usage
```
fly dashboard metrics [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-g, --grafana Open Grafana metrics dashboard directly
-h, --help help for metrics
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly dashboard](/flyctl/cmd/fly_dashboard) - Open web browser on Fly Web UI for this app
# fly deploy
Source: https://docs.fly.io/flyctl/cmd/fly_deploy
Deploy Fly applications from source or an image using a local or remote builder.
To disable colorized output and show full Docker build output, set the environment variable NO\_COLOR=1.
## Usage
```
fly deploy [WORKING_DIRECTORY] [flags]
```
## Options
```
-a, --app string Application name
--build-arg stringArray Set of build time variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--build-context-warn-size string Warn when the Docker build context is larger than this. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb). Set to 0 to disable. Also set with FLY_BUILD_CONTEXT_WARN_SIZE.
--build-only Build but do not deploy
--build-secret stringArray Set of build secrets of NAME=VALUE pairs. Can be specified multiple times. See https://docs.docker.com/engine/reference/commandline/buildx_build/#secret
--build-target string Set the target build stage to build if the Dockerfile has more than one stage
--buildkit Deploy using buildkit-based remote builder
--buildpacks-docker-host string Address to docker daemon that will be exposed to the build container.
If not set (or set to empty string) the standard socket location will be used.
Special value 'inherit' may be used in which case DOCKER_HOST environment variable will be used.
This option may set DOCKER_HOST environment variable for the build container if needed.
--buildpacks-volume strings Mount host volume into the build container, in the form ':[:]'.
- 'host path': Name of the volume or absolute directory path to mount.
- 'target path': The path where the file or directory is available in the container.
- 'options' (default "ro"): An optional comma separated list of mount options.
- "ro", volume contents are read-only.
- "rw", volume contents are readable and writeable.
- "volume-opt==", can be specified more than once, takes a key-value pair consisting of the option name and its value.
Repeat for each volume in order (comma-separated lists not accepted)
--compression string Compression algorithm to use for the image. Options are "zstd" or "gzip". Defaults to "gzip". (default "gzip")
--compression-level int Compression level to use for the image. Defaults to 7. (default 7)
-c, --config string Path to application configuration file
--deploy-retries string Number of times to retry a deployment if it fails (default "auto")
--depot string[="true"] Deploy using depot to build the image (default "auto")
--depot-scope string The scope of the Depot builder's cache to use (org or app) (default "org")
--detach Return immediately instead of monitoring deployment progress
--dns-checks Perform DNS checks during deployment (default true)
--dockerfile string Path to a Dockerfile. Defaults to the Dockerfile in the working directory.
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--exclude-machines strings Deploy to all machines except machines with these IDs. Multiple IDs can be specified with comma separated values or by providing the flag multiple times.
--exclude-regions strings Deploy to all machines except machines in these regions. Multiple regions can be specified with comma separated values or by providing the flag multiple times.
--file-literal stringArray Set of literals in the form of /path/inside/machine=VALUE pairs where VALUE is the content. Can be specified multiple times.
--file-local stringArray Set of files in the form of /path/inside/machine= pairs. Can be specified multiple times.
--file-secret stringArray Set of secrets in the form of /path/inside/machine=SECRET pairs where SECRET is the name of the secret. Can be specified multiple times.
--flycast Allocate a private IPv6 addresses
--ha Create spare machines that increases app availability (default true)
-h, --help help for deploy
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
--https-failover Determines whether to failover to plain internet(https) communication with remote builders if wireguard fails (also --http-failover) (default true)
--ignorefile string Path to a Docker ignore file. Defaults to the .dockerignore file in the working directory.
-i, --image string The Docker image to deploy
--image-label string Image label to use when tagging and pushing to the fly registry. Defaults to "deployment-{timestamp}".
--label stringArray Add custom metadata to an image via docker labels
--lease-timeout string Time duration to lease individual machines while running deployment. All machines are leased at the beginning and released at the end.The lease is refreshed periodically for this same time, which is why it is short.flyctl releases leases in most cases. (default "13s")
--local-only Perform builds locally using the local docker daemon. The default is --remote-only.
--max-concurrent int Maximum number of machines to operate on concurrently. (default 8)
--max-unavailable float Max number of unavailable machines during rolling updates. A number between 0 and 1 means percent of total machines (default 0.33)
--nixpacks Deploy using nixpacks to build the image
--no-cache Do not use the build cache when building the image
--no-public-ips Do not allocate any new public IP addresses
--now Deploy now without confirmation
--only-machines strings Deploy to machines only with these IDs. Multiple IDs can be specified with comma separated values or by providing the flag multiple times.
--primary-region string Override primary region in fly.toml configuration.
--process-groups strings Deploy to machines only in these process groups
--push Push image to registry after build is complete
--recreate-builder Recreate the builder app, if it exists
--regions strings Deploy to machines only in these regions. Multiple regions can be specified with comma separated values or by providing the flag multiple times.
--release-command-timeout string Time duration to wait for a release command finish running, or 'none' to disable. (default "5m0s")
--remote-only Perform builds on a remote builder instance instead of using the local docker daemon. This is the default. Use --local-only to build locally.
-s, --signal string Signal to stop the machine with for bluegreen strategy (default: SIGINT)
--skip-release-command Do not run the release command during deployment.
--smoke-checks Perform smoke checks during deployment (default true)
--strategy string The strategy for replacing running instances. Options are canary, rolling, bluegreen, or immediate. The default strategy is rolling.
--update-only Do not create Machines for new process groups
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
--volume-initial-size int The initial size in GB for volumes created on first deploy
--wait-timeout string Time duration to wait for individual machines to transition states and become healthy. (default "5m0s")
--wg Determines whether communication with remote builders are conducted over wireguard or plain internet(https) (default true)
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly dig
Source: https://docs.fly.io/flyctl/cmd/fly_dig
Make DNS requests against Fly.io's internal DNS server. Valid types include
AAAA and TXT (the two types our servers answer authoritatively), AAAA-NATIVE
and TXT-NATIVE, which resolve with Go's resolver (they're slower,
but may be useful if diagnosing a DNS bug) and A and CNAME
(if you're using the server to test recursive lookups.)
Note that this resolves names against the server for the current organization. You can
set the organization with -o \; otherwise, the command uses the organization
attached to the current app (you can pass an app in with -a \).
## Usage
```
fly dig [type] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dig
-o, --org string The target Fly.io organization
-s, --short Just print the answers, not DNS record details
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly docs
Source: https://docs.fly.io/flyctl/cmd/fly_docs
View Fly documentation on the Fly.io website. This command will open a
browser to view the content.
## Usage
```
fly docs [flags]
```
## Options
```
-h, --help help for docs
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly doctor
Source: https://docs.fly.io/flyctl/cmd/fly_doctor
The DOCTOR command allows you to debug your Fly environment
## Usage
```
fly doctor [flags]
```
## Available Commands
* [diag](/flyctl/cmd/fly_doctor_diag) - Send diagnostic information about your applications back to Fly.io.
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for doctor
-j, --json JSON output
-o, --org string The name of the organization to use for WireGuard tests. (default "personal")
-v, --verbose Print extra diagnostic information.
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly doctor diag
Source: https://docs.fly.io/flyctl/cmd/fly_doctor_diag
Send diagnostic information about your applications back to Fly.io,
to help diagnose problems.
This command will collect some local system information and a few files
that you'd be sending us anyways in order to deploy, notably any Dockerfiles
you might have associated with this application.
## Usage
```
fly doctor diag [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
--force Send diagnostics even if we can't find your local Fly.io app
-h, --help help for diag
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly doctor](/flyctl/cmd/fly_doctor) - The DOCTOR command allows you to debug your Fly environment
# fly extensions
Source: https://docs.fly.io/flyctl/cmd/fly_extensions
Extensions are additional functionality that can be added to your Fly apps
## Usage
```
fly extensions [command] [flags]
```
## Available Commands
* [arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
* [kubernetes](/flyctl/cmd/fly_extensions_kubernetes) - Provision and manage Kubernetes clusters
* [sentry](/flyctl/cmd/fly_extensions_sentry) - Setup a Sentry project for this app
* [storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
* [vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
* [wafris](/flyctl/cmd/fly_extensions_wafris) - Provision and manage Wafris WAFs (Web Application Firewalls)
## Options
```
-h, --help help for extensions
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly extensions arcjet
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet
Provision and manage Arcjet
## Usage
```
fly extensions arcjet [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_arcjet_create) - Create an Arcjet site
* [dashboard](/flyctl/cmd/fly_extensions_arcjet_dashboard) - Visit the Arcjet dashboard
* [destroy](/flyctl/cmd/fly_extensions_arcjet_destroy) - Permanently destroy an Arcjet site
* [list](/flyctl/cmd/fly_extensions_arcjet_list) - List your Arcjet sites
* [status](/flyctl/cmd/fly_extensions_arcjet_status) - Show details about an Arcjet site setup
## Options
```
-h, --help help for arcjet
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions arcjet create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet_create
Create an Arcjet site
## Usage
```
fly extensions arcjet create [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for create
-n, --name string The name of your application
-o, --org string The target Fly.io organization
-r, --region string The target region (see 'flyctl platform regions')
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
# fly extensions arcjet dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet_dashboard
Visit the Arcjet dashboard
## Usage
```
fly extensions arcjet dashboard [site_name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
# fly extensions arcjet destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet_destroy
Permanently destroy an Arcjet site
## Usage
```
fly extensions arcjet destroy [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
# fly extensions arcjet list
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet_list
List your Arcjet sites
## Usage
```
fly extensions arcjet list [flags]
```
## Options
```
-h, --help help for list
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
# fly extensions arcjet status
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_arcjet_status
Show details about an Arcjet site setup
## Usage
```
fly extensions arcjet status [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for status
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions arcjet](/flyctl/cmd/fly_extensions_arcjet) - Provision and manage Arcjet
# fly extensions kubernetes
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_kubernetes
Provision and manage Kubernetes clusters
## Usage
```
fly extensions kubernetes [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_kubernetes_create) - Provision a Kubernetes cluster for an organization
* [destroy](/flyctl/cmd/fly_extensions_kubernetes_destroy) - Permanently destroy a Kubernetes cluster
* [list](/flyctl/cmd/fly_extensions_kubernetes_list) - List your Kubernetes clusters
* [save-kubeconfig](/flyctl/cmd/fly_extensions_kubernetes_save-kubeconfig) - Save the kubeconfig file of your cluster
## Options
```
-h, --help help for kubernetes
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions kubernetes create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_kubernetes_create
Provision a Kubernetes cluster for an organization
## Usage
```
fly extensions kubernetes create [flags]
```
## Options
```
-h, --help help for create
-n, --name string The name of your cluster
-o, --org string The target Fly.io organization
--output string The output path to save the kubeconfig file
-r, --region string The target region (see 'flyctl platform regions')
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions kubernetes](/flyctl/cmd/fly_extensions_kubernetes) - Provision and manage Kubernetes clusters
# fly extensions kubernetes destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_kubernetes_destroy
Permanently destroy a Kubernetes cluster
## Usage
```
fly extensions kubernetes destroy [name] [flags]
```
## Options
```
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions kubernetes](/flyctl/cmd/fly_extensions_kubernetes) - Provision and manage Kubernetes clusters
# fly extensions kubernetes list
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_kubernetes_list
List your Kubernetes clusters
## Usage
```
fly extensions kubernetes list [flags]
```
## Options
```
-h, --help help for list
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions kubernetes](/flyctl/cmd/fly_extensions_kubernetes) - Provision and manage Kubernetes clusters
# fly extensions kubernetes save-kubeconfig
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_kubernetes_save-kubeconfig
Save the kubeconfig file of your cluster
## Usage
```
fly extensions kubernetes save-kubeconfig [cluster name] [flags]
```
## Options
```
-h, --help help for save-kubeconfig
--output string The output path to save the kubeconfig file
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions kubernetes](/flyctl/cmd/fly_extensions_kubernetes) - Provision and manage Kubernetes clusters
# fly extensions sentry
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_sentry
Setup a Sentry project for this app
## Usage
```
fly extensions sentry [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_sentry_create) - Provision a Sentry project for a Fly.io app
* [dashboard](/flyctl/cmd/fly_extensions_sentry_dashboard) - View application errors in the Sentry dashboard
* [destroy](/flyctl/cmd/fly_extensions_sentry_destroy) - Permanently destroy a Sentry project
* [list](/flyctl/cmd/fly_extensions_sentry_list) - List your Sentry projects
## Options
```
-h, --help help for sentry
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions sentry create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_sentry_create
Provision a Sentry project for a Fly.io app
## Usage
```
fly extensions sentry create [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for create
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions sentry](/flyctl/cmd/fly_extensions_sentry) - Setup a Sentry project for this app
# fly extensions sentry dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_sentry_dashboard
View application errors in the Sentry dashboard
## Usage
```
fly extensions sentry dashboard [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions sentry](/flyctl/cmd/fly_extensions_sentry) - Setup a Sentry project for this app
# fly extensions sentry destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_sentry_destroy
Permanently destroy a Sentry project
## Usage
```
fly extensions sentry destroy [project-name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions sentry](/flyctl/cmd/fly_extensions_sentry) - Setup a Sentry project for this app
# fly extensions sentry list
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_sentry_list
List your Sentry projects
## Usage
```
fly extensions sentry list [flags]
```
## Options
```
-h, --help help for list
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions sentry](/flyctl/cmd/fly_extensions_sentry) - Setup a Sentry project for this app
# fly extensions storage
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage
Provision and manage Tigris object storage buckets
## Usage
```
fly extensions storage [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_storage_create) - Provision a Tigris object storage bucket
* [dashboard](/flyctl/cmd/fly_extensions_storage_dashboard) - Visit the Tigris dashboard
* [destroy](/flyctl/cmd/fly_extensions_storage_destroy) - Permanently destroy a Tigris object storage bucket
* [list](/flyctl/cmd/fly_extensions_storage_list) - List your Tigris object storage buckets
* [status](/flyctl/cmd/fly_extensions_storage_status) - Show details about a Tigris storage bucket
* [update](/flyctl/cmd/fly_extensions_storage_update) - Update an existing Tigris object storage bucket
## Options
```
-h, --help help for storage
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions storage create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_create
Provision a Tigris object storage bucket
## Usage
```
fly extensions storage create [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for create
-n, --name string The name of your bucket
-o, --org string The target Fly.io organization
-p, --public Objects in the bucket should be publicly accessible
--shadow-access-key string Shadow bucket access key
--shadow-endpoint string Shadow bucket endpoint
--shadow-name string Shadow bucket name
--shadow-region string Shadow bucket region
--shadow-secret-key string Shadow bucket secret key
--shadow-write-through Write objects through to the shadow bucket
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions storage dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_dashboard
Visit the Tigris dashboard
## Usage
```
fly extensions storage dashboard [bucket_name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions storage destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_destroy
Permanently destroy a Tigris object storage bucket
## Usage
```
fly extensions storage destroy [storage-bucket-name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions storage list
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_list
List your Tigris object storage buckets
## Usage
```
fly extensions storage list [flags]
```
## Options
```
-h, --help help for list
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions storage status
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_status
Show details about a Tigris storage bucket
## Usage
```
fly extensions storage status [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for status
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions storage update
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_storage_update
Update an existing Tigris object storage bucket
## Usage
```
fly extensions storage update [flags]
```
## Options
```
-a, --app string Application name
--clear-custom-domain Remove a custom domain from a bucket
--clear-shadow Remove an existing shadow bucket
-c, --config string Path to application configuration file
--custom-domain string A custom domain name pointing at your bucket
-h, --help help for update
-o, --org string The target Fly.io organization
--private Set a public bucket to be private
-p, --public Objects in the bucket should be publicly accessible
--shadow-access-key string Shadow bucket access key
--shadow-endpoint string Shadow bucket endpoint
--shadow-name string Shadow bucket name
--shadow-region string Shadow bucket region
--shadow-secret-key string Shadow bucket secret key
--shadow-write-through Write objects through to the shadow bucket
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions storage](/flyctl/cmd/fly_extensions_storage) - Provision and manage Tigris object storage buckets
# fly extensions vector
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector
Provision and manage Upstash Vector index
## Usage
```
fly extensions vector [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_vector_create) - Provision a Upstash Vector index
* [dashboard](/flyctl/cmd/fly_extensions_vector_dashboard) - Visit the Upstash Vector dashboard on the Upstash web console
* [destroy](/flyctl/cmd/fly_extensions_vector_destroy) - Permanently destroy an Upstash Vector index
* [list](/flyctl/cmd/fly_extensions_vector_list) - List your Upstash Vector index
* [status](/flyctl/cmd/fly_extensions_vector_status) - Show details about an Upstash Vector index
## Options
```
-h, --help help for vector
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions vector create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector_create
Provision a Upstash Vector index
## Usage
```
fly extensions vector create [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for create
-n, --name string The name of your cluster
-o, --org string The target Fly.io organization
-r, --region string The target region (see 'flyctl platform regions')
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
# fly extensions vector dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector_dashboard
Visit the Upstash Vector dashboard on the Upstash web console
## Usage
```
fly extensions vector dashboard [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
# fly extensions vector destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector_destroy
Permanently destroy an Upstash Vector index
## Usage
```
fly extensions vector destroy [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
# fly extensions vector list
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector_list
List your Upstash Vector index
## Usage
```
fly extensions vector list [flags]
```
## Options
```
-h, --help help for list
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
# fly extensions vector status
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_vector_status
Show details about an Upstash Vector index
## Usage
```
fly extensions vector status [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for status
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions vector](/flyctl/cmd/fly_extensions_vector) - Provision and manage Upstash Vector index
# fly extensions wafris
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_wafris
Provision and manage Wafris WAFs (Web Application Firewalls)
## Usage
```
fly extensions wafris [command] [flags]
```
## Available Commands
* [create](/flyctl/cmd/fly_extensions_wafris_create) - Provision a Wafris WAF
* [dashboard](/flyctl/cmd/fly_extensions_wafris_dashboard) - Visit the Wafris dashboard
* [destroy](/flyctl/cmd/fly_extensions_wafris_destroy) - Permanently destroy a Wafris WAF
## Options
```
-h, --help help for wafris
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
# fly extensions wafris create
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_wafris_create
Provision a Wafris WAF
## Usage
```
fly extensions wafris create [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for create
-n, --name string The name of your WAF
-o, --org string The target Fly.io organization
-r, --region string The target region (see 'flyctl platform regions')
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions wafris](/flyctl/cmd/fly_extensions_wafris) - Provision and manage Wafris WAFs (Web Application Firewalls)
# fly extensions wafris dashboard
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_wafris_dashboard
Visit the Wafris dashboard
## Usage
```
fly extensions wafris dashboard [firewall_name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for dashboard
-o, --org string The target Fly.io organization
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions wafris](/flyctl/cmd/fly_extensions_wafris) - Provision and manage Wafris WAFs (Web Application Firewalls)
# fly extensions wafris destroy
Source: https://docs.fly.io/flyctl/cmd/fly_extensions_wafris_destroy
Permanently destroy a Wafris WAF
## Usage
```
fly extensions wafris destroy [name] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for destroy
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly extensions wafris](/flyctl/cmd/fly_extensions_wafris) - Provision and manage Wafris WAFs (Web Application Firewalls)
# fly
Source: https://docs.fly.io/flyctl/cmd/fly_help
This is flyctl, the Fly.io command line interface.
## Usage
```
fly [flags]
```
## Available Commands
* [agent](/flyctl/cmd/fly_agent) - Commands that manage the Fly agent, a background process that manages flyctl wireguard connections
* [apps](/flyctl/cmd/fly_apps) - Manage apps.
* [auth](/flyctl/cmd/fly_auth) - Manage authentication
* [certs](/flyctl/cmd/fly_certs) - Manage certificates
* [checks](/flyctl/cmd/fly_checks) - Manage health checks
* [config](/flyctl/cmd/fly_config) - Manage an app's configuration
* [console](/flyctl/cmd/fly_console) - Run a console in a new or existing machine
* [consul](/flyctl/cmd/fly_consul) - Enable and manage Consul clusters
* [dashboard](/flyctl/cmd/fly_dashboard) - Open web browser on Fly Web UI for this app
* [deploy](/flyctl/cmd/fly_deploy) - Deploy Fly applications
* [dig](/flyctl/cmd/fly_dig) - Make DNS requests against Fly.io's internal DNS server
* [docs](/flyctl/cmd/fly_docs) - View Fly documentation
* [doctor](/flyctl/cmd/fly_doctor) - The DOCTOR command allows you to debug your Fly environment
* [extensions](/flyctl/cmd/fly_extensions) - Extensions are additional functionality that can be added to your Fly apps
* [image](/flyctl/cmd/fly_image) - Manage app image
* [incidents](/flyctl/cmd/fly_incidents) - Show incidents
* [ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
* [jobs](/flyctl/cmd/fly_jobs) - Show jobs at Fly.io
* [launch](/flyctl/cmd/fly_launch) - Create and configure a new app from source code or a Docker image
* [logs](/flyctl/cmd/fly_logs) - View app logs
* [machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
* [mcp](/flyctl/cmd/fly_mcp) - flyctl Model Context Protocol.
* [mpg](/flyctl/cmd/fly_mpg) - Manage Managed Postgres clusters.
* [orgs](/flyctl/cmd/fly_orgs) - Commands for managing Fly organizations
* [ping](/flyctl/cmd/fly_ping) - Test connectivity with ICMP ping messages
* [platform](/flyctl/cmd/fly_platform) - Fly platform information
* [postgres](/flyctl/cmd/fly_postgres) - Unmanaged Postgres cluster commands
* [proxy](/flyctl/cmd/fly_proxy) - Proxies connections to a Fly Machine.
* [redis](/flyctl/cmd/fly_redis) - Launch and manage Redis databases managed by Upstash.com
* [releases](/flyctl/cmd/fly_releases) - List app releases
* [scale](/flyctl/cmd/fly_scale) - Scale app resources
* [secrets](/flyctl/cmd/fly_secrets) - Manage application secrets with the set and unset commands.
* [services](/flyctl/cmd/fly_services) - Show the application's services
* [settings](/flyctl/cmd/fly_settings) - Manage flyctl settings
* [sftp](/flyctl/cmd/fly_sftp) - Get or put files from a remote VM.
* [ssh](/flyctl/cmd/fly_ssh) - Use SSH to log into or run commands on Machines
* [status](/flyctl/cmd/fly_status) - Show app status
* [storage](/flyctl/cmd/fly_storage) - Provision and manage Tigris object storage buckets
* [synthetics](/flyctl/cmd/fly_synthetics) - Synthetic monitoring
* [tokens](/flyctl/cmd/fly_tokens) - Manage Fly.io API tokens
* [version](/flyctl/cmd/fly_version) - Show version information for the flyctl command
* [volumes](/flyctl/cmd/fly_volumes) - Manage Fly Volumes.
* [wireguard](/flyctl/cmd/fly_wireguard) - Commands that manage WireGuard peer connections
## Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
-h, --help help for fly
--verbose Verbose output
```
## See Also
# fly image
Source: https://docs.fly.io/flyctl/cmd/fly_image
Manage app image
## Usage
```
fly image [command] [flags]
```
## Available Commands
* [show](/flyctl/cmd/fly_image_show) - Show image details.
* [update](/flyctl/cmd/fly_image_update) - Updates the app's image to the latest available version.
## Options
```
-h, --help help for image
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly image show
Source: https://docs.fly.io/flyctl/cmd/fly_image_show
Show image details.
## Usage
```
fly image show [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for show
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly image](/flyctl/cmd/fly_image) - Manage app image
# fly image update
Source: https://docs.fly.io/flyctl/cmd/fly_image_update
Update the app's image to the latest available version.
The update will perform a rolling restart against each Machine, which may result in a brief service disruption.
## Usage
```
fly image update [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for update
--image string Target a specific image
--skip-health-checks Skip waiting for health checks between VM updates.
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly image](/flyctl/cmd/fly_image) - Manage app image
# fly incidents
Source: https://docs.fly.io/flyctl/cmd/fly_incidents
Show incidents that may be affecting your organization.
## Usage
```
fly incidents [command] [flags]
```
## Available Commands
* [hosts](/flyctl/cmd/fly_incidents_hosts) - Show hosts' incidents
* [list](/flyctl/cmd/fly_incidents_list) - List active incidents.
## Options
```
-h, --help help for incidents
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly incidents hosts
Source: https://docs.fly.io/flyctl/cmd/fly_incidents_hosts
Show hosts' incidents affecting applications
## Usage
```
fly incidents hosts [command] [flags]
```
## Available Commands
* [list](/flyctl/cmd/fly_incidents_hosts_list) - List hosts' issues affecting the application
## Options
```
-h, --help help for hosts
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly incidents](/flyctl/cmd/fly_incidents) - Show incidents
# fly incidents hosts list
Source: https://docs.fly.io/flyctl/cmd/fly_incidents_hosts_list
List hosts' issues affecting the application
## Usage
```
fly incidents hosts list [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly incidents hosts](/flyctl/cmd/fly_incidents_hosts) - Show hosts' incidents
# fly incidents list
Source: https://docs.fly.io/flyctl/cmd/fly_incidents_list
List the active incidents that may be affecting your apps or deployments.
## Usage
```
fly incidents list [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
-o, --org string The target Fly.io organization
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly incidents](/flyctl/cmd/fly_incidents) - Show incidents
# fly ips
Source: https://docs.fly.io/flyctl/cmd/fly_ips
Commands for managing IP addresses associated with an application
## Usage
```
fly ips [command] [flags]
```
## Available Commands
* [allocate](/flyctl/cmd/fly_ips_allocate) - Allocate recommended IP addresses
* [allocate-egress](/flyctl/cmd/fly_ips_allocate-egress) - Allocate app-scoped egress IPs
* [allocate-v4](/flyctl/cmd/fly_ips_allocate-v4) - Allocate an IPv4 address
* [allocate-v6](/flyctl/cmd/fly_ips_allocate-v6) - Allocate an IPv6 address
* [list](/flyctl/cmd/fly_ips_list) - List allocated IP addresses
* [private](/flyctl/cmd/fly_ips_private) - List instances private IP addresses
* [release](/flyctl/cmd/fly_ips_release) - Release ingress IP addresses
* [release-egress](/flyctl/cmd/fly_ips_release-egress) - Release egress IP addresses
## Options
```
-h, --help help for ips
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly ips allocate
Source: https://docs.fly.io/flyctl/cmd/fly_ips_allocate
Allocate recommended IP addresses for the application
## Usage
```
fly ips allocate [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for allocate
-r, --region string The target region (see 'flyctl platform regions')
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips allocate-egress
Source: https://docs.fly.io/flyctl/cmd/fly_ips_allocate-egress
Allocates a pair of egress IP addresses for an app
## Usage
```
fly ips allocate-egress [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for allocate-egress
-r, --region string The target region (see 'flyctl platform regions')
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips allocate-v4
Source: https://docs.fly.io/flyctl/cmd/fly_ips_allocate-v4
Allocates an IPv4 address to the application
## Usage
```
fly ips allocate-v4 [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for allocate-v4
-r, --region string The target region (see 'flyctl platform regions')
--shared Allocates a shared IPv4
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips allocate-v6
Source: https://docs.fly.io/flyctl/cmd/fly_ips_allocate-v6
Allocates an IPv6 address to the application
## Usage
```
fly ips allocate-v6 [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for allocate-v6
--network string Target network name for a Flycast private IPv6 address
-o, --org string The target Fly.io organization
--private Allocate a private IPv6 address
-r, --region string The target region (see 'flyctl platform regions')
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips list
Source: https://docs.fly.io/flyctl/cmd/fly_ips_list
Lists the IP addresses allocated to the application
## Usage
```
fly ips list [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips private
Source: https://docs.fly.io/flyctl/cmd/fly_ips_private
List instances private IP addresses, accessible from within the Fly network
## Usage
```
fly ips private [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for private
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips release
Source: https://docs.fly.io/flyctl/cmd/fly_ips_release
Releases one or more ingress IP addresses from the application
## Usage
```
fly ips release [flags] ADDRESS ADDRESS ...
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for release
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly ips release-egress
Source: https://docs.fly.io/flyctl/cmd/fly_ips_release-egress
Releases one or more egress IP addresses from the application
## Usage
```
fly ips release-egress [flags] ADDRESS ADDRESS ...
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for release-egress
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly ips](/flyctl/cmd/fly_ips) - Manage IP addresses for apps
# fly jobs
Source: https://docs.fly.io/flyctl/cmd/fly_jobs
Show jobs at Fly.io, including maybe ones you should apply to
## Usage
```
fly jobs [flags]
```
## Available Commands
* [open](/flyctl/cmd/fly_jobs_open) - Open fly.io/jobs
## Options
```
-h, --help help for jobs
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly jobs open
Source: https://docs.fly.io/flyctl/cmd/fly_jobs_open
Open browser to [https://fly.io/jobs/](https://fly.io/jobs/)
## Usage
```
fly jobs open [flags]
```
## Options
```
-h, --help help for open
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly jobs](/flyctl/cmd/fly_jobs) - Show jobs at Fly.io
# fly launch
Source: https://docs.fly.io/flyctl/cmd/fly_launch
Create and configure a new app from source code or a Docker image. Options passed after double dashes ("--") will be passed to the language specific scanner/dockerfile generator.
## Usage
```
fly launch [flags]
```
## Options
```
--attach Attach this new application to the current application
--auto-stop string Automatically suspend the app after a period of inactivity. Valid values are 'off', 'stop', and 'suspend' (default "stop")
--build-arg stringArray Set of build time variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--build-context-warn-size string Warn when the Docker build context is larger than this. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb). Set to 0 to disable. Also set with FLY_BUILD_CONTEXT_WARN_SIZE.
--build-only Build but do not deploy
--build-secret stringArray Set of build secrets of NAME=VALUE pairs. Can be specified multiple times. See https://docs.docker.com/engine/reference/commandline/buildx_build/#secret
--build-target string Set the target build stage to build if the Dockerfile has more than one stage
--buildkit Deploy using buildkit-based remote builder
--buildpacks-docker-host string Address to docker daemon that will be exposed to the build container.
If not set (or set to empty string) the standard socket location will be used.
Special value 'inherit' may be used in which case DOCKER_HOST environment variable will be used.
This option may set DOCKER_HOST environment variable for the build container if needed.
--buildpacks-volume strings Mount host volume into the build container, in the form ':[:]'.
- 'host path': Name of the volume or absolute directory path to mount.
- 'target path': The path where the file or directory is available in the container.
- 'options' (default "ro"): An optional comma separated list of mount options.
- "ro", volume contents are read-only.
- "rw", volume contents are readable and writeable.
- "volume-opt==", can be specified more than once, takes a key-value pair consisting of the option name and its value.
Repeat for each volume in order (comma-separated lists not accepted)
--command string The command to override the Docker CMD.
--compression string Compression algorithm to use for the image. Options are "zstd" or "gzip". Defaults to "gzip". (default "gzip")
--compression-level int Compression level to use for the image. Defaults to 7. (default 7)
-c, --config string Path to application configuration file
--copy-config Use the configuration file if present without prompting
--db string[="true"] Provision a Postgres database. Options: mpg (managed postgres), upg/legacy (unmanaged postgres), or true (default type)
--deploy-retries string Number of times to retry a deployment if it fails (default "auto")
--depot string[="true"] Deploy using depot to build the image (default "auto")
--depot-scope string The scope of the Depot builder's cache to use (org or app) (default "org")
--detach Return immediately instead of monitoring deployment progress
--dns-checks Perform DNS checks during deployment (default true)
--dockerfile string Path to a Dockerfile. Defaults to the Dockerfile in the working directory.
--dockerignore-from-gitignore If a .dockerignore does not exist, create one from .gitignore files
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--exclude-machines strings Deploy to all machines except machines with these IDs. Multiple IDs can be specified with comma separated values or by providing the flag multiple times.
--exclude-regions strings Deploy to all machines except machines in these regions. Multiple regions can be specified with comma separated values or by providing the flag multiple times.
--file-literal stringArray Set of literals in the form of /path/inside/machine=VALUE pairs where VALUE is the content. Can be specified multiple times.
--file-local stringArray Set of files in the form of /path/inside/machine= pairs. Can be specified multiple times.
--file-secret stringArray Set of secrets in the form of /path/inside/machine=SECRET pairs where SECRET is the name of the secret. Can be specified multiple times.
--flycast Allocate a private IPv6 addresses
--from string A github repo URL to use as a template for the new app
--generate-name Always generate a name for the app, without prompting
--ha Create spare machines that increases app availability (default true)
-h, --help help for launch
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
--https-failover Determines whether to failover to plain internet(https) communication with remote builders if wireguard fails (also --http-failover) (default true)
--ignorefile string Path to a Docker ignore file. Defaults to the .dockerignore file in the working directory.
-i, --image string The Docker image to deploy
--image-label string Image label to use when tagging and pushing to the fly registry. Defaults to "deployment-{timestamp}".
--internal-port int Set internal_port for all services in the generated fly.toml (default -1)
--into string Destination directory for github repo specified with --from
--json Generate configuration in JSON format
--label stringArray Add custom metadata to an image via docker labels
--lease-timeout string Time duration to lease individual machines while running deployment. All machines are leased at the beginning and released at the end.The lease is refreshed periodically for this same time, which is why it is short.flyctl releases leases in most cases. (default "13s")
--local-only Perform builds locally using the local docker daemon. The default is --remote-only.
--max-concurrent int Maximum number of machines to operate on concurrently. (default 8)
--max-unavailable float Max number of unavailable machines during rolling updates. A number between 0 and 1 means percent of total machines (default 0.33)
--name string Name of the new app
--nixpacks Deploy using nixpacks to build the image
--no-cache Do not use the build cache when building the image
--no-db Skip automatically provisioning a database
--no-deploy Do not immediately deploy the new app after fly launch creates and configures it
--no-github-workflow Skip automatically provisioning a GitHub fly deploy workflow
--no-object-storage Skip automatically provisioning an object storage bucket
--no-public-ips Do not allocate any new public IP addresses
--no-redis Skip automatically provisioning a Redis instance
--now Deploy now without confirmation
--only-machines strings Deploy to machines only with these IDs. Multiple IDs can be specified with comma separated values or by providing the flag multiple times.
-o, --org string The target Fly.io organization
--path string Path to the app source root, where fly.toml file will be saved (default ".")
--primary-region string Override primary region in fly.toml configuration.
--process-groups strings Deploy to machines only in these process groups
--push Push image to registry after build is complete
--recreate-builder Recreate the builder app, if it exists
-r, --region string The target region (see 'flyctl platform regions')
--regions strings Deploy to machines only in these regions. Multiple regions can be specified with comma separated values or by providing the flag multiple times.
--release-command-timeout string Time duration to wait for a release command finish running, or 'none' to disable. (default "5m0s")
--remote-only Perform builds on a remote builder instance instead of using the local docker daemon. This is the default. Use --local-only to build locally.
--secret stringArray Set of secrets in the form of NAME=VALUE pairs. Can be specified multiple times.
-s, --signal string Signal to stop the machine with for bluegreen strategy (default: SIGINT)
--smoke-checks Perform smoke checks during deployment (default true)
--strategy string The strategy for replacing running instances. Options are canary, rolling, bluegreen, or immediate. The default strategy is rolling.
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
-v, --volume strings Volume to mount, in the form of :/path/inside/machine[:]
--volume-initial-size int The initial size in GB for volumes created on first deploy
--wait-timeout string Time duration to wait for individual machines to transition states and become healthy. (default "5m0s")
--wg Determines whether communication with remote builders are conducted over wireguard or plain internet(https) (default true)
--yaml Generate configuration in YAML format
-y, --yes Accept all confirmations (also --auto-confirm)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly logs
Source: https://docs.fly.io/flyctl/cmd/fly_logs
View application logs as generated by the application running on
the Fly platform.
Logs can be filtered to a specific machine using the --machine/-m flag or
to all machines running in a specific region using the --region/-r flag.
By default logs are continually streamed until the command is aborted.
Use --no-tail to only fetch the logs in the buffer.
## Usage
```
fly logs [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for logs
-j, --json JSON output
-m, --machine string Filter by machine ID
-n, --no-tail Do not continually stream logs
-r, --region string The target region (see 'flyctl platform regions')
-s, --select Select a machine from a list
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly machine
Source: https://docs.fly.io/flyctl/cmd/fly_machine
Manage Fly Machines. Fly Machines are super-fast, lightweight VMs that can be created,
and then quickly started and stopped as needed with flyctl commands or with the
Machines REST fly.
## Usage
```
fly machine [command] [flags]
```
## Available Commands
* [api-proxy](/flyctl/cmd/fly_machine_api-proxy) - Establish a proxy to the Machine API through a Wireguard tunnel for local connections
* [clone](/flyctl/cmd/fly_machine_clone) - Clone a Fly Machine
* [cordon](/flyctl/cmd/fly_machine_cordon) - Deactivate all services on a machine
* [create](/flyctl/cmd/fly_machine_create) - Create, but don't start, a machine
* [destroy](/flyctl/cmd/fly_machine_destroy) - Destroy Fly machines
* [exec](/flyctl/cmd/fly_machine_exec) - Execute a command on a machine
* [kill](/flyctl/cmd/fly_machine_kill) - Kill (SIGKILL) a Fly machine
* [leases](/flyctl/cmd/fly_machine_leases) - Manage machine leases
* [list](/flyctl/cmd/fly_machine_list) - List Fly machines
* [place](/flyctl/cmd/fly_machine_place) - Simulate Machine placements
* [restart](/flyctl/cmd/fly_machine_restart) - Restart one or more Fly machines
* [run](/flyctl/cmd/fly_machine_run) - Run a machine
* [start](/flyctl/cmd/fly_machine_start) - Start one or more Fly machines
* [status](/flyctl/cmd/fly_machine_status) - Show current status of a running machine
* [stop](/flyctl/cmd/fly_machine_stop) - Stop one or more Fly machines
* [suspend](/flyctl/cmd/fly_machine_suspend) - Suspend one or more Fly machines
* [uncordon](/flyctl/cmd/fly_machine_uncordon) - Reactivate all services on a machine
* [update](/flyctl/cmd/fly_machine_update) - Update a machine
* [wait](/flyctl/cmd/fly_machine_wait) - Wait for a machine to reach a state
## Options
```
-h, --help help for machine
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly](/flyctl/cmd/fly_help) - The Fly.io command line interface
# fly machine api-proxy
Source: https://docs.fly.io/flyctl/cmd/fly_machine_api-proxy
Establish a proxy to the Machine API through a Wireguard tunnel for local connections
## Usage
```
fly machine api-proxy [flags]
```
## Options
```
-h, --help help for api-proxy
-o, --org string The target Fly.io organization
-q, --quiet Don't print progress indicators for WireGuard
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine clone
Source: https://docs.fly.io/flyctl/cmd/fly_machine_clone
Clone a Fly Machine. The new Machine will be a copy of the specified Machine. If the original Machine has a volume, then a new empty volume will be created and attached to the new Machine.
## Usage
```
fly machine clone [machine_id] [flags]
```
## Options
```
-a, --app string Application name
--attach-volume string Existing volume to attach to the new Machine in the form of [:/path/inside/machine]
--clear-auto-destroy Disable auto destroy setting on the new Machine
--clear-cmd Set empty CMD on the new Machine so it uses default CMD for the image
-c, --config string Path to application configuration file
--detach Return immediately instead of monitoring deployment progress
--from-snapshot string Clone attached volumes and restore from snapshot, use 'last' for most recent snapshot. The default is an empty volume.
-h, --help help for clone
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
--name string Optional name for the new Machine
--override-cmd string Set CMD on the new Machine to this value
-r, --region string The target region (see 'flyctl platform regions')
--standby-for strings Comma separated list of Machine IDs to watch for. You can use '--standby-for=source' to create a standby for the cloned Machine.
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
--volume-requires-unique-zone Require volume to be placed in separate hardware zone from existing volumes. Default true. (default true)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine cordon
Source: https://docs.fly.io/flyctl/cmd/fly_machine_cordon
Deactivate all services on a machine
## Usage
```
fly machine cordon [...] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for cordon
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine create
Source: https://docs.fly.io/flyctl/cmd/fly_machine_create
Create, but don't start, a machine
## Usage
```
fly machine create [command] [flags]
```
## Options
```
-a, --app string Application name
--autostart Automatically start a stopped Machine when a network request is received (default true)
--autostop string[="stop"] Automatically stop a Machine when there are no network requests for it. Options include 'off', 'stop', and 'suspend'. (default "off")
--build-context-warn-size string Warn when the Docker build context is larger than this. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb). Set to 0 to disable. Also set with FLY_BUILD_CONTEXT_WARN_SIZE.
--build-depot Build your image with depot.dev
--build-nixpacks Build your image with nixpacks
--cachedrive-size string Cache drive size in MB. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 10gb). Set to 0 to disable.
-c, --config string Path to application configuration file
--detach Return immediately instead of monitoring deployment progress
--dockerfile string The path to a Dockerfile. Defaults to the Dockerfile in the working directory.
--entrypoint string The command to override the Docker ENTRYPOINT.
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--file-literal stringArray Set of literals to write to the Machine, in the form of /path/inside/machine=VALUE pairs, where VALUE is the base64-encoded raw content. Can be specified multiple times.
--file-local stringArray Set of files to write to the Machine, in the form of /path/inside/machine= pairs. Can be specified multiple times.
--file-secret stringArray Set of secrets to write to the Machine, in the form of /path/inside/machine=SECRET pairs, where SECRET is the name of the secret. The content of the secret must be base64 encoded. Can be specified multiple times.
-h, --help help for create
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
--id string Machine ID, if previously known
--kernel-arg stringArray A list of kernel arguments to provide to the init. Can be specified multiple times.
--machine-config string Read Machine config from a JSON file or string (use --config for app configuration)
-m, --metadata stringArray Metadata in the form of NAME=VALUE pairs. Can be specified multiple times.
-n, --name string Machine name. Will be generated if omitted.
--org string The organization that will own the app
-p, --port strings The external ports and handlers for services, in the format: port[:machinePort][/protocol[:handler[:handler...]]])
For example: --port 80/tcp --port 443:80/tcp:http:tls --port 5432/tcp:pg_tls
To remove a port mapping use '-' as handler. For example: --port 80/tcp:-
-r, --region string The target region (see 'flyctl platform regions')
--restart string Set the restart policy for a Machine. Options include 'no', 'always', and 'on-failure'.
Default is 'on-failure' for Machines created by 'fly deploy' and Machines with a schedule. Default is 'always' for Machines created by 'fly m run'.
--rm Automatically remove the Machine when it exits. Sets the restart-policy to 'never' if not otherwise specified.
--rootfs-persist string Whether to persist the root filesystem across restarts. Options include 'never', 'always', and 'restart'.
--rootfs-size string Root filesystem size in GB. Accepts a plain number (in GB) or a human-readable size (e.g. 2gb, 5gb). Uses an overlayfs to allow the root filesystem to exceed its default size. Set to 0 to unset.
--schedule string Schedule a Machine run at hourly, daily, weekly and monthly intervals
--skip-dns-registration Do not register the machine's 6PN IP with the internal DNS system
--standby-for strings For Machines without services, a comma separated list of Machine IDs to act as standby for.
--swap-size string Swap size in MB. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb).
--use-zstd Enable zstd compression for the image
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
-v, --volume strings Volume to mount, in the form of :/path/inside/machine[:]
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine destroy
Source: https://docs.fly.io/flyctl/cmd/fly_machine_destroy
Destroy one or more Fly machines.
This command requires a machine to be in a stopped or suspended state unless the force flag is used.
## Usage
```
fly machine destroy [flags] ID ID ...
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-f, --force force kill machine regardless of current state
-h, --help help for destroy
-i, --image string remove all machines with the specified image hash
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine exec
Source: https://docs.fly.io/flyctl/cmd/fly_machine_exec
Execute a command on a machine
## Usage
```
fly machine exec [machine-id] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
--container string Container to run the command in
-h, --help help for exec
-j, --json JSON output
--no-container Run the command on the machine itself rather than in one of its containers
--timeout int Timeout in seconds
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine kill
Source: https://docs.fly.io/flyctl/cmd/fly_machine_kill
Kill (SIGKILL) a Fly machine
## Usage
```
fly machine kill [id] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for kill
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine leases
Source: https://docs.fly.io/flyctl/cmd/fly_machine_leases
Manage machine leases
## Usage
```
fly machine leases [command] [flags]
```
## Available Commands
* [clear](/flyctl/cmd/fly_machine_leases_clear) - Clear machine leases
* [view](/flyctl/cmd/fly_machine_leases_view) - View machine leases
## Options
```
-h, --help help for leases
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine leases clear
Source: https://docs.fly.io/flyctl/cmd/fly_machine_leases_clear
Clear machine leases
## Usage
```
fly machine leases clear [machine-id] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for clear
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine leases](/flyctl/cmd/fly_machine_leases) - Manage machine leases
# fly machine leases view
Source: https://docs.fly.io/flyctl/cmd/fly_machine_leases_view
View machine leases
## Usage
```
fly machine leases view [machine-id] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for view
-j, --json JSON output
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine leases](/flyctl/cmd/fly_machine_leases) - Manage machine leases
# fly machine list
Source: https://docs.fly.io/flyctl/cmd/fly_machine_list
List Fly machines
## Usage
```
fly machine list [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for list
-j, --json JSON output
-q, --quiet Only list machine ids
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine place
Source: https://docs.fly.io/flyctl/cmd/fly_machine_place
Simulate a batch of Machine placements across multiple regions
## Usage
```
fly machine place [flags]
```
## Options
```
-c, --config string Path to application configuration file
--count int number of machines to place
-h, --help help for place
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
-j, --json JSON output
-o, --org string The target Fly.io organization
--region string comma-delimited list of regions to place machines
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
--volume-name string name of the volume to place machines
--volume-size int size of the desired volume to place machines
--weights strings comma-delimited list of key=value weights to adjust placement preferences. e.g., 'region=5,spread=10'
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine restart
Source: https://docs.fly.io/flyctl/cmd/fly_machine_restart
Restart one or more Fly machines
## Usage
```
fly machine restart [...] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
--force Force stop the machine(s)
-h, --help help for restart
-s, --signal string Signal to stop the machine with (default: SIGINT)
--skip-health-checks Restarts app without waiting for health checks.
--time int Seconds to wait before killing the machine
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine run
Source: https://docs.fly.io/flyctl/cmd/fly_machine_run
Run a machine
## Usage
```
fly machine run [command] [flags]
```
## Options
```
-a, --app string Application name
--autostart Automatically start a stopped Machine when a network request is received (default true)
--autostop string[="stop"] Automatically stop a Machine when there are no network requests for it. Options include 'off', 'stop', and 'suspend'. (default "off")
--build-context-warn-size string Warn when the Docker build context is larger than this. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb). Set to 0 to disable. Also set with FLY_BUILD_CONTEXT_WARN_SIZE.
--build-depot Build your image with depot.dev
--build-nixpacks Build your image with nixpacks
--cachedrive-size string Cache drive size in MB. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 10gb). Set to 0 to disable.
--command string Used with --shell. The command to run, if we're shelling into the Machine now (in case you don't have bash). (default "/bin/bash")
-c, --config string Path to application configuration file
--container string Container to update with the new image, files, etc; defaults to "app" or the first container in the config.
--detach Return immediately instead of monitoring deployment progress
--dockerfile string The path to a Dockerfile. Defaults to the Dockerfile in the working directory.
--entrypoint string The command to override the Docker ENTRYPOINT.
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
--file-literal stringArray Set of literals to write to the Machine, in the form of /path/inside/machine=VALUE pairs, where VALUE is the base64-encoded raw content. Can be specified multiple times.
--file-local stringArray Set of files to write to the Machine, in the form of /path/inside/machine= pairs. Can be specified multiple times.
--file-secret stringArray Set of secrets to write to the Machine, in the form of /path/inside/machine=SECRET pairs, where SECRET is the name of the secret. The content of the secret must be base64 encoded. Can be specified multiple times.
-h, --help help for run
--host-dedication-id string The dedication id of the reserved hosts for your organization (if any)
--id string Machine ID, if previously known
--kernel-arg stringArray A list of kernel arguments to provide to the init. Can be specified multiple times.
--machine-config string Read Machine config from a JSON file or string (use --config for app configuration)
-m, --metadata stringArray Metadata in the form of NAME=VALUE pairs. Can be specified multiple times.
-n, --name string Machine name. Will be generated if omitted.
--org string The organization that will own the app
-p, --port strings The external ports and handlers for services, in the format: port[:machinePort][/protocol[:handler[:handler...]]])
For example: --port 80/tcp --port 443:80/tcp:http:tls --port 5432/tcp:pg_tls
To remove a port mapping use '-' as handler. For example: --port 80/tcp:-
-r, --region string The target region (see 'flyctl platform regions')
--restart string Set the restart policy for a Machine. Options include 'no', 'always', and 'on-failure'.
Default is 'on-failure' for Machines created by 'fly deploy' and Machines with a schedule. Default is 'always' for Machines created by 'fly m run'.
--rm Automatically remove the Machine when it exits. Sets the restart-policy to 'never' if not otherwise specified.
--rootfs-persist string Whether to persist the root filesystem across restarts. Options include 'never', 'always', and 'restart'.
--rootfs-size string Root filesystem size in GB. Accepts a plain number (in GB) or a human-readable size (e.g. 2gb, 5gb). Uses an overlayfs to allow the root filesystem to exceed its default size. Set to 0 to unset.
--schedule string Schedule a Machine run at hourly, daily, weekly and monthly intervals
--shell Open a shell on the Machine once created (implies --it --rm). If no app is specified, an app for interactive shells is created or reused. The Machine is destroyed when the shell exits; the app is retained for future shells. See also --command and --user.
--skip-dns-registration Do not register the machine's 6PN IP with the internal DNS system
--standby-for strings For Machines without services, a comma separated list of Machine IDs to act as standby for.
--swap-size string Swap size in MB. Accepts a plain number (in MB) or a human-readable size (e.g. 512mb, 1gb).
--use-zstd Enable zstd compression for the image
--user string Used with --shell. The username, if we're shelling into the Machine now. (default "root")
--vm-cpu-kind string The kind of CPU to use ('shared' or 'performance') (also --vm-cpukind)
--vm-cpus int Number of CPUs (also --cpus)
--vm-memory string Memory (in megabytes) to attribute to the VM (also --memory)
--vm-size string The VM size to set machines to. See "fly platform vm-sizes" for valid values
-v, --volume strings Volume to mount, in the form of :/path/inside/machine[:]
--wg Determines whether communication with remote builders are conducted over wireguard or plain internet(https) (default true)
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine start
Source: https://docs.fly.io/flyctl/cmd/fly_machine_start
Start one or more Fly machines
## Usage
```
fly machine start [...] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for start
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine status
Source: https://docs.fly.io/flyctl/cmd/fly_machine_status
Show current status of a running machine
## Usage
```
fly machine status [id] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-d, --display-config Display the machine config as JSON
-h, --help help for status
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine stop
Source: https://docs.fly.io/flyctl/cmd/fly_machine_stop
Stop one or more Fly machines
## Usage
```
fly machine stop [...] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for stop
-s, --signal string Signal to stop the machine with (default: SIGINT)
--timeout int Seconds to wait before sending SIGKILL to the machine
-w, --wait-timeout duration Time duration to wait for individual machines to transition states and become stopped.
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine suspend
Source: https://docs.fly.io/flyctl/cmd/fly_machine_suspend
Suspend one or more Fly machines
## Usage
```
fly machine suspend [...] [flags]
```
## Options
```
-a, --app string Application name
-c, --config string Path to application configuration file
-h, --help help for suspend
-w, --wait-timeout duration Duration to wait for individual Machines to be suspended.
```
## Global Options
```
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
```
## See Also
* [fly machine](/flyctl/cmd/fly_machine) - Manage Fly Machines.
# fly machine uncordon
Source: https://docs.fly.io/flyctl/cmd/fly_machine_uncordon
Reactivate all services on a machine
## Usage
```
fly machine uncordon [