Running apps on UDP and TCP

You can run any containerized app on our platform, regardless of what protocols it talks over, including apps that rely on UDP in addition to TCP. This guide walks you through how to set up your UDP app on Fly.io. Before getting started, here’s the summary of important points about using UDP on Fly.io (that we’ll expand on below):

You need a dedicated IPv4 address. You can’t use a shared IPv4 address or an IPv6 address for UDP. We support public IPv6 and shared IPv4 for TCP.
The UDP side of your app needs to bind to the special fly-global-services address. But the TCP side of your app can’t; it should bind to 0.0.0.0.
The UDP side of your app needs to bind to the same port that is used externally. Fly will not rewrite the port; Fly only rewrites the IP address for UDP packets.
Don’t assume a particular MTU for UDP. We use some of each packet in transit, and your protocol should discover the path MTU on its own.
There’s no per-flow load balancing for UDP.
QUIC isn’t supported through UDP services. You can technically get it working, but with a lot of caveats.

You need a dedicated IPv4 address

You’ll need a dedicated IPv4 address for your app to accept UDP packets. We don’t support UDP over public IPv6. Every Fly.io app gets a public IPv6 address and a shared IPv4 address that will work for the TCP side of your app.

Note: UDP does work over Fly.io IPv6 private networking.

Your app needs to bind to the `fly-global-services` address

To receive UDP packets, your app needs to bind to the special fly-global-services address (that’s it; that’s the address). Just as importantly: it needs to reply to messages using that address. You usually need to explicitly bind your UDP service to fly-global-services. Sorry, but 0.0.0.0, *, and INADDR_ANY generally won’t do: Linux will use the wrong source address in replies if you use those. But: TCP can’t use fly-global-services You can’t bind your TCP service to fly-global-services; the TCP side of your app should probably bind to *.

Why the `fly-global-services` address?

For the most part, UDP “just works” on Fly.io, the way you’d expect it to. The thing about forwarding UDP that makes it different from HTTP is that UDP messages don’t have HTTP headers. When a standard proxy receives an HTTP message and sends it to its target, it stashes the original source address in the headers, so the target knows who it’s talking to. You can’t generally do that with UDP messages. But you need to know who you’re talking to in order to respond to a UDP message! We use ⚡️eBPF magic⚡️ to shuttle UDP packets across our forwarding fabric while transparently mapping source addresses. What that means for you is that you can generally just write a standard socket program that reads UDP messages and replies to them based on the source address of the packet.

If we didn’t do this fly-global-services thing, our UDP proxying logic would also try to proxy your app’s own DNS traffic; it can’t otherwise tell the difference between UDP packets you’re sending to respond to a user and UDP packets your app generates in the ordinary course of its business.

Your app needs to listen on the same port externally and internally

Fly will only rewrite the IP addresses for UDP packets. The destination port for inbound packets will not be rewritten. That means if your app receives UDP traffic on port 5000 externally, the app needs to bind to fly-global-services:5000 to receive that UDP traffic.

Don’t assume a fixed MTU

An invariant on almost every network is the maximum packet size, the MTU. Typically, the MTU is 1500 bytes. If you exceed 1500 bytes, your packets will fragment. You don’t want to fragment. Fly.io takes a bite out of your packets for two purposes:

We forward all traffic across a WireGuard mesh, and WireGuard eats 60 bytes of every packet.
The UDP proxy forwarding system we use grabs another dozen bytes to record the original addresses of the packets --- the source address we saw when your client’s packet hit our edge. You don’t see these bytes; they’re stripped off the packet before it’s delivered to your app. But they’re used in transit.

On top of that, the path between a client and your app can cross links with smaller MTUs, so the space you actually have varies. Don’t hardcode a packet size and expect it to fit. Instead, your protocol should do path MTU discovery based on packet loss: start small, probe larger sizes, and back off when probes go missing (RFC 8899 describes this for datagram protocols). Most QUIC servers and clients already do this. If you’re building something custom on UDP, you need to do it yourself.

There’s no per-flow load balancing

UDP services don’t do per-flow load balancing. At any given time, a single Machine receives all the packets from clients near a region, whether they belong to a new flow or an existing one. That Machine can change, and when it does, all existing clients switch to the new Machine at the same time. So packets from the same flow can reach different Machines, and your app shouldn’t rely on a flow staying on one Machine. This is different from TCP. If a Machine reaches its load limit and can’t accept new TCP connections, new connections go to other Machines and existing connections stay where they are. UDP traffic never gets spread out like this.

QUIC isn’t supported

QUIC isn’t explicitly supported through Fly.io UDP services right now. You can technically get it working, but with a lot of caveats: there’s no per-flow load balancing, the MTU advice above applies, and you should expect breakage. The same goes for HTTP/3, which runs over QUIC: serving it from a udp service on port 443 won’t work the way you might expect. Proper QUIC support at our edge is on the roadmap, but we don’t have a timeline for it yet.

Example UDP app

Let’s build a simple echo service. You can play the home version at this GitHub repository. We’ll listen on port 5000, UDP and TCP, and just bat back whatever clients send us. Use flyctl launch to create a fly.toml file. Use it to wire up the ports.

[[services]]
  internal_port = 5000
  protocol = "udp"

  [[services.ports]]
    port = "5000"

[[services]]
  internal_port = 5000
  protocol = "tcp"

  [[services.ports]]
    port = "5000"

We’re going to build this app in Go (don’t worry, it’s simple, you’ll be able to follow it even if you aren’t a gopher). It’s going to be vanilla socket code.

Let’s See Some Code

You light up UDP and TCP with standard socket code. Usually, the only fussy bit will be the fly-global-services bind for UDP. Here’s what that looks like in Go:

func main() {
  // in other programming languages, this might look like:
	//    s = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
	//    s.bind("fly-global-services", port)

	udp, err := net.ListenPacket("udp", fmt.Sprintf("fly-global-services:%d", port))
	if err != nil {
		log.Fatalf("can't listen on %d/udp: %s", port, err)
	}

	// in other programming languages, this might look like:
	//    s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
	//    s.bind("0.0.0.0", port)
	//    s.listen()

	tcp, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
	if err != nil {
		log.Fatalf("can't listen on %d/tcp: %s", port, err)
	}

	go handleTCP(tcp)

	handleUDP(udp)
}

The handlers for this app are textbook Go code; after the UDP bind, nothing else in your app will care about Fly.io. Here’s TCP:

func handleTCP(l net.Listener) {
	for {
		conn, err := l.Accept()
		if err != nil {
			if errors.Is(err, net.ErrClosed) {
				return
			}

			log.Printf("error accepting on %d/tcp: %s", port, err)
			continue
		}

		go handleConnection(conn)
	}
}

func handleConnection(c net.Conn) {
	defer c.Close()

	lines := bufio.NewReader(c)

	for {
		line, err := lines.ReadString('\n')
		if err != nil {
			return
		}

		c.Write([]byte(line))
	}
}

And here’s UDP:

func handleUDP(c net.PacketConn) {
	packet := make([]byte, 2000)

	for {
		n, addr, err := c.ReadFrom(packet)
		if err != nil {
			if errors.Is(err, net.ErrClosed) {
				return
			}

			log.Printf("error reading on %d/udp: %s", port, err)
			continue
		}

		c.WriteTo(packet[:n], addr)
	}
}

In a real Go app doing something like DNS, you might have a single thread reading messages off the UDP socket and passing them to a pool of workers over a buffered channel; you might not want to fork off a thread for every UDP message you receive. This is just an echo service, so we’ll dispense with that detail. This program, with a simple Dockerfile and the fly.toml configuration above, should just work on Fly.io. Boot it up and throw some UDP packets at it with netcat:

echo hello world | nc -w1 -u udp-echo-sample.fly.dev 5000

hello world

​You need a dedicated IPv4 address

​Your app needs to bind to the fly-global-services address

​Why the fly-global-services address?

​Your app needs to listen on the same port externally and internally

​Don’t assume a fixed MTU

​There’s no per-flow load balancing

​QUIC isn’t supported

​Example UDP app

​Let’s See Some Code

You need a dedicated IPv4 address

Your app needs to bind to the `fly-global-services` address

Why the `fly-global-services` address?

Your app needs to listen on the same port externally and internally

Don’t assume a fixed MTU

There’s no per-flow load balancing

QUIC isn’t supported

Example UDP app

Let’s See Some Code