Using volumes with FKS

Fly Kubernetes is in closed beta and not recommended for critical production usage. To report issues or provide feedback, email us at beta@fly.io.

Fly Kubernetes supports persistent volumes and generic ephemeral volumes. There are general considerations to keep in mind when using volumes in Fly Kubernetes:

Volumes are built on top of Fly Volumes. They are local NVME drives and require your Pod to be deployed in the same region as the volume.
Only one persistent or ephemeral volume can be mounted at a time.
The maximum storage size available is 500GB.

Persistent Volumes

PersistentVolumes are managed through PersistentVolumeClaims (PVCs). PersistentVolumes (PVs) are tied to the lifecycle of a PersistentVolumeClaim. By default, creating a PVC creates an underlying PV and deleting a PVC deletes the PV bound to it. Below is an example of configuring a PersistentVolume to use with a Pod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  selector:
    matchLabels:
      region: iad
  storageClassName: flyio-volume
  accessModes:
    - ReadWriteOncePod
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      volumeMounts:
        - mountPath: "/var/www/html"
          name: mypd
      resources:
        limits:
          memory: "512Mi"
          cpu: "1"
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim

In our configuration:

The storageClassName must be set to either flyio-volume or flyio-volume-retain. You can view the available storage classes with kubectl get sc. The default StorageClass is flyio-volume.
selector is optional. It serves a dual purpose. First, it ensures that a PVC is bound to a PV in the region specified in the selector. Second, if a PVC requires dynamic provisioning of the underlying Fly Volume, it is provisioned in the specified region. If not set, the volume is created in the region of the cluster.
accessModes only supports ReadWriteOncePod.

After applying the configuration, you can view the PVC and PV generated by the definition using kubectl.

> kubectl get pvc
NAME          STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myclaim       Bound    pvc-r1plo75g59d25l0r   5Gi        RWOP           flyio-volume   9s

> kubectl get pv
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-r1plo75g59d25l0r   5Gi        RWOP           Delete           Bound    default/myclaim       flyio-volume            55s

Reclaim policy

Fly Kubernetes supports both the Delete and Retain policy for PersistentVolumes.

StorageClass flyio-volume has a reclaim policy set to Delete
StorageClass flyio-volume-retain has a reclaim policy set to Retain

By default, flyio-volume is used. When using the flyio-volume-retain StorageClass, you are responsible for deleting the PersistentVolume object and its underlying Fly Volume. The details of the underlying storage are found in the annotations of the PersistentVolume. Using the PV from above

> kubectl describe pv pvc-r1plo75g59d25l0r
Name:            pvc-r1plo75g59d25l0r
Labels:          region=iad
Annotations:     fly.io/app: fks-default-vz5dlqz7v4ylrnpq
                 fly.io/region: iad
                 pv.kubernetes.io/provisioned-by: volume.csi.fly.io
                 volume.fly.io/id: vol_r1plo75g59d25l0r
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    flyio-volume
Status:          Bound
Claim:           default/myclaim
Reclaim Policy:  Delete
Access Modes:    RWOP
VolumeMode:      Filesystem
Capacity:        5Gi
Node Affinity:   <none>
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            volume.csi.fly.io
    FSType:            ext4
    VolumeHandle:      vol_r1plo75g59d25l0r
    ReadOnly:          false
    VolumeAttributes:  <none>
Events:                <none>

The annotation fly.io/app specifies which Fly app the volumes belongs to. The annotation volume.fly.io/id gives you the ID of the volume. You can delete the Fly Volume using flyctl

fly vol rm pvc-r1plo75g59d25l0r -a fks-default-vz5dlqz7v4ylrnpq

Generic ephemeral volumes

Fly Kubernetes supports ephemeral volumes through generic ephemeral volumes. It uses the same underlying PVC and PV machinery. Kubernetes handles creating the PVC and its corresponding PV and deleting both objects when the Pod is deleted. Below is an example of creating a Pod with a generic ephemeral volume:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
      image: nginx:latest
      volumeMounts:
        - mountPath: "/var/www/html"
          name: scratch-volume
      resources:
        limits:
          memory: "512Mi"
          cpu: "1"
  volumes:
    - name: scratch-volume
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes:
              - ReadWriteOncePod
            storageClassName: "flyio-volume"
            resources:
              requests:
                storage: 5Gi

For generic ephemeral volumes to work as expected, storageClassName must be set to flyio-volume.

​Persistent Volumes

​Reclaim policy

​Generic ephemeral volumes

Persistent Volumes

Reclaim policy

Generic ephemeral volumes