Kubernetes — Storage Resources

Kubernetes

Containers

Storage

Published

December 4, 2024

Modified

April 29, 2025

Overview

Kubernetes abstractions for storage — Storage resource types

Persistent Volumes (PVs)
- …represents a volume of storage in a storage system
- …information required by applications to consume a volume
- …different back-ends define what a “volume” represents
Persistent Volume Claims (PVC)
- …request of an application for a volume
- …the PVC is bound to a suitable PV on a storage back-end
- …represents the connection from storage to the application
Storage Classes (SC)
- …dynamically request some volume of storage
- …request a volume claim against a storage class
- …a corresponding PV is created on the storage back-end

Dynamic provisioning (otherwise an administrator needs to create PVs manually):

Building Blocks: Raw Block PVs in Rook-Ceph, Rohan Gupta, Jose Rivera, DevConf.CZ 2020

Architecture

Volumes¹ — Share data with containers in a pod

…many types of volumes …pods can use multiple types simultaneously
…mount at the specified paths within the container
Use-cases…
- …configuration files with ConfigMap & Secret
- …temporary scratch space
- …read-only access to data in a different container image
- …sharing a file-system between two container within a pod
- …sharing a file-system between two different pods
- …persistent storage (not bound to the pod life-cycle)
Ephemeral volumes …have the lifetime of a pod
Persistent volumes …exist beyond the lifetime of a pod

Persistent Volumes

Persistent Volumes² (PV) …abstract representation of physical or networked storage

Why use persistent volumes?
- Data persistence …vital to stateful applications and databases
- Abstraction from the back-end storage…
  - …allows to manage storage resources independently
  - …enables admins control utilization and optimization of storage hardware
- Access control …ensure data integrity and security
There are two ways PVs may be provisioned: statically or dynamically

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-storage
spec:
  capacity:
    storage: 500Mi 
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  hostPath:
    path: /srv

kubectl describe pv $pv_name

Lifecycle & Phases

Lifecycle …several stages

…create …static or dynamic PV declaration
…bind …occurs when a PVC requests storage
…match …Kubernetes links PVC with suitable PV
…access …pod granted access to the bound PV
…using …pod(s) read and write data
…reclaim …when PVC deleted (determined by policy)

PersistentVolume will be in one of the following phases

Available …free resource that is not yet bound to a claim
Bound …volume is bound to a claim
Released …claim deleted …associated storage resource not reclaimed (yet)
Failed …volume has failed its (automated) reclamation

Parameters

capacity.storage — Valid suffix Mi,Gi,Ti
volumeModes — How storage will be accessed
- Filesystem (default) — Mounted to a directory
  - …if empty block device as back-end
  - …file-system created automatically
- Block — Use a volume as a raw block device
  - …must match on both PV and PVC
  - …present storage without a formatted file-system
  - …useful for applications like databases
  - …some storage back-ends may provide constant I/O performance & lower latency

persistentVolumeReclaimPolicy³ — What to do with a volume after released

Retain — Manual reclamation of the resource…
- …remains in the Released state
- …can be reclaimed using the same storage definition
Delete — Resource will be deleted automatically
Recycle — Volume scrubbed …storage available to new claims

Persistent Volume Claims

Used to specific requirements to the storage resource…

…independent of the internals of the storage provider
Ensures a volume ‘claim’ to be portable across numerous back-ends
Isolates storage-related concerns from the workload

apiVersion: v1
kind: PersistentVolumeClaim
#…
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi

storageClassName requests a particular storage class
accessModes supports the following access modes
- ReadWriteOnce — read-write by a single pod
- ReadOnlyMany — read-only by multiple pods
- ReadWriteMany — read-write by multiple pods
resources.requests.storage …suffix Mi,Gi,Ti

PVC stuck in terminating state:

# get the volume name (prefixed with pvc-*)
kubectl get pvc $name
# check if the volume is still attached to a node
kubectl get volumeattachment | grep pvc-$volume_id

# force the remove of the mount
kubectl patch pvc -p '{"metadata":{"finalizers":null}}' $name

# remove a PVC if stuck ...requires manual cleanup on the storage provider
kubectl delete pvc --grace-period=0 --force $name

Storage Classes

# list all storage classes
kubectl get sc
kubectl get storageclass
kubectl describe storageclass $name

# list installed CSI drivers
kubectl get csidriver

Storage Class⁴ — Define the “classes” of storage

…includes storage provisioners & reclaim policies
Facilitates dynamic provisioning of storage…
- …for on-demand deployment of stateful applications
- …by automating creation of persistent volumes
Storage abstraction for underlying storage providers
- …automate storage management & hide storage provider details
- …enables portable and provider-agnostic storage claims
- …reduces manual intervention in storage allocation
Implement storage policies…
- …automate storage reclaim policies (when storage is released)
- …define multiple storage tiers (SSD, HDD, etc)

CSI (Container Storage Interface) — Standard for Storage provider integration

…enable storage vendors to create custom storage plugins
…translating storage requests into platform-specific storage resources

Parameters that define specific settings for the storage provisioner:

# Storage provider to create the volume on…
#    …name of the CSI driver
#    …for example kubernetes.io/nfs
provisioner: csi-driver.example-vendor.example

# Lifecycle of a persistent volume (PV)
#    …after its associated persistent volume claim (PVC) is deleted
reclaimPolicy: Retain

Examples

Commonly used volume types…

emptyDir — Simple empty directory …for transient data
hostPath — Directory from the work node
nfs — Mount of an NFS share
configMap — Non-confidential configuration …key-value pairs
secret — Passwords, Oath tokens, SSH keys, etc.
downwardAPI — Access pod metadata & cluster context
persistantVolumeClaim — Use pre- & dynamic provisioned persistent storage

Mount other network storage for example cephfs, rbd, etc.

EmptyDir

An emptyDir volume is created when a pod is assigned to a node…

…on storage local to the worker node
…deleted when the pod is removed …safe across container crashes
Why use an emptydir?
- User for temporary scratch space
- Share files between container running in the same pod

apiVersion: v1
kind: Pod
metadata:
  name: test-emptydir
spec:
  volumes:
    - name: opt
      emptyDir: {}
  containers:
    - name: container-one
      volumeMounts:
        - name: opt
          mountPath: /opt
      image: alpine
      command: ["/bin/sh"]
      args: ["-c", "sleep 10000"]
    - name: container-two
      volumeMounts:
        - name: opt
          mountPath: /opt
      image: alpine
      command: ["/bin/sh"]
      args: ["-c", "sleep 10000"]

# write a file into the mounted volume from the first container
kubectl exec test-emptydir -c container-one -- cp /etc/hostname /opt

# read the file from the second container
kubectl exec test-emptydir -c container-two -- cat /opt/hostname

Use an tmpfs file-system in memory instead of local storage:

volumes:
  - name: html
    emptyDir:
      medium: Memory

Host Path

kind: PersistentVolume
metadata:
  name: project-storage
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/project/        # absolute path on the host node

Mount the host node’s file-system into a pod

Capacity is determined by the available space on the host node
No Quotas or limits imposed …responsibility outside Kubernetes
Data persistent as long as host node is operational
Typical use-cases:
- Testing & development
- Access logs & configuration files on the host
- Share data between pods on the same host

ConfigMap

# create a ConfigMap directly from literal values
kubectl create configmap project-config \
        --from-literal=key1=value1 --from-literal=key2=value2

# create a ConfigMap from a file
kubectl create configmap project-config \
        --from-file=project-config.properties

kubectl create configmap project-config \
        --from-file=/path/to/project-config/

Manage configuration settings for applications

For non-confidential configuration only (cf. Kubernetes secrets)
Values can be strings, numbers, JSON, YAML
Version control configuration separate form application code & container images

Example of a configuration file nginx.conf used by a pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user nginx;
    worker_processes  1;
    events {
      worker_connections  10240;
    }
    http {
      server {
          listen       80;
          server_name  localhost;
          location / {
            root   /srv/htdocs;
            index  index.html index.htm;
        }
      }
    }

---

apiVersion: v1
kind: Pod
metadata:
  name: website
spec:
  containers:
  - name: web-server
    image: nginx:alpine
    ports:
      - containerPort: 80 
    volumeMounts:
      - name: html
        mountPath: /srv
        readOnly: true
      - name: nginx-conf
        mountPath: /etc/nginx/nginx.conf
        subPath: nginx.conf
        readOnly: true

Footnotes

Kubernetes Volumes
https://kubernetes.io/docs/concepts/storage/volumes/↩︎
Persistent Volumes, Kubernetes Documentation
https://kubernetes.io/docs/concepts/storage/persistent-volumes/↩︎
Reclaiming, Kubernetes Documentation
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming ↩︎
Storage Classes, Kubernetes Documentation
https://kubernetes.io/docs/concepts/storage/storage-classes ↩︎

--- title: 'Kubernetes — Storage Resources' categories: - Kubernetes - Containers - Storage date: 2024/12/04 date-modified: 2025/04/29 toc-expand: 3 --- ## Overview Kubernetes abstractions for storage — Storage resource types - **Persistent Volumes** (PVs) - …represents a volume of storage in a storage system - […information required by applications to consume a volume]{.text-primary} - …different back-ends define what a "volume" represents - **Persistent Volume Claims** (PVC) - …[request of an application for a volume]{.text-primary} - …the PVC is bound to a suitable PV on a storage back-end - …represents the connection from storage to the application - **Storage Classes** (SC) - …dynamically request some volume of storage - …request a volume claim against a storage class - …a corresponding PV is created on the storage back-end Dynamic provisioning (otherwise an administrator needs to create PVs manually): ![Building Blocks: Raw Block PVs in Rook-Ceph, Rohan Gupta, Jose Rivera, DevConf.CZ 2020](dynamic-provisioning.png){.lightbox} ## Architecture Volumes[^VlEAi] — Share data with containers in a pod [^VlEAi]: Kubernetes Volumes <https://kubernetes.io/docs/concepts/storage/volumes/> - …many types of volumes …pods can use multiple types simultaneously - …mount at the specified paths within the container - Use-cases… - …configuration files with `ConfigMap` & `Secret` - …temporary scratch space - …read-only access to data in a different container image - …sharing a file-system between two container within a pod - …sharing a file-system between two different pods - …persistent storage (not bound to the pod life-cycle) - **Ephemeral** volumes …have the lifetime of a pod - **Persistent** volumes …exist beyond the lifetime of a pod ## Persistent Volumes Persistent Volumes[^HNdGM] (PV) …abstract representation of physical or networked storage [^HNdGM]: Persistent Volumes, Kubernetes Documentation <https://kubernetes.io/docs/concepts/storage/persistent-volumes/> * _Why use persistent volumes?_ - **Data persistence** …vital to stateful applications and databases - **Abstraction** from the back-end storage… - …allows to manage storage resources independently - …enables admins control utilization and optimization of storage hardware - **Access control** …ensure data integrity and security * There are two ways PVs may be provisioned: **statically or dynamically** ```yml apiVersion: v1 kind: PersistentVolume metadata: name: example-storage spec: capacity: storage: 500Mi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage hostPath: path: /srv ``` ```bash kubectl describe pv $pv_name ``` ### Lifecycle & Phases **Lifecycle** …several stages - …create …static or dynamic PV declaration - …bind …occurs when a PVC requests storage - …match …Kubernetes links PVC with suitable PV - …access …pod granted access to the bound PV - …using …pod(s) read and write data - …reclaim …when PVC deleted (determined by policy) PersistentVolume will be in one of the following phases - `Available` …free resource that is not yet bound to a claim - `Bound` …volume is bound to a claim - `Released` …claim deleted …associated storage resource not reclaimed (yet) - `Failed` …volume has failed its (automated) reclamation ### Parameters - **`capacity.storage`** — Valid suffix `Mi`,`Gi`,`Ti` - **`volumeModes`** — How storage will be accessed - `Filesystem` (default) — Mounted to a directory - …if empty block device as back-end - …file-system created automatically - `Block` — Use a volume as a raw block device - …must match on both PV and PVC - …present storage without a formatted file-system - …useful for applications like databases - …some storage back-ends may provide constant I/O performance & lower latency **`persistentVolumeReclaimPolicy`**[^DvC94] — What to do with a volume after released [^DvC94]: Reclaiming, Kubernetes Documentation <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming> * `Retain` — Manual reclamation of the resource… - …remains in the `Released` state - …can be reclaimed using the same storage definition * `Delete` — Resource will be deleted automatically * `Recycle` — Volume scrubbed …storage available to new claims ## Persistent Volume Claims Used to specific requirements to the storage resource… - …independent of the internals of the storage provider - Ensures a volume ‘claim’ to be portable across numerous back-ends - Isolates storage-related concerns from the workload ```yml apiVersion: v1 kind: PersistentVolumeClaim #… spec: storageClassName: local-storage accessModes: - ReadWriteOnce resources: requests: storage: 200Mi ``` - **`storageClassName`** requests a particular storage class - **`accessModes`** supports the following access modes - `ReadWriteOnce` — read-write by a single pod - `ReadOnlyMany` — read-only by multiple pods - `ReadWriteMany` — read-write by multiple pods - **`resources.requests.storage`** …suffix `Mi`,`Gi`,`Ti` PVC stuck in terminating state: ```bash # get the volume name (prefixed with pvc-*) kubectl get pvc $name # check if the volume is still attached to a node kubectl get volumeattachment | grep pvc-$volume_id # force the remove of the mount kubectl patch pvc -p '{"metadata":{"finalizers":null}}' $name # remove a PVC if stuck ...requires manual cleanup on the storage provider kubectl delete pvc --grace-period=0 --force $name ``` ## Storage Classes ```bash # list all storage classes kubectl get sc kubectl get storageclass kubectl describe storageclass $name # list installed CSI drivers kubectl get csidriver ``` Storage Class[^8ikgf] — Define the **“classes” of storage** [^8ikgf]: Storage Classes, Kubernetes Documentation <https://kubernetes.io/docs/concepts/storage/storage-classes> * …includes storage provisioners & reclaim policies * [Facilitates dynamic provisioning of storage…]{.text-primary} - …for on-demand deployment of stateful applications - …by automating creation of persistent volumes * Storage abstraction for underlying storage providers - …automate storage management & hide storage provider details - […enables portable and provider-agnostic storage claims]{.text-primary} - …reduces manual intervention in storage allocation * Implement storage policies… - …automate storage reclaim policies (when storage is released) - …define multiple storage tiers (SSD, HDD, etc) **CSI (Container Storage Interface)** — Standard for Storage provider integration * …enable storage vendors to create custom storage plugins * …translating storage requests into platform-specific storage resources Parameters that define specific settings for the storage provisioner: ```yaml # Storage provider to create the volume on… # …name of the CSI driver # …for example kubernetes.io/nfs provisioner: csi-driver.example-vendor.example # Lifecycle of a persistent volume (PV) # …after its associated persistent volume claim (PVC) is deleted reclaimPolicy: Retain ``` ## Examples Commonly used volume types… - `emptyDir` — Simple empty directory …for transient data - `hostPath` — Directory from the work node - `nfs` — Mount of an NFS share - `configMap` — Non-confidential configuration …key-value pairs - `secret` — Passwords, Oath tokens, SSH keys, etc. - `downwardAPI` — Access pod metadata & cluster context - `persistantVolumeClaim` — Use pre- & dynamic provisioned persistent storage Mount other network storage for example `cephfs`, `rbd`, etc. ### EmptyDir An `emptyDir` volume is created when a pod is assigned to a node… - …on storage local to the worker node - **…deleted when the pod is removed …safe across container crashes** - Why use an `emptydir`? - User for temporary scratch space - Share files between container running in the same pod ```yaml apiVersion: v1 kind: Pod metadata: name: test-emptydir spec: volumes: - name: opt emptyDir: {} containers: - name: container-one volumeMounts: - name: opt mountPath: /opt image: alpine command: ["/bin/sh"] args: ["-c", "sleep 10000"] - name: container-two volumeMounts: - name: opt mountPath: /opt image: alpine command: ["/bin/sh"] args: ["-c", "sleep 10000"] ``` ```bash # write a file into the mounted volume from the first container kubectl exec test-emptydir -c container-one -- cp /etc/hostname /opt # read the file from the second container kubectl exec test-emptydir -c container-two -- cat /opt/hostname ``` Use an tmpfs file-system in memory instead of local storage: ```yaml volumes: - name: html emptyDir: medium: Memory ``` ### Host Path ```yaml kind: PersistentVolume metadata: name: project-storage spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi hostPath: path: /data/project/ # absolute path on the host node ``` **Mount the host node's file-system into a pod** - Capacity is determined by the available space on the host node - No Quotas or limits imposed …responsibility outside Kubernetes - Data persistent as long as host node is operational - Typical use-cases: - Testing & development - Access logs & configuration files on the host - Share data between pods on the same host ### ConfigMap ```bash # create a ConfigMap directly from literal values kubectl create configmap project-config \ --from-literal=key1=value1 --from-literal=key2=value2 # create a ConfigMap from a file kubectl create configmap project-config \ --from-file=project-config.properties kubectl create configmap project-config \ --from-file=/path/to/project-config/ ``` **Manage configuration settings for applications** - For non-confidential configuration only (cf. Kubernetes secrets) - Values can be strings, numbers, JSON, YAML - Version control configuration separate form application code & container images Example of a configuration file `nginx.conf` used by a pod: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-conf data: nginx.conf: | user nginx; worker_processes 1; events { worker_connections 10240; } http { server { listen 80; server_name localhost; location / { root /srv/htdocs; index index.html index.htm; } } } --- apiVersion: v1 kind: Pod metadata: name: website spec: containers: - name: web-server image: nginx:alpine ports: - containerPort: 80 volumeMounts: - name: html mountPath: /srv readOnly: true - name: nginx-conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf readOnly: true ```