Backup Kubernetes PV with Velero: The new approach

Kubernetes is so cool and trendy! However there is always a question that remains... How do you backup PV when you have statefull application? Velero is a usefull tool that is able to backup your kubernetes ressource and also PVs.

Until now you had to specify Velero annotation when you want to backup pod volumes. But with the new version of Velero (1.5 and above) you can now backup PV by default and annotate resources when you don't want to backup the PV. So we change the paradigm here. In this article we will show you how up upgrade velero to 1.5 and switch your pod volume backup approach in order to backup all volume by default.

To summarize there is 2 approach:

opt-in pod volume backup approach

This is the only approach that exist for version 1.4 and below. Velero, by default, uses this approach to discover pod volumes that need to be backed up using restic, where every pod containing a volume to be backed up using restic must be annotated with the volume’s name.

Instructions to back up using this approach are as follows

kubectl -n your_namespace annotate pod/your_pod_name backup.velero.io/backup-volumes=your_volume_name,your_volume_name2

opt-out volume backup approach

This approach is available since version 1.5. In this approach, Velero will back up all pod volumes using restic with the exception of:

Volumes mounting the default service account token, kubernetes secrets, and config maps

Hostpath volumes

to exclude restic backup of volume pvc1-vm, you would run:

kubectl -n your_namespace annotate pod/your_pod_name backup.velero.io/backup-volumes-excludes=pvc1-vm

Upgrade velero and restic

Here is how to upgrade velero and restic from 1.4 to 1.5. First you have to upgrade the velero CLI, to do so just install the new version.

Get the latest version of velero CLI here. At the time of this article the latest version is 1.5.3.

untar and copy the binary to your bin path

tar -xvf velero-v1.5.3-linux-amd64.tar.gz
sudo cp velero-v1.5.3-linux-amd64/velero /usr/bin/

To check what is your CLI version:

velero version --client-only

output shoud be like

Client:
        Version: v1.5.3
        Git commit: 123109a3bcac11dbb6783d2758207bac0d0817cb

Next update the Velero custom resource definitions (CRDs) to include schema changes across all CRDs that are at the core of the new features in this release:

velero install --crds-only --dry-run -o yaml | kubectl apply -f -

Finally update the container image used by the Velero deployment and the restic daemon set:

kubectl set image deployment/velero \
    velero=velero/velero:v1.5.3 \
    --namespace velero

kubectl set image daemonset/restic \
    restic=velero/velero:v1.5.3 \
    --namespace velero

** Backup pod volume by default**

Okay so now your Velero is up to date and you want to change your approach for the opt-out in order to backup pod volume by default. To do so you need to edit the velero deployment to add a lauch argument

--default-volumes-to-restic

kubectl edit deploy velero -n velero

Go to spec.template.spec.containers.args and add this argument. See part of the yaml below.

... spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: deploy: velero strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: prometheus.io/path: /metrics prometheus.io/port: "8085" prometheus.io/scrape: "true" creationTimestamp: null labels: component: velero deploy: velero spec: containers: - args: - server - --features= - --default-volumes-to-restic #ADD THIS LINE command: - /velero ...

You need now to restart the pod. You can for example scale down the deployment and scale it up again.

kubectl scale deploy velero -n velero --replicas=0

wait pods to be removed..

kubectl scale deploy velero -n velero --replicas=1

Here we go you can now test it. For the example I will backup a whole namespace called gophish that contain a deployment with a PVC.

velero backup create my-backup --include-namespaces gophish

when it is done you can describe the backup

velero backup describe my-backup

The output:

Name:         my-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.16.13
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=16

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  gophish
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-01-20 11:05:10 +0100 CET
Completed:  2021-01-20 11:05:21 +0100 CET

Expiration:  2021-02-19 11:05:10 +0100 CET

Total items to be backed up:  25
Items backed up:              25

Velero-Native Snapshots: <none included>

Restic Backups (specify --details for more information):
  Completed:  1

You can see at the bottom that a restic backup has been done. This is the pod volume that has been backed up without any annotation.

Enjoy!!

Feel free to comment this article if you have some question.

www.cisel.ch

Source: https://velero.io/docs