Migration between two clusters that also have shared storage

[djjudas21]

I have a bare metal Kubernetes cluster with 2 CSI storage classes - hyperconverged Ceph, and off-cluster TrueNAS. The cluster is also configured for Velero backup to S3.

I now want to migrate to a new cluster using Velero to do its thing and recreate the new volumes. But, the new cluster is also connected to the same TrueNAS. For the volumes on TrueNAS, it will be way faster to create the existing PVs on the new cluster and magically have access to the data, rather than migrate them to and from and the same NAS via S3. I've already verified with the author of the TrueNAS CSI driver that it's possible to access my TrueNAS RWX PVs from two clusters.

I've made a basic diagram to show what I'm talking about.

Is it possible to do this? Obviously I can get Velero to make backups and exclude the TrueNAS PVs, but when I come to restore the workloads I guess I will need to manually create the PVs to point at my existing TrueNAS volumes, then do the restore, which will reinstate the PVCs and allow the workloads to mount the volumes.

[blackpiglet]

It's possible, but Velero only does that in a special case.
https://github.com/vmware-tanzu/velero/blob/804d73c4f2349f1ca9bd3d6c751956e1d2021c01/pkg/restore/restore.go#L1241-L1247
This is the logic that Velero can restore the PV with the volume as it was backed up.
It requires the PV's ReclaimPolicy set to Retain, and the PVs are not backed up by filesystem backup or snapshot backup.

This can be done by CLI velero backup create --default-volumes-to-fs-backup=false --snapshot-volumes=false

[djjudas21]

Brilliant, thanks for the advice. If I want to exclude just some volumes, will it be OK to label those PVs with something to exclude them from Velero, rather than a blanket --default-volumes-to-fs-backup=false --snapshot-volumes=false?

[blackpiglet]

Please check this document https://velero.io/docs/v1.15/resource-filtering/.
The most up-to-date way to filter volume is the resource policy https://velero.io/docs/v1.15/resource-filtering/#supported-conditions, but it doesn't support filtering by labels.
There are some ways to filter by label or annotation:

• https://velero.io/docs/v1.15/resource-filtering/#veleroioexclude-from-backuptrue
• https://velero.io/docs/v1.15/resource-filtering/#--selector
• https://velero.io/docs/v1.15/resource-filtering/#--or-selector
• https://velero.io/docs/v1.15/file-system-backup/#to-back-up

Please check which one is your best choice.