#Block Storage Disaster Recovery

RBD Mirror is a feature of Ceph Block Storage (RBD) that enables asynchronous data replication between different Ceph clusters, providing cross-cluster Disaster Recovery (DR). Its core function is to synchronize data in a primary-backup mode, ensuring rapid service takeover by the backup cluster when the primary cluster fails.

#Terminology

#Backup Configuration

#Prerequisites

• Prepare two clusters capable of deploying Alauda Build of Rook-Ceph: a Primary cluster and a Secondary cluster, with network connectivity between them.
• Both clusters must run the same platform version (v3.12 or later).
• Create distributed storage services in both Primary and Secondary clusters.
• Create block storage pools with identical names in both Primary and Secondary clusters.
• Please ensure that the following two images have been uploaded to the platform's private image repository:
  • quay.io/csiaddons/k8s-controller:v0.12.0 -> <registry>/csiaddons/k8s-controller:v0.12.0
  • quay.io/csiaddons/k8s-sidecar:v0.12.0 -> <registry>/csiaddons/k8s-sidecar:v0.12.0

#Procedures

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF

#Planned Migration

Use cases: Datacenter maintenance, technology refresh, disaster avoidance, etc.

#Relocation

The Relocation operation is the process of switching production to a backup facility(normally your recovery site) or vice versa.

For relocation, access to the image on the primary site should be stopped. The image should now be made primary on the secondary cluster so that the access can be resumed there.

#Prerequisites

• The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

#Procedures

Follow the below steps for planned migration of workload from the Primary cluster to the Secondary cluster:

1. Scale down all the application pods which are using the mirrored PVC on the Primary cluster.

2. Update VolumeReplications for all the PVCs which mirroring is enabled on the Primary cluster.

   Set spec.replicationState to secondary.

3. Create VolumeReplications for all the PVCs for which mirroring is enabled on the Secondary.

   Example

   cat << EOF | kubectl apply -f -
   apiVersion: replication.storage.openshift.io/v1alpha1
   kind: VolumeReplication
   metadata:
     name: <vr-name>
     namespace: <namespace>
   spec:
     autoResync: false
     volumeReplicationClass: rbd-volumereplicationclass
     replicationState: primary
     dataSource:
       apiGroup: ""
       kind: PersistentVolumeClaim
       name: <pvc-name>
   EOF
   1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
   2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
   3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

4. Check VolumeReplication CR status to verify if the image is marked primary on the secondary site.

5. Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

#Disaster Recovery

Use cases: Natural disasters, Power failures, System failures, and crashes, etc.

#Failover (abrupt shutdown)

In case of Disaster recovery, create VolumeReplication CR at the Secondary Site.

Since the connection to the Primary Site is lost, the operator automatically sends a GRPC request down to the driver to forcefully mark the dataSource as primary on the Secondary Site.

#Prerequisites

• The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

#Procedures

1. Create VolumeReplications for all the PVCs for which mirroring is enabled on the Secondary.

   Example

   cat << EOF | kubectl apply -f -
   apiVersion: replication.storage.openshift.io/v1alpha1
   kind: VolumeReplication
   metadata:
     name: <vr-name>
     namespace: <namespace>
   spec:
     autoResync: false
     volumeReplicationClass: rbd-volumereplicationclass
     replicationState: primary
     dataSource:
       apiGroup: ""
       kind: PersistentVolumeClaim
       name: <pvc-name>
   EOF
   1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
   2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
   3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

2. Check VolumeReplication CR status to verify if the image is marked primary on the secondary site.

3. Once the Image is marked as primary, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.

#Failback (post-disaster recovery)

Once the failed cluster is recovered on the primary site and you want to failback from secondary site, follow the below steps:

#Prerequisites

• The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

#Procedures

1. Scale down the running applications (if any) on the primary site. Ensure that all persistent volumes in use by the workload are no longer in use on the primary cluster.

2. Update VolumeReplication CR replicationState from primary to secondary on the primary site.

3. Scale down the applications on the secondary site.

4. Update VolumeReplication CR replicationState state from primary to secondary in secondary site.

5. On the primary site, verify the VolumeReplication status is marked as volume ready to use.

6. Once the volume is marked to ready to use, change the replicationState state from secondary to primary in primary site.

7. Scale up the applications again on the primary site.