Transferring data between Kubernetes clusters

I recently switched from Digitalocean’s managed Kubernetes service to GKE (Google). Reasons are some instability and the fact that they force you to upgrade your Kubernetes version every other month. Don’t get me wrong, Digitalocean is still a great place to spin up fairly cheap VMs. Not to mention their great support service.

Most of my applications running are stateless, so simply pointing arogcd to my new cluster was enough. However, I also had some applications which used PersistentVolumeClaims, or PVCs for short. Depending on the Kubernetes service, a PVC can be a virtual disk attached to your cluster’s nodes. I use it to to store database data like monogdb data and customer data (simple files).

One of way transferring data would be to use the kubectl cp command and copy the data to your machine and then back to the new cluster. However, this does not always work as kubectl cp does some magic tarring in the background and copying over a lot of data using this method is prone to errors.

So I came up with a better way. The trick is to spin up a pod in each cluster, mount all PVCs in those pods and transfer data between those pods. “Transfer data between those pods”, I hear you say… “But they are running in different clusters!” Indeed, but with some magic called ssh and a service of type LoadBalancer, you can do this. Have a look at this drawing: