Backup AWS Cloud Express Plus, restore on local Lightsail instance

Management and Administration
,
1

I would like to keep an on-premises backup instance of my cloud deployment for disaster recovery and testing purposes.

To accomplish this, I have been attempting to restore to the local instance using the duplicity backup files stored in S3, but am running into an issue with path differences between the environments.

Steps I have taken so far and tested successfully:

  1. Created Ubuntu Xenial VirtualBox VM similar to the old OpenEMR Appliance.
  2. Installed OpenEMR v6.0.0 Lightsail instance on VM using launch.sh.
  3. Scripted a download of the duplicity backup files from S3 using AWS CLI, placing them in /root/backups (where local backups would go).
  4. Configured the gnupg decryption that duplicity needs to use during recovery.
  5. Modified restore.sh so that duplicity does decryption on the files, even though it is not a cloud instance.

The issue I’m having at this point is that when duplicity expands the backup files, it does so to the same absolute path that they were backed-up from. So during recovery, the cloud files are expanded into /mnt/docker when they need to go into /var/lib/docker.

A couple questions:

Is there a better strategy for my use case (backup cloud, restore local) that people are using?

I’m not sure why the cloud deployment is mounting /mnt/docker and using it as a symlink source for /var/lib/docker - something to do with EBS I’m sure. Could that be replicated on a local Lightsail instance somehow? Is there another workaround for this, perhaps just moving /mnt/docker to /var/lib/docker after the duplicity operation in restore.sh?

These are probably questions for @jesdynf who put in the effort on these mechanisms, but I would appreciate any advice.

Finally, in case it might save anyone some time… I attempted to install a Lightsail instance within a Sysbox Docker container (the whole docker inside docker thing). It can be installed and run just fine, but Sysbox treats /var/lib/docker specially, and problems occur when the OpenEMR backup/restore process starts mucking with the Docker volume mounts there.

1 Like
2

Hi,

What you’re trying to do is reasonable, yes. Express Plus is reasonably agnostic about where it runs past the initial boot, and I’m always delighted to have people ask questions about restoring backups before they’re needed.

And you were correct to suspect EBS as the reason for the shenanigans; I set it up that way to take advantage of encrypted volumes with managed keys. It’s not special and if you establish the /mnt/docker symlink it should work just fine; alternately, you can tell Duplicity to use a different path to restore. https://askubuntu.com/questions/520152/extract-a-folder-from-duplicity-archive looks like it discusses this.

1 Like
3

Thanks! Your suggestion worked great.

restore.sh:

@@ -30,7 +30,7 @@ if [ -f /root/cloud-backups-enabled ]; then
   export PASSPHRASE
   duplicity --force s3://s3.amazonaws.com/$S3/Backup /
 else
-  duplicity --no-encryption --force file:///root/backups /
+  duplicity --force --file-to-restore mnt file:///root/backups /var/lib
 fi
 
 DR_SETSWAP=0

Fixing this alerted me to another issue. My recently deployed express plus instance didn’t have any xtrabackups created for the last few days. Only one full and one incr since deployment. I noticed that /root/allsetup.ok did not exist in the mysql-xtrabackup container. I am not sure how there was one successful full and one successful incremental backup without it, nor do I know what might have deleted that file a couple days after deployment. To fix, I touched /root/allsetup.ok and /root/force-full-backup then removed /tmp/xbackup.lock in the container, then executed /etc/cron.daily/duplicity-backups on the host.

1 Like
4

I don’t know what could’ve caused that either, but noted. Glad it worked out!