OpenEMR 6.1.0 Devops

Okay, back and forth on some submission issues but I cleared those hurdles, Express has now been submitted to AWS for posting.

Current standing:

Lightsail (shared hosting): 6.1.0 OK
Express: In process at Marketplace
Express Plus: 6.1.0 OK
Standard: In process at Marketplace

Next steps:
Express Plus recovery pathing via specified parent’s S3/KMS
Graviton support for ARM (just as a proof of concept)

Is it finally time to retire Full Stack? I’m not sorry I built it, it was the foundations for everything else, but nobody uses it, I never got a request to update it to 6.0.0, and I’m not sure I would’ve honored it if I had, because Standard does 95% of what it does better – basically everything but the clustering.

3 Likes

Quick update:

  • Express is now up in Marketplace.
  • Marketplace staff have questions about Standard I’ll need to answer but I should straighten out tonight.
  • The first brushes of a recovery path for Express Plus are in progress.
4 Likes

Alright, Standard 6.1.0 is up in Marketplace. I don’t think I’ll get much on Express Plus recovery done today or tomorrow but I’d at least like to do some stack testing this weekend.

2 Likes

Okay, so lot of things coming, some of which nobody but me will care about, some of which nobody but me will care about in a very different way. (None of these are committed, they’re in a dev branch I’m working with.)

  • Lightsail can be launched from and point to a devops branch for testing that doesn’t involve pushing live and finding out.
  • Lightsail has an “empty” launch mode now that will skip autoconfiguration.
  • Lightsail uses better technology for waitloops and won’t exit the launcher script before system configuration is done, which will mean no more dumb “wait for it and it’ll be done eventually” issues, if the script is running installation isn’t done yet.
  • Express Plus now uses gp3 volumes, not gp2.
  • Express Plus (and everything that leans on Lightsail) should have the same trick, the stack won’t finish until /everything’s done/.
  • Express Plus UTF16 problems from PowerShell and Python conflicts are now understood and worked around.
  • Express Plus developer mode will now ask for a devops branch to launch from.
  • Express Plus recovery mode is coming – just like Standard, you should be able to specify an EP parent stack key and bucket, and have recovery mode load those backups into the instance. The recovery stack has its own, seperate key and bucket, and only read access to both resources, so the backups themselves are inviolate and can’t be injured by the recovery instance. As long as your instance doesn’t call out it should be safe to spawn and examine and delete – or to migrate to and leave the old instance behind, if you’re trying to (say) migrate regions.
2 Likes

Oh my goodness but I can’t believe it all hooked together.

Express Plus recovery is in – seamless recovery from backups via child stack. I’ll do a longer writeup later; for now, you can verify by:

  • Launching the regular Express Plus stack.
  • Make changes (create patient, add document to patient, change password on admin user).
  • Manually connect to instance and force backups to run (sudo bash and then /etc/cron.daily/duplicity-backups).
  • Verify backups have been created in S3 bucket (probably from the AWS console, it’s what I did).
  • Launch the recovery template from the devops repo and answer the questions it asks with your stack’s KMS key arn and your stack’s bucket name.
  • When the new stack comes up, verify you can log in with your new password and retrieve the document you uploaded from the patient.
3 Likes

Current project queue, let’s see how far we get this weekend:

  • Starting work on scripted ingestion of OpenEMR backups (the tarball) directly into Lightsail because it can’t be /that/ hard to get right if I’ve got the docker-compose file with the answers right there.
  • You know what, I’ve never messed with Amazon’s SSM Agent and maybe it’d be nice if people could one-click their way into their instances. (But will that cause HIPAA concerns? I may learn how to do it just to decide that no, I’m not going to allow it.)
  • Proof-of-concept Lightsail fork for ARM instances (no xtradb backups, sorry) for AWS Graviton and Raspberry Pi.
1 Like

Well, fancy that.

The Ubuntu 20.04 AMI I use for Lightsail supports EC2 Instance Connect out of the box – buuuut there’s a catch about security groups! If you’re properly careful, your ssh port is only open to your personal IP, and EIC sessions launched from the AWS Console don’t come from your browser, they come from IPs at the mothership. You need to identify the IP range the requests will come from* if you want the connection to go through, or you need to download the console Instant Connect application for your desktop.

You could also use the SSM stuff, but that’ll take an assigned IAM role and if you’re using Express Plus or Standard I’ve already got such a role in place, so you’ll need to add policies.

(*): Troubleshoot connecting with EC2 Instance Connect or just curl -s https://ip-ranges.amazonaws.com/ip-ranges.json| jq -r '.prefixes[] | select(.region=="us-east-1") | select(.service=="EC2_INSTANCE_CONNECT") | .ip_prefix' if you want to get on with things.

2 Likes

Oh god, I can’t believe this is a thing, I cannot believe this is a thing

mkdir /tmp/backup-ingestion
cp /tmp/backup-ingestion
tar -xf ~/emr_backup.tar --no-same-owner

# restore site
mkdir webroot
tar -zxf openemr.tar.gz -C webroot
rm openemr.tar.gz
# still in progress

# restore database
gzip -d openemr.sql.gz
echo 'USE openemr;' | cat - openemr.sql | docker exec -i $(docker ps | grep _openemr | cut -f 1 -d " ") /bin/sh -c 'mysql -p"$MYSQL_ROOT_PASS"'
rm openemr.sql

Still a WIP, but I was 100% not aware that “piping a SQL log into a container” was a workable decision. I figured I’d end up doing a buncha file copies, but no.

1 Like

Okay! First draft of a single-site OpenEMR backup file import into Lightsail.

This still can’t be used (hardcoded file names, bleah) but it’s what I’m thinking, if anybody finds it useful to read or correct. Cleanup and final testing are ongoing.

#!/bin/bash

# Some notes.
# One: We do not try to handle multisite, and we assume your openemr DB is named openemr. Will make that a parameter later.
# Two: Current I target only the sites directory, which may leave customization behind. You'll want to extend this recovery to
#      pick up those changes (and maybe rerun composer?) but I'm not familiar with that part. I just know copying node_modules
       *has* to be the wrong decision.
# Three: You'll want a /lot/ of room to unpack. An eight-gig instance won't cut it.

mkdir /tmp/backup-ingestion
cd /tmp/backup-ingestion
tar -xf ~/emr_backup.tar --no-same-owner

# retrieve site
mkdir webroot
tar -zxf openemr.tar.gz -C webroot
rm openemr.tar.gz
docker cp $(docker ps | grep _openemr | cut -f 1 -d " "):/var/www/localhost/htdocs/openemr/sites/default/sqlconf.php webroot/sites/default
docker cp webroot/sites/default $(docker ps | grep _openemr | cut -f 1 -d " "):/var/www/localhost/htdocs/openemr/sites/default-recovery

# straighten out internal permissions
docker exec -i $(docker ps | grep _openemr | cut -f 1 -d " ") /bin/sh -s << "EOF"
cd /var/www/localhost/htdocs/openemr/sites
chown -R apache:root default-recovery
chmod -R 400 default-recovery
chmod 500 default-recovery
chmod -R 500 default-recovery/LBF default-recovery/images
chmod -R 700 default-recovery/documents
mv default /root/default-old
mv default-recovery default
EOF

# restore database
gzip -d openemr.sql.gz
echo 'USE openemr;' | cat - openemr.sql | docker exec -i $(docker ps | grep _openemr | cut -f 1 -d " ") /bin/sh -c 'mysql -p"$MYSQL_ROOT_PASS"'
rm openemr.sql

# swift kick to PHP
docker restart $(docker ps | grep _openemr | cut -f 1 -d " ")

cd /
rm -rf /tmp/backup-ingestion

echo Restore operation complete!
1 Like

It’s in but untested, I’ll run that testing tomorrow.

Note that this says “Lightsail” on the tin buuuuut I suspect it’d work with anything Dockerized, because it picks up the container’s MySQL host, so even Standard could probably use this script because the DB load should seamlessly pick up the RDS credentials. That’s one of the reasons I do the MySQL load through the OpenEMR container when I could just talk to the MySQL container instead – I want those passwords, and the MySQL container might not exist (Standard) or might be named something different (ARM).

1 Like

Test test test–

So while I’m waiting ~20m per test cycle, I can share the process here. No guarantee it’s functional yet.

  • Launch Ubuntu 20.04 instance in AWS. (t2.small is adequate, but don’t use a teeny 8gig volume, it’s too small)
  • Install Lightsail: curl -s https://raw.githubusercontent.com/openemr/openemr-devops/master/packages/lightsail/launch.sh | bash -s -- -s 0
  • While I’m waiting for the install, scp (well, I use WinSCP because I’m lazy but whatever) the emr_backup.tar file to the instance.
  • Move to /root/openemr-devops/packages/lightsail and then flip the permissions on the ingestion script.
  • Launch with ./ingestion.sh /home/ubuntu/emr_backup.tar.

The restore currently only picks up the sites directory and leaves customized code behind, but that’s more “because I don’t know how to do it better and safely” than an aspiration, improvement is still ongoing. See the whole thing at openemr-devops/ingestion.sh at master · openemr/openemr-devops · GitHub .

1 Like

Huh, it actually worked out of the box. Didn’t expect that.

2 Likes

So I haven’t decided what I think about this.

I’ve adjusted the Lightsail launch script to be able to operate with an ARM instance. Cranks right up with an ARM t4g.small, some of the cheapest per-hour cores Amazon will sell you. But… my XtraBackup solution is strictly x86 – I don’t (yet) know how to compile my frankly complicated custom MySQL container into ARM compatibility, since it’s hitting fairly exotic repos and I don’t know if /those/ are ARM-compatible at all.

I made it work, sure, we use a community MariaDB container here. And I do have backups for the patient records. But at the end of the day, if you launch Lightsail on an ARM instance you won’t get automated database backups and because Lightsail is fire-and-forget there’s not a great way to warn the user this has happened.

I’m considering disabling it or putting it behind a launch switch so that it /can’t/ work without user awareness. I like smooth launches but I don’t like “oh, you clicked the wrong radio button so your patient records never got saved”.

1 Like

100% success! Well, until people tell me it’s all broken, but that’s fine.

2 Likes

Okay, so I’ve been slapped back hard trying to generalize the backup ingestion script. Neither rsync nor (apparently) composer wanted to play nice with anything I was trying to do, and some of my explored concepts (like deleting missing files on source) were clearly not going to work out with the backup I was inspecting.

I had hoped to generalize the machinery and be able to run modified OpenEMR code picked up from a backup in the recovered instance. This endeavor 100% failed, probably because I don’t know enough about OpenEMR’s toolchain and how anything actually loads or functions… or anything about how PHP anything functions, really. I’m not saying it’s not possible, but I don’t know how to do it and I may have to leave it to an interested user to expand the script to pick up their own specific forms or templates or classes and move them over.

(Which would be easy, just patch the filecopy bit of the script, right? But I don’t know what’s what and what matters, and trying to Just Rsync It Over gave me permission problems, ownership problems, compilation problems, and launch problems, and I never got close to fixing anything I broke.)

This week I’ve cleaned out a lot of older issues against the repo, leaving fairly difficult features requests I’m not immediately ready to tangle with, and one log rotation question I should /really/ sit down and settle.

Am thinking of trialing a Lightsail variant with the flex docker, XDebug, and VS Code’s Remote bridging to allow remote development or patching on a live cloud server. Is that a good idea?

1 Like

If you’re taking votes, I’d appreciate a fix for the log rotation. I was planning to use Apache’s own rotatelogs utility to keep a circular list of size-based rotated files.

Well, we had some issues with it ( it’s worth reviewing logrotate openemr container · Issue #161 · openemr/openemr-devops · GitHub ) that never got shaken out, but it /was/ like a million Alpine versions ago and we do launch a crond inside the OpenEMR container so logrotate shooooould be functional. Maybe things are better now.

Cracked log rotation, at least well enough for you to tweak it if you don’t like my decisions. It’s not in production containers yet but you can see how to do it in 7.0.0: log rotation · openemr/openemr-devops@98c2b5f · GitHub and modify your own stuff to match until it gets in later.

1 Like

Yup, that’s just what I was going to do, looks good. Piped logs avoid the problem of having to reload httpd. Might not be for everybody, though.