OpenEMR 6.1.0 Devops

It’s in but untested, I’ll run that testing tomorrow.

Note that this says “Lightsail” on the tin buuuuut I suspect it’d work with anything Dockerized, because it picks up the container’s MySQL host, so even Standard could probably use this script because the DB load should seamlessly pick up the RDS credentials. That’s one of the reasons I do the MySQL load through the OpenEMR container when I could just talk to the MySQL container instead – I want those passwords, and the MySQL container might not exist (Standard) or might be named something different (ARM).

1 Like

Test test test–

So while I’m waiting ~20m per test cycle, I can share the process here. No guarantee it’s functional yet.

  • Launch Ubuntu 20.04 instance in AWS. (t2.small is adequate, but don’t use a teeny 8gig volume, it’s too small)
  • Install Lightsail: curl -s https://raw.githubusercontent.com/openemr/openemr-devops/master/packages/lightsail/launch.sh | bash -s -- -s 0
  • While I’m waiting for the install, scp (well, I use WinSCP because I’m lazy but whatever) the emr_backup.tar file to the instance.
  • Move to /root/openemr-devops/packages/lightsail and then flip the permissions on the ingestion script.
  • Launch with ./ingestion.sh /home/ubuntu/emr_backup.tar.

The restore currently only picks up the sites directory and leaves customized code behind, but that’s more “because I don’t know how to do it better and safely” than an aspiration, improvement is still ongoing. See the whole thing at openemr-devops/ingestion.sh at master · openemr/openemr-devops · GitHub .

1 Like

Huh, it actually worked out of the box. Didn’t expect that.

2 Likes

So I haven’t decided what I think about this.

I’ve adjusted the Lightsail launch script to be able to operate with an ARM instance. Cranks right up with an ARM t4g.small, some of the cheapest per-hour cores Amazon will sell you. But… my XtraBackup solution is strictly x86 – I don’t (yet) know how to compile my frankly complicated custom MySQL container into ARM compatibility, since it’s hitting fairly exotic repos and I don’t know if /those/ are ARM-compatible at all.

I made it work, sure, we use a community MariaDB container here. And I do have backups for the patient records. But at the end of the day, if you launch Lightsail on an ARM instance you won’t get automated database backups and because Lightsail is fire-and-forget there’s not a great way to warn the user this has happened.

I’m considering disabling it or putting it behind a launch switch so that it /can’t/ work without user awareness. I like smooth launches but I don’t like “oh, you clicked the wrong radio button so your patient records never got saved”.

1 Like

100% success! Well, until people tell me it’s all broken, but that’s fine.

2 Likes

Okay, so I’ve been slapped back hard trying to generalize the backup ingestion script. Neither rsync nor (apparently) composer wanted to play nice with anything I was trying to do, and some of my explored concepts (like deleting missing files on source) were clearly not going to work out with the backup I was inspecting.

I had hoped to generalize the machinery and be able to run modified OpenEMR code picked up from a backup in the recovered instance. This endeavor 100% failed, probably because I don’t know enough about OpenEMR’s toolchain and how anything actually loads or functions… or anything about how PHP anything functions, really. I’m not saying it’s not possible, but I don’t know how to do it and I may have to leave it to an interested user to expand the script to pick up their own specific forms or templates or classes and move them over.

(Which would be easy, just patch the filecopy bit of the script, right? But I don’t know what’s what and what matters, and trying to Just Rsync It Over gave me permission problems, ownership problems, compilation problems, and launch problems, and I never got close to fixing anything I broke.)

This week I’ve cleaned out a lot of older issues against the repo, leaving fairly difficult features requests I’m not immediately ready to tangle with, and one log rotation question I should /really/ sit down and settle.

Am thinking of trialing a Lightsail variant with the flex docker, XDebug, and VS Code’s Remote bridging to allow remote development or patching on a live cloud server. Is that a good idea?

1 Like

If you’re taking votes, I’d appreciate a fix for the log rotation. I was planning to use Apache’s own rotatelogs utility to keep a circular list of size-based rotated files.

Well, we had some issues with it ( it’s worth reviewing logrotate openemr container · Issue #161 · openemr/openemr-devops · GitHub ) that never got shaken out, but it /was/ like a million Alpine versions ago and we do launch a crond inside the OpenEMR container so logrotate shooooould be functional. Maybe things are better now.

Cracked log rotation, at least well enough for you to tweak it if you don’t like my decisions. It’s not in production containers yet but you can see how to do it in 7.0.0: log rotation · openemr/openemr-devops@98c2b5f · GitHub and modify your own stuff to match until it gets in later.

1 Like

Yup, that’s just what I was going to do, looks good. Piped logs avoid the problem of having to reload httpd. Might not be for everybody, though.

I agree in theory – I’m really disappointed in how the pipe names its files – but in practice we shouldn’t ship with “and the logs get bigger forever lol” but there isn’t a solution that isn’t either rotatelogs or logrotate with a crond.

Also this lets the Kubernetes guys tee off it to get their logs into /dev/stdout and /dev/stderr which I’m told is a laudable thing.

This OpenEMR-Standard.json does not work. When I attempt to create a cloudformation stack in Virginia (us-east-1), I get the following error:

The following resource types are not supported for resource import: AWS::S3::BucketPolicy,AWS::EC2::VPCGatewayAttachment,AWS::EC2::SecurityGroupIngress,AWS::EC2::SecurityGroupIngress,AWS::EC2::SecurityGroupIngress,AWS::EC2::Route

The “learn more” link for this error links to the following AWS documentation:

I don’t know anything about importing or drift detection – I’m certainly not trying to make use of them and haven’t attempted to test any such capacity. Can you go into more detail about how you’re trying to launch the stack? Because it did launch fine when I posted it, I swear.

The first time I did this, I used the AWS Marketplace Openemr STANDARD and chose launch cloudformation and then picked your template file … that caused the errors above.

Then I decided to simply go to cloudformation directly with the following parameters:

I simply when to create a cloudformation stack, chose the template file from the post/link above in your message, and then used these parameters:

AdminPassword ****************
EC2KeyPair OpenEMRkeyPair
PatientRecords 100
PracticeStorage 200
RDSInstanceSize db.t3.medium
RDSPassword ******************
UserCidr 10.0.0.0/16
UserSubnetPrivate1 10.0.2.0/24
UserSubnetPrivate2 10.0.4.0/24
UserSubnetPublic1 10.0.1.0/24
UserSubnetPublic2 10.0.3.0/24
WebserverInstanceSize t3.large

This time, I encountered the following before the stack rollback began due to failure to create the stack:

2022-07-03 23:55:48 UTC-0500 ig
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 OpenEMRKey
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 rtTablePublic
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 rtTablePrivate
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 PublicSubnet1
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 PrivateSubnet1
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:47 UTC-0500 S3Bucket
CREATE_FAILED Resource creation cancelled
2022-07-03 23:55:46 UTC-0500 PublicSubnet1
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 PrivateSubnet1
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 PrivateSubnet2
CREATE_FAILED Template error: Fn::Select cannot select nonexistent value at index 1
2022-07-03 23:55:46 UTC-0500 ApplicationSecurityGroup
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 SysAdminAccessSG
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 DBSecurityGroup
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 PublicSubnet2
CREATE_FAILED Template error: Fn::Select cannot select nonexistent value at index 1
2022-07-03 23:55:46 UTC-0500 rtTablePublic
CREATE_IN_PROGRESS -
2022-07-03 23:55:46 UTC-0500 rtTablePrivate
CREATE_IN_PROGRESS -
2022-07-03 23:55:43 UTC-0500 VPC
CREATE_COMPLETE -
2022-07-03 23:55:31 UTC-0500 VPC
CREATE_IN_PROGRESS Resource creation Initiated
2022-07-03 23:55:31 UTC-0500 ig
CREATE_IN_PROGRESS Resource creation Initiated
2022-07-03 23:55:31 UTC-0500 OpenEMRKey
CREATE_IN_PROGRESS Resource creation Initiated
2022-07-03 23:55:30 UTC-0500 S3Bucket
CREATE_IN_PROGRESS Resource creation Initiated
2022-07-03 23:55:29 UTC-0500 S3Bucket
CREATE_IN_PROGRESS -
2022-07-03 23:55:29 UTC-0500 VPC
CREATE_IN_PROGRESS -
2022-07-03 23:55:29 UTC-0500 OpenEMRKey
CREATE_IN_PROGRESS -
2022-07-03 23:55:28 UTC-0500 ig
CREATE_IN_PROGRESS -
2022-07-03 23:55:22 UTC-0500 oe-virginia-610-2022-07-03-2352h
CREATE_IN_PROGRESS User Initiated

I have seen the above error in the past with 6.0.0. The way I got around it was to choose a t2.medium instead of a t3.medium and to make sure that the database version was 5.7.33 in the cloudformation stack file. I don’t have this option here because I cannot choose t2.medium for my ec2. I suppose I could edit the template file to allow t2.medium somehow …
RBL

Okay, you should be able to just pick Standard from the Marketplace and /not/ select my json at all, that’s what might’ve triggered the drift tracking if it thought you were trying to sideload a bunch of changes. Just pick Standard and the json it lays out for you and tell me what happens, please?

t2.* won’t help, those generations aren’t possible for the stack anymore because I migrated to the new instances and those require different volume mounting schemes.

The logs you’ve supplied here don’t make any sense to me – those events shouldn’t be able to fail Fn::Selects. Can you disable stack rollback long enough to screenshot the section with the CREATE_FAILED entries?

1 Like

Okay, I figured out the problem. My us-east-1a did not have a default subnet based on an issue I had worked on in the past with AWS Support. I repaired the problem using the following aws CLI command from my laptop (after creating a programmatic access user with admin privileges:

aws ec2 create-default-subnet --availability-zone us-east-1a --region us-east-1

Here is what AWS Support told me: (in that case, it was an issue with us-west-2):
… you were trying to create a stack with multiple subnets and were utilizing “!Select”, “!GetAZs” to get the availability zones under the us-west-2 region. But while doing so, it was throwing Template error: Fn::Select cannot select nonexistent value at index 1 error. This above error is caused when there is no value at index 1 of the list entered in “!Select” Intrinsic function. That means “!GetAZs” is returning only one Availability zone. So I have tried to replicate the issue on my end and found out that this is caused when we have less than two default subnet under default VPC. once I added defaults subnets in all regions, then the issue was resolved. “!GetAZs” utilizes default subnets to estimate the availability zones. Problem: Having Only 1 default Subnets in us-west-2 in the account Solution: you can create default subnets in all availability zones using the below command. aws ec2 create-default-subnet --availability-zone us-west-2b --region us-west-2 https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html#create-default-subnet

1 Like

Oh goodness I don’t think I would’ve guessed “bug in CloudFormation’s AZ enumeration” as the source of error. I’ll keep this in mind in case it bites anybody else but I can’t armor the template against this issue in any useful way.

Thanks for getting to the bottom of it!

1 Like