Docker, Google and SSL

OpenEMR Development
#1

TLDR: Has anyone using docker set up SSL yet with the automated (docker-compose) or other methods?

I have been working on getting the docker open-emr image up on google infrastructure, and have been successful so far. I have been documenting my process and will make that available once I get it cleaned up. I could not get it working following google’s tutorial at https://cloud.google.com/community/tutorials/docker-compose-on-container-optimized-os because the file system on the container optimized images is read only.

Their smallest (1 shared cpu, .6gb memory) was too short on memory. The next largest (g1-small (1 vCPU, 1.7 GB memory)) is roughly $11 a month to run and it is working for me so far in development (and may even work for my wife). I used a ubuntu 16.04 image, installed docker, pulled my source from github and was working on a live dynamic IP within an hour or so.

I was able to secure a static address, figure out how to get the a name dns entries to add a forward from one of my wife’s domains to the address in google’s cloud, and can access the site through http.

I want to enable SSL and the ports are forwarded appropriately, but I can’t seem to figure out how to accomplish the certificate part in openemr. I tried to follow the instructions under Administration/Other/Certificates, but failed once I could not restart the apache service within the docker container. I then saw that I could put the DOMAIN and EMAIL environment variables to automate the process but I got the following message in the log:

dhillison@another-emr:~/openemr$ docker container logs 3813cc9ff372        
WARNING: SETTING AN EMAIL VIA $EMAIL is HIGHLY RECOMMENDED IN ORDER TO
         RECEIVE ALERTS FROM LETSENCRYPT ABOUT YOUR SSL CERTIFICATE.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for emr.capitalspeechpath.com
Using the webroot path /var/www/localhost/htdocs/openemr for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. emr.capitalspeechpath.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://emr.capitalspeechpath.com/.well-known/acme-challenge/C27aszh6Snl2ZbSryceQQpijk39RfTrGk_xhMoURrUM: Timeout
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: emr.capitalspeechpath.com
   Type:   connection
   Detail: Fetching
   http://emr.capitalspeechpath.com/.well-known/acme-challenge/C27aszh6Snl2ZbSryceQQpijk39RfTrGk_xhMoURrUM:
   Timeout

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
WARNING: SETTING AN EMAIL VIA $EMAIL is HIGHLY RECOMMENDED IN ORDER TO
         RECEIVE ALERTS FROM LETSENCRYPT ABOUT YOUR SSL CERTIFICATE.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None

It kept trying to redo this step and getting the error until letsencrypt stopped responding to its requests. I ended the container, so I don’ have access to the lets encrypt log. Is it necessary for me to get it or am I doing something else boneheaded that needs to be fixed first?

Has anyone using docker set up SSL yet with the automated or other methods?

Thanks,
Derek

#2

Have you set up a virtual host for port 443 (ssl.conf)? I had trouble with the ssl mod wasn’t starting so I added Listen 443
LoadModule ssl_module modules/mod_ssl.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
to top of openemr.conf.
Also check sites/default/document permissions for write flag(nothing to do with ssl). Easier to restart container to restart apache but I believe you can restart apache inside the container but I haven’t tested yet.

#3

TLDR: When I check the certificate in Chrome, it shows OpenEMR CA as the issuing authority. Should this not show letsencrypt instead?

Thank you for the quick response! I did figure out how to restart the container (docker container restart ), but not apache, see the end of the message.

I forgot to preface everything how new all of this to me this is. This is my first docker, php, linux programming effort, so there may be basic stupid assumptions that I don’t know. I also probably don’t understand how ssl works from a server perspective.

It is a testament to prior efforts of everyone to create such a robust solution, the help I receive here and from the wiki, and google searches that I can even get it running with no prior specific experience. I last did web development in vbscript in IIS and MS SQL server, and that was 2003! I have been programming in r and other languages since

The 443 port is currently forwarded to 8081, and when I go there chrome won’t let me in because it is using a self-signed certificate, I think because it says my operating system If I tell chrome its ok, everything works as it should with a self-signed certificate. I think the port is working.

I found the lines you gave in ssl.conf, not openemr.conf. I also tried adding these lines below to the httpd file as shown in administration/other/certificates after unzipping the created file and copying them into the correct location, but I still get the privacy error in chrome.
SSLEngine on
SSLCertificateFile /etc/ssl/certs/Server.crt
SSLCertificateKeyFile /etc/ssl/certs/Server.key
SSLCACertificateFile /etc/ssl/certs/CertificateAuthority.crt

There may have been a way to restart apache that I did not know…
I tried
docker exec -it /bin/sh

cd /etc/init.d/
/etc/init.d # ls
apache2 dcron
/etc/init.d # apache2 restart
/bin/sh: apache2: not found
I also saw references to using service and rc-service, but these were not in the alpine base image that is used for the docker image.

#4

hi @dhillison ,

We very recently brought in fixes to the docker to fix some letsencrypt bugs. I don’t know details, but either @jesdynf or @toolbox should be able to provide you further enlightenment.

-brady

#5

Need to keep your certificates in ssl.conf. If you already have certificates then add their location to SSLCertificate directives. Looks like your ssl mod is loading so may be virtual host setup(your’re right the Verified by should be Let’s Encrypt). In both openemr.conf and ssl.conf the virtual host is set to <VirtualHost _default_:443> and openemr.conf loads self signed certificates therefor we can’t be sure which is taking president. Try to change virtual host in ssl.conf to start with something like this
<VirtualHost *:443>
DocumentRoot “/var/www/localhost/htdocs”
ServerName yourserver.com ##this should be exact domain used in certificate
ServerAdmin your-email@somewhere
ErrorLog logs/ssl_error.log
TransferLog logs/ssl_access.log

If this doesn’t work then one of the guy’s Brady mentioned may be able to shed a better light. This worked for me.

#6

Hello, Derek.

We have confirmed success with the Let’s Encrypt tooling that ships on the docker. Setting DOMAIN and EMAIL variables in the docker-compose, making sure 443 is world-readable, and making sure your hostname is assigned to your instance, should be all that’s needed.

What you’re describing indicates you’ve got the first of the three correct – Docker’s LE is certainly trying to latch. I suspect it’s some issue with your network config.

#7

For what it’s worth, the docker images come with pre-built configs for ssl, and shouldn’t require messing with apache configs for that purpose. It seems, like @jesdynf suggests, that there may be a networking issue preventing access. Additionally, @dhillison may have an old version of the image, which contained a bug that tried to do the domain verification without starting apache first. You can make sure you have the right image by making sure the container is stopped/killed/removed, and doing a docker image prune before trying again.

side note: we should probably consider changing something so whatever setup in question doesn’t retry over and over by default and make LE’s rate limits unhappy.

#8

Thank you @toolbox I really appreciate all your work on getting this running on docker. I re-pulled the image from master, and will try again this afternoon by editing the docker-compose file with domain and email. Looking at my error log from that previous run, it was looking for the acme file on the server and failing, and if Apache is not up yet, the file would not be available. I will report what I find with the current (last night’s) master. I agree that a more graceful failure would be better than continuing to error.

From the docker-compose file, the network looks right with 80 -> 8080 and 443 -> 8081. If I don’t enable ssl in any way, port 8080 works just fine. With SSL on and a manually placed and configured location to the certs, a https connection to 8081 works, but the cert shows as signed by openemr. I generated these certs using the interface under administration, so they might not have generated correctly or I might not configure the ssl.conf and openemr.conf files correctly.

@jesdynf: In google console, I typed hostname in the instance, and it took the name of the instance, which in my case did NOT match the domain. I will create an instance that uses this name and try again. This might have been a key factor in its failure.

Thanks,
Derek

#9

Still say that there are two virtual host on same port with one in openemr.conf configured with default cert’s. I’m just not sure having two virtual hosts configured on same domain and port is a good idea(default in this case). Who gets served? Imo.

#10

I really appreciate all the help. I am now officially several hundred feet below the level of necessary knowledge, but I am swimming upwards thanks to everyone’s help.

Here are links to the conf and docker-compose files. Let me know if you see any errors. I put all the .key and .crt files in /etc/apache2/ssl/certs and edited the file locations in the conf files. Both files contained links to pem files and virtual host information in their original forms. I am describing several test runs with combinations of these files.

The first used the attached conf files with only updated directory and filenames to key and crt files. Also commented out the DOMAIN and EMAIL environment variables in the docker-compose. Result: Openemr signed certificate and privacy error

The second used the original conf files from the sloboy/opnemer master that I pulled from openemr/openemr master last night and the uncommented docker-compose. Result: Looping restart with message as in OP.

A third run with all original files, docker-compose commented out DOMAIN and EMAIL. Result: localhost signed certificate.

(updated) I will try to pull openemr/openemr master down to instance and try that one just to make sure I did pull request correct. Edited docker-compose file to contain DOMAIN and EMAIL variables, and it loops in restart as above.

After that will try changing hostname that is given to the container and use conf files and cert files I downloaded.

https://storage.googleapis.com/slo-open-emr/openemr.conf
https://storage.googleapis.com/slo-open-emr/ssl.conf
https://storage.googleapis.com/slo-open-emr/docker-compose.yml

#11

@toolbox I just re-read your post. I am not using one of the available docker images, rather I am building them from github repository. I have been making small customizations to the code, so the official images don’t work for what I am doing. Where on github can I pull the code that builds the images?

#12

@dhillson, you’re looking for GitHub - openemr/openemr-devops: OpenEMR administration and deployment tooling ; check the /docker folder, and pick the version you’re after. Take a look at our Lightsail launch script openemr-devops/launch.sh at master · openemr/openemr-devops · GitHub for an example of an end-to-end launch.

#13

In ssl.conf for server name we have emr.xxxxxxxxxxx.com:443. We already know port and looks like you’re using a sub domain, what’s your A record? Is the domain of cert a sub domain? and moreover just humor me and in ssl.conf change virtual host to *:443 and change server name to A record’s name. No need for :443 port on server name.

#14

@sjpadgett I will try to change those and report back.

Here is a link to a picture of the A record for the domain. Don’t laugh, my wife can do the page updates herself on wix, and then I don’t have to. I think the A record is working from a DNS standpoint because http://emr.XXXXX.com:8080 and http://emr.XXXXX.com:8081 go to the google provided ip. I used emr.XXXX.com when asking for the certificate, rather than XXXX.com.

https://docs.google.com/drawings/d/1mIcpdcolQNdPcdqJNN7penbxM45zjlfoQVzw9mTXbZc/edit?usp=sharing

#15

Yep emr.xxxxxxx.com for server name should work but I had trouble with sub domains in docker but, IT i’m not. If A record for server name doesn’t work I would try out just the domain xxxxx.com see if cert fails.

#16

thank you so much @jesdynf I have been playing with the Dockerfile in the 5.0.0 release of the devops on github. I cloned the master release from openemr and use the Dockerfile (from devops) to build an image. When I run docker-compose, the container fails to start because it can’t find where the run_openemr.sh file is. This happened both on google instance and my own development environment.

I then tried to use the given docker-compose file from the devops 5.0.0 readme, where it downloads the openemr image directly. it will run fine until I give it a DOMAIN and EMAIL env variable and restart. It then gets into a restart loop because of the LE error in the OP. I am sure it is a pebcak problem, but I need to read and study some more. Do you have any thoughts why either method would fail?

One issue may be that I keep also seeing a httpd error where it cannot determine a fully determined server name, and it appears it should be the url of the server set in httpd.conf. This might be the source of my problems. I am currently trying to set this using the manual approach of installing certificates, but the only way I know how is to exec into the running containers to set this up. if the container is continually restarting, I can’t exec into it. I tried spamming up arrow and enter to try:)

#17

I’m not sure what’s going on in your environment, or why you might not be able to find run_openemr.sh – I suspect you weren’t in the right directory when you tried to use docker-compose. Take a second look at the Lightsail script I quoted to see how to clone the image and launch the docker. (And how to force the use of a local Dockerfile, via build, instead of the image.)

Once you’re sure you’re able to launch the local Docker, built locally, see if you’re still having that issue with the crashing LE.

#18

@jesdynf @sjpadgett I figured out what was wrong with the lets encrypt failing! I was using non-standard ports of 8080 and 8081 for http and https respectively. Lets encrypt apparently only looks for 80 when verifying. I only learned this by trying to get the lets encrypt certs manually and it threw the missing verification error.

Once i fixed the ports in docker-compose and changed the port forwarding on the google instance, everything is golden so far. This was definitely a pebcak problem. Now I just have to troubleshoot a password issue, but I will repost if I cant figure it out over the next few days.

Somehow I also got the docker-compose to work with the openemr image, but I don’t know what I did.

Thank you for everything!

#19

Glad it all worked out! I’ll make sure to add “are you changing the ports” to my list of debugging questions.

#20

Hey Asher, I’m sort of a newbie to Docker and it seems like you’re one of the go-to devs on this issue. We have a running OpenEMR instance in a Docker container that was created through GCP launchpad. Is it possible to set the DOMAIN and EMAIL environment variables from within a Docker host VM created using the GCP launchpad? Can you pass them in through the command line via “docker run” or do they have to be built into the image (ie: in the docker-compose.yml file?) Ultimately, I’m trying to figure out best practices for updating code, and configuring SSL for an existing instance of OpenEMR running in the GCP container which was created using the default instructions. Perhaps it would be better to build my own image, pointing to their new custom source code git repo, copy their volume data and re-deploy to a different VM? Thanks!