Our experience with the new OpenEMR 4.1.0

Day-to-Day Users
1

drbowen wrote on Friday, August 26, 2011:

My practice, BPUC, upgraded to 4.1.0-dev-tip downloaded on 08-22-2011 from github and went live on our production server 08-23-2011

OpenEMR server Ubuntu 10.04,
apache2/maverick 2.2.16-1ubuntu3.1
mysql  5.1.49
PHP 5.3.3-1ubuntu9.3 with Suhosin-Patch

Issues after upgrade:

1. Initial log-in to change the password to stronger encryption strength works…unless you type your password wrong. Then password has to be re-entered by the admin to fix this.

2. A number of the queries are very slow that absolutely kill performance on the patient “demographics page.

“pnotes”, “vitals”, “New Encounters”, “Reports”, any of the “lists” such as “Medications” were improved substantially from 40 seconds per click to normal performance after Jeremy Wallace (of MI-Squared) indexed these tables.

“Clinical Reminders” Still is a major problem.  Current query time on our database is 45 seconds per click.  This happens every time we go to the “demographics” page which only happens 10-15 times per encounter.  Clinical Reminders has only the default entries with no new data.

3. “Medication Reconciliation Performed?” check box is broken.  I can’t access this check box from Linux (Ubuntu, Centos) or Windows (7 professional) client machines.

Good Things:

The MI-Squared Lab Corp LEN (laboratory exchange network) is importing live data without problems.

Looking forward to going live with the ZMG e-prescription service.  I will report back as we get the kinks shaken out.

Sam Bowen, MD

2

bradymiller wrote on Friday, August 26, 2011:

Hi Sam,

Can you have Jeremy post what indexes he placed (github commit would be ideal, but anything will suffice).

For the Clinical Reminders, a great way you can test these out is by going to Admin->Alerts and selecting on for each rule separately (for Passive Alerts) and let us know the relative timing of the wait then (this will allow isolation of the rules that are taking so much time so we can figure out ways to optimize them).

Also, to get an idea, about how many patients do you have in the database and what is your server hardware (and is anything else but OpenEMR being served from it)?

Regarding the Medication Reconciliation box, note it is only checkable after checking ‘Transition/Transfer of Care?’

thanks,
-brady

3

drbowen wrote on Friday, August 26, 2011:

Jeremy had a family emergency is out of pocket right now.  I will encourage him to do this when he becomes available.

This particular chipset does not support virtualization and the server has the one instance of OpenEMR.  All other file, print, fax, dhcp are handled by different machines so this server really has 100% of its resources for just serving our production instance.

ASUS TW510-E2/AA4 Workstaion Tower/5U AMD Opteron 200 Socket 940 DDR 4xSATA Hotswap 600W Black&Silver
AMD Opteron 280 Dual-core 2.4GHz socket 940 OEM OSA280FAA6CB
Seagate ST31000340NS - SATA HDD 1TB Barracuda ES.2 Enterprise SATA 3.0GB/s 7200rpm 32MB 3.5-in Bulk RAID-5
2GB 256X72 PC3200 ECC Registered LP DDR (Original Micron) x 2 (4 g RAM total)

22532   patient_data
103751 form_encounter
107052 form_dictation
141543 pnotes
173639 lists

When we indexed the affected tables performance improved substantially but the did not affect the delay on the “Clinical reminders.”  The “Clinical Reminders,” “Patient Reminders,” and “Vitals” all appear to finish at the same time.  Normally when everything is tuned correctly there is little noticeable delay when the demographics page is loading.  Jeremy had spent a couple of days trying to track down the issue before getting called away.  He was thinking that there was a call that had to time out before finding the correct setting and executing the query.  I think that he was thinking this because all of the delays seemed to be the same 40-45 second time and did not seem to be related to the underlying table size.

Sam Bowen, MD

4

drbowen wrote on Friday, August 26, 2011:

It is a software RAID 5.

Sam Bowen, MD

5

bradymiller wrote on Saturday, August 27, 2011:

Hi Sam,

Thanks for the additional information. The size of your database is a very nice real life testing case. When Jeremy’s back, then will be a good idea to get the indexes he used and to also have him go through each rule in Admin->Alerts to see if can isolate if this is indeed one bad query. As you have noticed, the patient summary screen is rather intensive with a bunch of ajax calls to grab data.

-brady

6

yehster wrote on Saturday, August 27, 2011:

I haven’t paid much attention to the performance issues with on the Demographics page, I’m using a recent 4.1 dev version. Only have 3-4 test patients in my system. 
However, when I look at my Firebug javascript console I intermittently see that a number of the Ajax calls don’t return successfully.

These are some of the calls that don’t succeed.  Sometimes they all succeed, and other times only one or two fail.

GET http://192.168.227.128/openemr/interface/patient_file/summary/pnotes_fragment.php
GET http://192.168.227.128/openemr/interface/patient_file/summary/disc_fragment.php
POST http://192.168.227.128/openemr/interface/patient_file/summary/clinical_reminders_fragment.php
GET http://192.168.227.128/openemr/interface/patient_file/summary/patient_reminders_fragment.php
GET http://192.168.227.128/openemr/interface/patient_file/summary/vitals_fragment.php

I wonder if it’s the multiple simultaneous calls that are somehow stomping on each other.  For me it’s not a data size issue (or at least it shouldn’t be, since my DB is nearly empty.)

As an aside, all of those ajax calls on such a frequently visited screen can’t be good for scalability.  Doing them as multiple seperate calls certainly adds overhead since the server has to handle 5+ seperate requests instead of fewer.

7

drbowen wrote on Saturday, August 27, 2011:

That would explain some of the intermittent behavior I have been noticing.  Sometimes these calls work very quickly and on other patients they don’t.

patient_reminders_fragment.php

Is the one that seems most intermittent.

while

clinical_reminders_fragment.php

Always takes a while just over 40 seconds

patient_reminders_fragment.php
vitals_fragment.php

Appear to load “simutaneously” with clinical _reminders.

I agree it doesn’t seem to the size of the table since it is always just over 40 seconds for each of the calls.

Sam Bowen, MD

8

drbowen wrote on Saturday, August 27, 2011:

Now that Yehster has pointed this out, I tend to agree.  This is more erratic than I thought, with different queries completing first on different patients but the wait time is always just over 40 seconds.

Sam Bowen, MD

9

bradymiller wrote on Saturday, August 27, 2011:

Hi,

Modern browser support 6 concurrent http ajax requests (beyond this, they then get queued). Rather than guess now what is the issue, need to gather more information:

1) Is this an issues on very small scale databases (I haven’t noted this delay issue then only several patients, and also haven’t noticed the ajax widgets ever not finishing loading)
2) Is this an issue on small-scale databases (hopefully somebody can weigh in)
3) What mysql indexes improved th performance?
4) Does turning off both the clinical reminders and/or patient reminders in globals improve performance of the vitals widget?
5) By focusing only on one passive/reminder rule at a time in Admin->Alerts what are the effects on both clinical reminders and patient reminders respectively(ie. is one rule causing the issue?)?
6) Are you seeing the same issue in new patients with no records?
7) Is there a difference in patient that have 100 encounters vs 5 encounters?

This type of testing should begin to help isolate the issue(s) as to whether it’s secondary mysql query issues, php script issues, and/or multiple ajax calls issues.

Also, please clarify the timing of patient_reminders_fragment.php and vitals_fragment.php. You seem to hint towards patient_reminders_fragment.php being both intermittent in time but always coinciding with when clinical_reminder_frsagment.php finishes at 40 seconds. These seem to be conflicting remarks; please clarify. Also, to you have any rules turned on for the patient reminders (by default, they are all off).

-brady

10

bradymiller wrote on Saturday, August 27, 2011:

Yehster,
I haven’t noticed these types of errors in my javascript error log, but am not using firebug. Will try out firebug.
-brady

11

bradymiller wrote on Saturday, August 27, 2011:

Wow,
Firebug is cool. Here’s what I get for the patient summary screen:
POST http://192.168.1.143/openemr/interface/patient_file/summary/pnotes_fragment.php 200 OK 165ms
GET http://192.168.1.143/openemr/interface/patient_file/summary/pnotes_fragment.php 200 OK 598ms
GET http://192.168.1.143/openemr/interface/patient_file/summary/disc_fragment.php 200 OK 630ms
POST http://192.168.1.143/openemr/interface/patient_file/summary/clinical_reminders_fragment.php 200 OK 611ms
GET http://192.168.1.143/openemr/interface/patient_file/summary/patient_reminders_fragment.php 200 OK 677ms
GET http://192.168.1.143/openemr/interface/patient_file/summary/vitals_fragment.php 200 OK 701ms

Will await further testing, but am curious whether the ordering of the ajax calls has any effects (for example, would calling vitals_fragment before clinical_reminders_fragment stop all it to then finish before clinical_reminders_fragment)

-brady

12

bradymiller wrote on Saturday, August 27, 2011:

Error, first line above should be:
POST http://192.168.1.143/openemr/interface/patient_file/summary/stats.php 200 OK 165ms

13

drbowen wrote on Saturday, August 27, 2011:

Well the queries really do vary.  Even on the same patient on three separate trials.  Length of ottal wait is the same whether is a new patinet with only one visit or a patient with 100 visits over 7 years.

I started with a user that has only been seen one time today and has almost nothing in the record.

First trial:

Demographics
pnotes
patient reminders
Disclosure

1 -2 seconds later

Appointmnets
Medical problems
Allergies
Medications
Surgeries

after 40 seconds

Clinical Reminders
Vitals

Second trial same patient:

Immediate

Demogrphics
pnotes

second

Appointments
Medical Problems
Allergies
Medications
Immunizations
Prescrioptions
Vitals

40 seconds later
Clinical Reminders
Patient reminders

Third trial same patient:

Immediate

Demogrphics
pnotes

second

Appointments
Medical Problems
Allergies
Medications
Immunizations
Prescrioptions
Vitals

40 seconds later
Clinical Reminders
Patient reminders
Appointments
Medical Problems
Allergies
Medications
Immunizations
Prescrioptions
Vitals

The second patient has been coming for ten years and has exactly 100 visits since April, 2004.

First trial:

Demographics
pnotes
patient reminders
Disclosure
Vitals

40 seconds later

Clinical Reminders
Appointments

41 seconds:

Medical problems
Allergies
Medications
Surgeries

Second trial:

Third trial:

Immediate

Demographics
pnotes
Patient reminders
Disclosures
Vitals
Appointments
Medical Problems
Allergies
Medications
Immunizations
Prescrioptions
Vitals

40 seconds later
Clinical Reminders

I have not fooled with any patient reminders or Clinical Reminders.  These are still as they were during the upgrade.

Sam Bowen, MD

14

bradymiller wrote on Saturday, August 27, 2011:

Hi Sam,

Very helpful. Still unclear what’s going on. Also, you may of already said, but what web browser (and version and client OS) are you using.

To nail this down will be helpful to get the firebug output (this is a simple add-on to firefox) for these examples (see my above firebug output which states whether call was a success along with the time it took) and see what happens when turn off/on the CDR modules and also play around with the rules (one thing to try right now is to turn off all the diabetes rules in Admin->Alerts since I expect those queries for filtering the diagnosis titles/codes may bog things down).

Also wonder if updating to the latest jquery for the patient summary script may help in managing the ajax calls(now using jquery.1.3.2 and sounds like 1.5+ does better with managing multiple ajax calls).

Linked this issue to the 4.1 QA process; hopefully get this sorted out before release:
http://www.openmedsoftware.org/wiki/QA/Release_Process#Version_4.1

-brady

15

bradymiller wrote on Saturday, August 27, 2011:

Art,

This would be a nice thing to also test with your demo data creator (ie. create thousands of patient data data points and see if can recreate this issue). This would then allow other developers to work on this issue rather than having Sam’s group only work on it since their database contains sensitive information.

-brady

16

drbowen wrote on Saturday, August 27, 2011:

This is happening from CENTOS 5.6 (current) with Firefox, Ubuntu mostly 10.04 with Firefox, Windows 7 Professional Firefox.  My personal workstation is a Ubuntu 11.04 with traditional Gnome 3.0, Firefox 6.0.

I am not sure of all the Firefox versions.  I will be glad to explore some more tomorrow but this seems to be happening pretty much with all the clients that we are using.

You’ll might ought to shoot me instructions on how to setup firebug.

Gotta go home for the rest of the evening.

Sam Bowen, MD

17

yehster wrote on Sunday, August 28, 2011:

Dr. Bowen,
Firebug is a Firefox Add-on.  Just use the appropriate menu and search for it depending on the firefox version you have.

You can also read more about it here:
http://getfirebug.com/

-Kevin Yeh

18

drbowen wrote on Sunday, August 28, 2011:

I note this is not going to all that helpful to solve the problem but I am running 4-5 days behind in completing my work right now and had to do something to speed up progress.  I set about trying to turn off the new features that are causing us so much trouble.

Under Administration:

“rules” - no obvious way to turn these off.  This is where to change/customize the Clinical Decisions Rules" if needed.

“patient reminders”  - I don’t think that this is even working.  When I click on “patient reminders” I get a long delay then just a white screen.  The Patient menu bar is still active as is the tree navigation bar on the left.  I presume this will be where and how to set up the telephone reminders, sms messages and the like?

“Alerts” - Ahah!  clickable boxes to turn off “alerts.”  I turned off all the alerts.

Now the Demographics page is a 2-3 seconds slower than usual.  It still takes 2-3 seconds for the “Clinical Reminders” and the “Lists” to show up but at least we will be able to get some work down.

Gotta get caught up before burning more time on this.  I will be glad to turn on Firebug after I get caught up.  I think I will play with the alerts one at a time after hours so that it doesn’t disrupt the practice so much.

Sam Bowen, MD

19

bradymiller wrote on Sunday, August 28, 2011:

Hi Sam,

You can actually turn off CDR functionalities from within Administration->Globals->CDR

Recommend reading the ‘Clinical Decision Rules’ link in the User Manual section of the wiki for an overview:
http://www.openmedsoftware.org/wiki/CDR_User_Manual

-brady

20

verbus wrote on Friday, September 02, 2011:

All,

You can use one of our test configurations for 4.1 testing.

OpenEMR is running in a tandem configuration on an IBM z10 mainframe.

http://server1.emrtsolutions.com/openemr/
http://server2.emrtsolutions.com/openemr/

User/pass : admin/admin

Tandem meaning a change made to one copy is a change to both.

Our OpenEMR online course beta site is here:

http://emrtsolutions.com/learning    

Verbus@gmail.com