We are getting serious issues with all our customers that have any data volume at all. All the report related to CQM, Patient Reminders, AMC - This example has 45,000. the error logs reported timeouts on access to log.inc called from cqm.php , so we disabled logging (to test) and the timeout moved to just cqm.php.
1. Ensure set_time_limit(0); is set for all the reports
2. Finish and implement yehsters solution for this in Reminders and Clinic Reports (AMC, CQM, etc): See post 45 from http://sourceforge.net/projects/openemr/forums/forum/202506/topic/4733322
3. Consider having the CDR sql queries bypass the audit engine
There are fudamental problems with how CQM, CDR,AMC and related components are implemented. Basically they lack the “join-fu” of well designed queries we have discussed before. Even with out the audit engine, each patient generates multiple individual queries for each “rule”. To solve this correctly is going to require significant changes.
I ran into the same problem last week when I tried to run AMC. The process stopped after 1 minute. When I searched in the forum, someone suggested to modify the cqm.php file by change the line 42 or 43 as follow:
<?php html_header_show(); set_time_limit(0)?>
Afterward, the process ran to the end but it took about 1/2 hour to finish for a database more than 4000 pt. However, I ran into another problem. I specified the date range for about 2 weeks, but the denominator for the "Provide clinical summaries for patients for each office visit. (AMC:170.304(h) ) was the sum of all the encounters in the past few years about 30,000. If denominator does not show correctly, we can forget about incentive money. Is there any way, can we correci this?, what files needs to be modified?
Using ajax won’t change the total time required to complete these operations. The performance scales linearly or worse with number of patients currently, so Tony’s 45K patients is likely to take 5 hours or more as compared to Bo’s 4K patients.
CQM/ACM needs to be something that can be run routinely to provide appropriate feedback to the users.
To truly be useful they need to be re-written using a few aggregate queries that let the database do the “heavy lifting” rather than bringing back data a few rows at a time and processing it in php. Done properly, I believe things could be sped up hundreds if not thousands of times from the current implementation.
For Tony’s purposes, it seems his short-term goal is to simply get the report working. My suggestions would take a day of coding at most and would make the reports work. Even if the report took 10 hours to build for large databases, it would still be much more useful than having no report. Your suggestion to completely refactor the CDR/AMC/CQM engine would take weeks (and possible several months); I’m not saying this is a bad thing, when developing the CDR engine, I figured in time (if OpenEMR became more successful) that newer generation CDR engines would be built/implemented. Probably best to leave the decision of optimizing vs. completely refactoring the engine to whomever plans to commit their resources to this.
Does the formatting of the input date/time from the demo look the same as the formatting of the date/time input fields in your instance?
Are you seeing any php errors?
Here’s the pertinent code for this denominator:
library/classes/rulesets/Amc/reports/AMC_304h/Denominator.php
library/classes/rulesets/Amc/reports/AMC_304h.php
library/classes/rulesets/Amc/library/AbstractAmcReport.php (see lines 173-178)
Here is the sql query I used to get the patient education number out of the database for the end of 2011, since that part of the AMC doesn’t work even if you keep it from timing out. Running this in phpmyadmin gives an instantaneous result and my database has about 8500 patients. I am a long way from understanding how you would do this, but couldn’t a set of queries like this be used to get numbers out of the database much more quickly?
SELECT Count(AM.pid) as CountPatients FROM `amc_misc_data` AM inner join `form_encounter` FE on FE.pid = AM.pid and FE.encounter=AM.map_id WHERE FE.date >= ‘2011-10-01’ and FE.date < ‘2012-01-01’ and AM.amc_id = ‘patient_edu_amc’
I’m unable to replicate the patient education number (AMC:170.302(m)) bug in the AMC report on the online demo: http://www.open-emr.org/wiki/index.php/OpenEMR_Demo
(both the denominator and numerator appear to be working fine in the demo)
What OS are you using?
Also, out of curiosity, how long did the AMC report take on your 8500 patient database?
I am guessing queries such as yours can be used for many of the AMC reports. They’ll become much more complicated for the CQM report. If a developer/contributor wants to take this on, then go for it. My opinion is that resources would be better spent on other projects and to instead spend minimal resources here to simply optimize the current engine as much as possible and to fix the bugs (again, though, this is simply my opinion here; the decision is really up to who does or funds the work).
Batch processing is appropriate for patient reminders because the results of previous batches don’t affect the results of subsequent batches However for AMC and CQM, the since the goal is an aggregation of data across all the batches there is strong interdependence. For the results to be trustworthy there would needed to be additional logic to verify results at each step. Plus what happens if someone else is making changes in the system while the results are being computed. Sure there are “policies” (like off hours analysis or data replication) that could address that, but it’s an important issue.
My opinion is that resources would be better spent on other projects and to instead spend minimal resources here to simply optimize the current engine as much as possible and to fix the bugs (again, though, this is simply my opinion here; the decision is really up to who does or funds the work).
I disagree here because being able to use these features efficiently would make getting Meaningful Use money much easier and therefore worthwhile.
I agree with both of you. I would be great if the thing actually works and run in the shortest time. However, for now as long as it works then I would be happy. As for my data base, I imported the data from my ubuntu to my windows xampp system and then ran the AMC. It took the same amount of time to finish (1/2 hour for 5000 patients) and specified date range seems to work fine. Something is fishy in my linux box. I will try it again this weekend with my linux box.
Just committed to sourceforge fix to remove the time limit on the report (plan to place this in the next 4.1.0 patch also).
Also Bo, it seems like there may be some sort of bug (perhaps related to php/mysql version or os) hiding somewhere since cverk also is reporting another bug from a different AMC rule. Please keep us posted if you find anything.
The cqm report runs for 25-30 minutes for a database of 500 patients.
Most time consuming rule is he adult_wt assessment one, which has an exclusion list go pregnancies and loops 1438 times for each patient. I can share the results of the time test I ran on 2 databases with around 500 patients.
Try out the CDR reports (CQM,AMC,etc.) in the most recent 4.1.1 patch (4), which just released today. Should be pleasantly surprised. Very curious what Tony’s 40,000 patient database will yield with this (I am crossing my fingers for 10 minutes to finish the report on the 40,000 patients). Please post time to complete your reports here, so we can see how things are going and assess where improvements need to be made.
