Direct Messaging support for testing

OpenEMR Development
1

lcmaas3 wrote on Friday, February 01, 2013:

I posted new code and patches to old code to add initial Direct Messaging send functionality to OpenEMR at https://github.com/lcmaas/openemr.git

The goal is to move towards meeting MU2 criteria 170.314(b)(2) and 170.314(e)(1).

The scripts will connect to a phiMail Direct messaging server and initiate transmission of a CCD to a specified recipient.  The CCD generated by the existing scripts is used, without an attached stylesheet.  Sending is available from inside Reports in the patient portal for patients, or from the Report view off the main patient info page for providers.

Successful submission to the server is logged.

To do: querying the server at regular intervals to receive status messages (such as final delivery or failure notifications) and eventually incoming Direct messages and CCDA’s for MU2 170.314(b)(1).

Anyone who wants to test out this messaging functionality will need login credentials for our sandbox server. Contact me for further information at <lcmaas@emrdirect.com>.

2

bradymiller wrote on Friday, February 01, 2013:

Hi Luis,

I placed a code review on github.

For a mechanism on querying the server at regular intervals, check out the getReminderCount() mechanism in interface/main/left_nav.php, which basically calls an ajax script on a regular basis. So, you could basically do something like this and place what you want to do in the ajax script. Of course, users will need to be logged in to do this and you may get over-pinged if, for example, several hundred users are on at the same time (so, at some point may want to only do it for admin users that are logged in or another group and/or control this feature further via globals). The nice thing about getting it in a nicely wrapped ajax script (ie. just call a function that does everything) is that it opens the option of running that same function via a cron script rather than relying on people logging in. The ajax script/function will be a nice building block, because then ZH can take it from there for how to store/process the ccds that get transferred in. And for logging success/failure would keep using the log as you are doing for now (which can be changed to another place.mechanism in the future, if needed).

Couple questions on the authentication. Will each provider need their own phimail name/password? Or does it cover the whole practice? And same question on the receiving end?

-brady
OpenEMR

3

bradymiller wrote on Friday, February 01, 2013:

Hi Luis,

Been thinking about how to regularly connect to your service via your regularly called ajax script/function with a structured mechanism in place to force the regular timing. My specific thoughts would be the following:
1. Make it a one way ajax request (ie. no need to get a reply)
2. Add session_write_close(); and set_time_limit(0); to the script (see library/ajax/execute_cdr_report.php for example and explanation of this) (that script is also a nice example of placing everything within a function that could be called by a scheduled commandline cron script, if desired)
3. Add a mysql table called something like service_connection with columns ‘service’(holds service token, such as phimail), ‘status’(holds pending or complete), and ‘date’(holds timestamp).
4. Would keep the mechanism very simple for now. It would:
a. First check the service_connection table for an entry of phimail that is pending OR (complete AND timestamp less than five minutes(or whatever time frequency you want)). If a token is returned, then would exit, otherwise would continue below.
b. Before connection is began, would place (or change) the phimail token to ‘pending’ and timestamp the date field.
c. Then the connection/work happens.
d. Then when finished, would change the phimail token to ‘complete’ and timestamp the date field.
(This mechanism would then only allow a certain connection frequency no matter how many users are connected to openemr and would be considered the poor man (ie. out of box) method. A cron commandline script could also be scheduled that runs your function for practices that want to ensure regular connections even when users are not logged in and can simply be run on top of the poor run method since the service_connection method will still limit to desired frequency)
5. For the ‘service_connection’ mechanism, suggest placing a service_connection.php script in the library directory that holds the standard function calls (which could be things like check_service(service_token,frequency), start_service(service_token) and complete_service(service_token). This would then be a very nice mechanism for other services in the future that will also need to be pinged regularly. At this point would only allow one token in the table per service, but at some point this could be expanded to add a new token every time to track the connections.

hope this makes sense,
-brady
OpenEMR

4

bradymiller wrote on Friday, February 01, 2013:

To clarify,
Note the check_service,start_service,complete_service are just to abstract out the process of dealing with service_connections mysql table. Your specific phimail service functionality would go into it’s own place and would call these functions to check if service needs to run, record the service is pending and record the service is complete,  respectively.
-brady

5

lcmaas3 wrote on Friday, February 01, 2013:

Brady,
I have incorporated your suggested changes.
I also added a Documentation/Direct_Messaging_README.txt with general information, and details on configuration and use.
Thanks also for all the ideas on adding regularly timed services; I’m going to look into these now.
-Luis

6

bradymiller wrote on Friday, February 01, 2013:

Hi Luis,

Just tested out your current code, which is working very well(at least on the OpenEMR side). This is ready to go into the official codebase as is. The options are:
1. I can just bring in your commits
2. I can rebase your commits into one commit (easier for patching to 4.1.1 then, if needed)
3. I can wait until you get the timed service stuff worked out (if you want to try to get the entire project into one commit, which is then easier for patching to 4.1.1).
(just let me know what you want to do and I’ll do that)

Regarding developer testing in your sandbox server, it may be best to have the developers working on the importing/processing of the CCDA to get involved, which is starting to get discussed in this thread:
http://sourceforge.net/projects/openemr/forums/forum/202506/topic/6707774

-brady
OpenEMR

7

lcmaas3 wrote on Friday, February 01, 2013:

Hi Brady,
Go ahead and bring in the changes I’ve made so far so folks can start familiarizing themselves with it. I’ve made the timed service calls a priority here and will push new code when we have something functional. I’ll take a look at the other thread, too, so we can coordinate receive functionality.
Luis

8

bradymiller wrote on Saturday, February 02, 2013:

Hi Luis,
I just merged your commits into the official codebase. Looking forward to what you come up with for the timed service calls.
thanks,
-brady
OpenEMR

9

lcmaas3 wrote on Monday, February 11, 2013:

Ok, round 2 of Direct Support is done. This task was quite involved. There are 4 major components:

1. Implementation of the actual message status and message receive functions of the phiMail API.  Straightforward, all within library/direct_message_check.inc.  OpenEMR can now receive status messages on previously sent CCDs and receive messages from any trusted Direct sender, including messages with CCDA, XML, JPG, PDF, or other attachments.

2. Extensible and robust mechanism for executing background services in a rate-limited manner.  Building from Brady’s idea, we currently piggy-back on the existing left_nav timer. This calls the manager script library/ajax/execute_background_services.php that iterates through a list of known background service function calls and executes them no more frequently than specified. We use the built in locking of tables in mysql to prevent race conditions. Care was taken to prevent the “is_running” flag from being left set if the background service function crashes or throws an exception. (The mechanism may need to be reset manually after a server failure or restart though.)  Other services can be easily added to this framework.

3. Storage mechanism for received attachments and non-text documents. We leverage the existing framework for Documents with some changes to the Controller class and C_Document class to extend the “upload” function so it will also accept documents received through Direct messaging.  Also, new messages (for now at least) are assigned to PID=0 but can later be assigned to the correct patient through the usual interfaces.  Would be nice if HTML files were as nicely integrated as PDFs and JPGs in the document viewer, since some Direct services use HTML.

4. Alert mechanism for received messages. Again, we have extended the pnotes features to allow the phiMail service to inject messages into the existing message framework. Currently these are routed to user “admin” and the patient is set to 0. The patient can be set (once) by the admin who can then forward the message to the correct party.  The message shows the details of the incoming Direct message, including any associated Documents. If the main message text was plaintext (text/plain) it is also included in the message, otherwise it is stored as a Document and referenced in the message.

Few other minor things, few new tables, should add a NoLogin Inactive user with username “phimail-service” for best operation. And, of course, one would need a Direct Messaging account to fully test this, which I can gladly set anyone up with (well, technically, our CA can only issue to US residents at this time.)

Comments and questions are welcome.

-Luis

10

bradymiller wrote on Thursday, February 14, 2013:

btw,
Here is the commit to Luis’s code:
http://github.com/lcmaas/openemr/commit/010b8aceb15389755274a18850236a041812a152
-brady
OpenEMR

11

bradymiller wrote on Thursday, February 14, 2013:

Hi Luis (and everybody else),

I placed a review on github. Pretty excited about these very promising features. Two main items here include the generic background service mechanism and then the phimail service(this plugs into the background service mechanism).
1. For the generic background service mechanism posted comments on github regarding code syntax and on incorporating everything within the background_service table. A report/gui to monitor/modify these would be really nice as discussed on github and is something I can tackle once the database schema is ironed out (if you don’t have the resources).
2. For the phimail service itself posted comments on github. Mainly, it will be dependent on the mechanisms that Z&H implements for uploading/storing/monitoring non-patient centric CCD/CCR/CCDAs, which is being discussed here:
http://sourceforge.net/projects/openemr/forums/forum/202506/topic/6707774
(so it makes sense to check in with them to see what mechanism they are using and can then piggyback on top of that)

btw, for everyone, here’s the MU2 item wiki page:
http://open-emr.org/wiki/index.php/Transitions_of_care_%E2%80%93_receive,_display,_and_incorporate_transition_of_care/referral_summaries_(MU2)

thanks and looking forward to the next revision,
-brady
OpenEMR

12

lcmaas3 wrote on Friday, February 15, 2013:

Hi all,

Thanks again to Brady for the thorough code review. We have moved all attributes of the background services to the database and made other changes as suggested. Hope to post revisions to github later today pending a few more tests. 

Also, FYI,  if we can get the code into the official code base in time, we will plan to take OpenEMR with phiMail integration to the official Direct Project community “connect-a-thon” next week.

Luis
EMR Direct

13

lcmaas3 wrote on Saturday, February 16, 2013:

Hi again,

The new revisions are posted to github: see https://github.com/lcmaas/openemr/commit/cde88931f3ec6a77d24a20256ba4325cd701cf15

Change summary: 
1. Most of the @ error suppression were removed; some are left in so as not to inadvertently reveal local potentially security sensitive settings (server names, file paths, etc.) on connection timeouts, i/o errors, etc.
2. All the background function attributes are now in the background_services table. This required adding supporting functions in interface/super/edit_globals.php to keep the background_services table up to date with globals changes. This should make it pretty easy to build a report interface for background services sometime in the future as Brady suggested.
3. Error alerts are now sent by regular email to the administrative notification address specified in the globals.
4. Direct message log now associates message with userID instead of username. Due to this, there are now two “service” no-login users (portal-user and phimail-service.) We fall back to user #1 if these service users are not defined.
5. All db->qstr references removed. We did have to leave the mysql_affected_rows() call in for now. The suggested alternative did not work.
6. Added a left_nav menu item to show unassigned “New Documents” under Miscellaneous to view the unassigned documents so the move to patient can be accessed.
7. Added site support to library/ajax/execute_background_services.php script, as well as a few optional CLI args / POST vars so that a specific service can be called if necessary (rather than checking all) and so that the delay between executions can be overridden.
8. added new entries to globals.inc so the execution interval can be changed and to specify the user who should receive the notification of new messages.
9. waiting to hear more input on where to route CCDA’s received as message attachments. They are now being registered as OpenEMR Documents (along with any other message attachments), so it would be straightforward to pass those document_id values to a processing queue at some point.

Luis
EMR Direct.

14

bradymiller wrote on Saturday, February 16, 2013:

Hi Luis,

I placed two reports in a new section entitled Reports->Service to show both the Background Services and the Direct Log(see the github link below). Very bare bones, but should have lots of room to grow as functionality is required/wished in the future. I’m still in the midst of reviewing/testing your code, so will post my review a bit later.

Here is the link to the reports code(note this branch was based on your working directmessaging branch and the report commit is the top commit):
http://github.com/bradymiller/openemr/commits/reports-added-to-directmessaging_1

-brady
OpenEMR

15

bradymiller wrote on Saturday, February 16, 2013:

Hi,

Just reviewed the code:
1. Should set the initial interval in globals to 5 so it is the same as what is initially in background_services
2. If I manually add a new document into the New Documents script, it is being imported (can see in directory documents/0, which makes sense), however I am then unable to see it in the New Documents script.
3. What are your thoughts on having an automatic flag in the background_services table. This then gives the option of only running a service manually (ie. via the force method). If automatic flag is disabled, this would allow only running the script manually in the Background Services report/gui or only running it at a specified time via cron.
4. My opinion is that this code is getting close to commit, so makes sense to place the tables and new data rows in the sql/database.sql

Otherwise, your code is beginning to grow on me. It took me awhile to get over placing service information within the users table (the community is becoming displeased with all the things that are stuffed within that table), but in the end I think it makes sense. What is a slowing things down here is that it’s not very easy to test this feature fully (I find myself making up database entries and changing around sql columns to try to mimic things); would definitely feel more comfortable if I could see the documents in the document viewer.

If anybody else is interested in this code (which seems to appropriately modify documents code(allows storing docs without patient mapping), pnotes codes (allows sending pnotes without patient mapping), and adds some entries to the users table, please check out the code in the above posted branch.

-brady
OpenEMR

16

lcmaas3 wrote on Saturday, February 16, 2013:

Hi Brady,

I will check out the reports and will change initial interval to 5 minutes.

Re #2: you are probably coming up against the issue that the Doc add function is entering NULL in documents.foreign_id instead of 0. This was initially a problem for us, too, hence the pid=‘00’ in the store function in direct_message_check.inc, as the Document class store function evaluates this to ‘0’ instead of NULL.

Re #3: Rather than adding another column, I propose that setting interval=0 means manual/cron only. I will modify execute_background_services to ignore services with interval=0.

Re #4: will do.

Re services in users table: I tried initially to get around this by using usernames instead of IDs but I do think your instinct to use the numeric IDs is a better one.

Re testing: I would be happy to set up a developer user account for you on the test server so you can get the “full feel” of the functionality and start sending and receiving test messages.

Luis
EMR Direct

17

bradymiller wrote on Saturday, February 16, 2013:

Hi Luis,

Re testing: yes, having a developer account would be very helpful (just send info via my email).

Re #2: Wonder if it will help to have the New Documents screen use ?patient_id=00 instead (my testing of this shows that at least the import and view is working here; places everything in a directory documents/00/ and the foreign_id gets a 0). Not sure if this will be compatible with the direct imports, but I can easily assess that with the developer account.

Re #3: That sounds good.

-brady
OpenEMR

18

lcmaas3 wrote on Saturday, February 16, 2013:

I changed the default interval to zero in the background_services table declaration.  I doubt 5 minutes is optimal for every service, so I expect each new service will declare its initial ‘optimal’ interval.  Otherwise, the default is now for a service will stay “manual only” until it is set up by the administrator.

19

lcmaas3 wrote on Saturday, February 16, 2013:

Re #2: we settled on /documents/direct/ as the place to put the attachments for now so that the source of the documents is clear, especially as the move to patient feature doesn’t move the actual file, only reassigns the foreign_id. Also, the actual file location doesn’t seem to matter to the list/view functions as long as the file path (document.url) is correct and foreign_id=0 (and not NULL). The ?patient_id=0 seems to work for us the same as ?patient_id=00 as long as foreign_id is right.

20

lcmaas3 wrote on Saturday, February 16, 2013:

code changes pushed.

https://github.com/lcmaas/openemr/commit/74099860fa04bdd8e02f295178fb6c5571da90de

Luis