Hello,
I have done a small bit of looking at this project. It is interesting and I hope it gets running reliably. I was wondering if the directory path issue could be helped witht he php DIRECTORY_SEPARATOR constant?
The error at standard_tables_capture.inc:251 may be the line $file = $dir . ‘/’ . $file; which could become:
$file = $dir.DIRECTORY_SEPARATOR.$file;
The DIRECTORY_SEPARATOR is big and ugly, but I am wondering how important it is for cross platform compatibility.
Regards
What is very odd to me is that the ICD9 import takes 5 minutes (and burns up the cpu the entire time), while the much larger ICD10 tables take only 15 seconds. Any reason to explain this?
Brady,
I got to see why last night my new “tip of the tree” ICD9 was hanging…. nice catch. I think my old test branch must have had the index created as my tests never had that issue. I just verified ICD10 takes about 33 seconds to load on my Mac OS X MacBook Pro. I don’t have a Windows platform to test with, so waiting on Arnab (or anyone else) to check out your ideas on that PHP table issue.
How do you get the RxNorm and Snomed to run so fast. My old branch took FORever!!! I haven’t looked over those tables in your latest branch. Did you change anything for those loads?
Error: File ‘C:WindowsTEMP\ICD10\2012_I10gem.txt’ not found (Errcode: 2
)
Where does the “WindowsTEMP” part of the path come from?
In the /interface/globals.php the temporary files directory is:
$GLOBALS = rtrim(sys_get_temp_dir(),’/’);
The temporary directory in Windows is usually “C:Windows\Temp”, so I would expect the temporary directory to be “C:\Windows\Temp” in the $GLOBALS. My experience with Windows and Xampp is quite limited, so I don’t know about different configurations.
In the /interface/globals.php the temporary files directory is:
$GLOBALS = rtrim(sys_get_temp_dir(),’/’);
line 241: $dir_icd = $GLOBALS['temporary_files_dir'].DIRECTORY_SEPARATOR.$type.DIRECTORY_SEPARATOR;
line 314: $db_load = "LOAD DATA LOCAL INFILE '#INFILE#' INTO TABLE #TABLENAME# FIELDS TERMINATED BY '\0' (@var) SET revision = 0, ";
line 334: $run_sql = $db_load;
line 335: $run_sql = str_replace("#INFILE#", $dir_icd . $filename, $run_sql);
It looks to me like ‘C:WindowsTEMP\ICD10’ part of the path is from the $dir_icd variable. What does the php_info() function give for the temporary directory?
Mac,
I have a specific question that I’d like to clarify/answer before this code gets committed to sourceforge:
Why are the icd10_dx_order_code and icd10_pcs_order_code not using SERIAL in the index column like all the other tables?
And for everyone else, please feel free to review/test this code. This feature is going to add 12 new mysql tables, so suggest getting your input in now before it gets committed to sourceforge if have issues with this (or agree with this).
All,
Has anyone ever tested the previous implementation of the Snomed/RxNorm on Windows. I based all the new ICD stuff on the old code and only tested Mac OS X and Ubuntu implementations. I think the problem Arnab is having is a path construction issue. If you see his path name in the error message there are 2 issues:
C:\Windows\TEMP\/ICD10/2012_I10gem.txt
this one has mixed slashes, both Windows and unix types, which obviously is a bug. The second path name issue I see in the log is
C:WindowsTEMP\ICD10\2012_I10gem.txt'
where the leading backslash and the backslash between the first two tuples is also missing. Another defect.
My last commit was to just change the paths to the temp files where the incoming zips get unzipped and processed. Obvious issues remain. I think *all* the the path names in the code need to modified to use the DIRECTORY_SEPARATOR PHP constant as kevmccor had recommended in an earlier post.
Arnab - could you post the following items?
1. most recent error message to see if it is exactly the same
2. GLOBAL setting for the temp directory
Brady,
Those 2 ICD10 tables don’t use SERIAL keys because the raw data feed already contains a serialized sequence number in column one. I just imported those columns “as is” as dx_id and pcs_id accordingly.
I tested the upgrade for the ICD10 (easy to mimick by simply increasing the load_release_date of all the ICD10 entries) and the icd10_dx_order_code and icd10_pcs_order_code tables do not get upgraded (all the other ICD10 tables do). I suspect this is because are importing the dx_id and pcs_id columns as is rather than creating them as SERIAL elements. Note I am pretty certain I caused this by placing primary keys on dx_id and pcs_id. So, the question is what will be better:
1. Remove the primary keys from dx_id and pcs_id columns?
2. Or to make them SERIAL columns and don’t manually add them during the import (I like this option, since it is then consistent with all the other table loads, and if ever load data from other sources into these tables, then may run into problems if no manual index exists as it does in the CMS stuff)
Thoughts? If it makes sense to not import these ids and instead use SERIAL, then was hoping you could show the best way to modify the icd_import() function to do this.
Brady, agreed using SERIAL is the way to go and is the reason why the upgrade doesn’t work as all the ids already exist in the table from the prior load.
I just verified ICD10 takes about 33 seconds to load on my Mac OS X MacBook Pro. I don’t have a Windows platform to test with, so waiting on Arnab (or anyone else) to check out your ideas on that PHP table issue.