Importing CSV "Invalid parameter

keithlofstrom wrote on Friday, March 09, 2012:

Thanks for all the postings about CSV patient data import so far … but the suggestions are either incomplete, I’m missing something, I’m misconfigured, … etc.

When I try to import, I get an
.  .  . “Invalid parameter for CSV import: Fields enclosed by” . . .
error, as shown, with no indication of what it thinks the fields are enclosed by.

I am using OpenEMR 4.1 .

I want to import a list of 1143 patients - I have 10 parameters to go into the patient_data table (insurance companies and patient insurance will be next).  I use linux, and Perl for manipulating text.  I did the suggested creation of a few example patients (George Jetson, Fname Lname, etc.) with different fields included and omitted to see what gets toggled, then exporting the CSV and examining it.  Most of the 83 fields are blank, some are my data, some are dates and fields and a few NULLs added by openemr somewhere.

Here are two example exported lines of CSV data, default settings for the export:

“7”;“Mr.”;“SpaceSpeak”;;“George”;“Jetson”;“J”;“2091-11-21”;“ContAddr”;“99961”;“ContCity”;“OS”;“USA”;“987654321”;“999-99-9999”;“Occupation”;;;;;“22”;“married”;“EmergencyContact”;“2012-03-08 13:22:00”;“Male”;;;“2”;“ContactEmail”;;“amer_ind_or_alaska_native”;“not_hisp_or_latin”;“Interpreter Fred”;“Not Migrant”;“147”;“789.01”;“Not Homeless”;“2444-04-14 00:00:00”;“7”;“7”;“User Def 1”;“User Def 2”;“User Def 3”;“User Def 4”;“NO”;“YES”;“YES”;;“NO”;“YES”;;“0”;“Patient”;;;;;;;;;;;;;;;;“standard”;NULL;NULL;“NO”;NULL;“eligible”;“MothersName”;“Guardian”;“YES”;“NO”;“NO”;“NO”;“0000-00-00 00:00:00”;“Not Dead Yet”;NULL
“8”;;;;“Ftest”;“LTest”;“Mtest”;“2043-12-01”;“Street Address”;“99991”;“Orbit City”;“OS”;“USA”;;“987-65-4321”;;;;;;“0”;;;“2012-03-09 08:59:23”;“Male”;;;“2”;;;;;;;;;;“0000-00-00 00:00:00”;“8”;“8”;;;;;;;;;;;;“0”;;;;;;;;;;;;;;;;;“standard”;NULL;NULL;“NO”;NULL;;;;;;;;“0000-00-00 00:00:00”;;NULL

The second line resembles the data I have to feed in.  I don’t have S.S. for everyone, and some of the zip codes are zip+4.
I’ve frobbed this a few dozen times, different delimiters or none, filling in the blank fields with “” , removing the \. and trying that.  The data is pretty well scrubbed.  So I assume I am missing something basic.

I could start over and learn about SQL and format my input records that way, I suppose, but I prefer to make CSV work, then document the heck out of the process on the wiki for future users who aren’t Perl coders. 

OpenEMR needs a book.  I’ve sold articles to software magazines; perhaps when I finish the book I am writing now, I can help write the OpenEMR book.  A lot more setup, debug, and maintenance information must be made explicit and accessible to the average IT geek, perhaps (horrors) the average small practice doctor.

tmccormi wrote on Friday, March 09, 2012:

Keith are you using phyMyAdmin or command line mysql?
-Tony

tmccormi wrote on Friday, March 09, 2012:

Also field 0 is an internal auto-increment field, it should be empty or NULL in your import.  Use PID and PUBID for preassigned id#'s.  After you do that then the error changes to “Invalid field count in CSV input on line 1.”  Number of columns must match exactly.

-Tony

tmccormi wrote on Friday, March 09, 2012:

oh … and I stripped all the backslashes from in front of the quotes and used phpMyadmin for the import, just so you know.
-Tony

keithlofstrom wrote on Saturday, March 10, 2012:

No joy.

I have been using phpMyAdmin - 2.11.10  (via the web interface in OpenEMR) - the blue list of tables, the Import tab.

As you suggested stripped off the forward slash, leaving the quotes, and emptied the first id field, while still providing incrementing PID and PUBID.   This is one of the many frobs I tried before.  No luck.

I still get the same error from phpMyadmin,  “Invalid parameter for CSV import: Fields enclosed by” .  What does that mean?  Error messages usually mean something.  Is there a way to turn on verbose logging?  Where?

I am using version OpenEMR v4.1.0 (6) .  I may have configured it wrong (is there a script that tests this?).

Here’s the CVS import parameters I used:

  Partial import                  0
  Format of imported file         CSV
  Replace table data with file   unchecked
  Ignore duplicate rows           unchecked
  Fields terminated by            ;
  Fields enclosed by              "
  Fields escaped by               (empty, I also tried \)
  Lines terminated by             AUTO
  Column names                    blank

Note:  I did a SQL dump and looked at that;  I can write a perl script to produce data records like that, if I know how much of the preamble to include.

keithlofstrom wrote on Saturday, March 10, 2012:

BTW, I counted 82 semicolons as field separators between 83 fields.  I made sure I matched that on every line.   No terminal semicolon.

One of my frobs was trying a terminal semicolon. no luck. Others were NULLS for all blank fields, no luck, and “” for all blank fields, no luck.   Same useless error message (useless in the sense that it does not help locate the actual error). 

I tried editing the CSV export, removing , blanking out ID, replacing PID and PUBID, and importing it again.  Same message.

blankev wrote on Saturday, March 10, 2012:

Did you clear the table content before uploading the CSV file?

When you exported the file to see if it worked, did you try to import the same file? Because if that works, there is something wrong with your process of creating the CSV file.

Clear the table, DO NOT delete the table!

I done it this way several times and it should work! But might takes some fine tuning before everything can be read by the computers on both sides and the OpenEMRhangs there just in between, and that is frustrating.  Remember to save as xxxxxx.csv  MS-Excell sometimes put some other saving-automatism in the filename.

keithlofstrom wrote on Monday, March 12, 2012:

Oh, my.

The table must be cleared before uploading?  How do you append more data (from another source) into it?  I am not at all interested in saving 1000 records, adding to it off line, deleting it, and reloading the bigger table.  That sounds like a formula for show-stopper failure, especially if this is done during work hours in an operating clinic.

Is there another way to get patient data in there, besides wiping the table or using the video game///////////GUI interface? 

blankev wrote on Monday, March 12, 2012:

In any database there are keys and these keys can be incremental or at random. What are you trying to invent?

OpenEMR is not a gaming program and playing with it on site can only be done with intention of working as a serious business program    ;-))

Be sure to put you computer on serious business and not on Gaming!!!!!! THAT MIGHT HELP.

Pimm

tmccormi wrote on Monday, March 12, 2012:

Data import from CSV is always tricky.  It’s hard to get the data order and column counts right when juggling that many fields.  You do NOT have to empty the table, but you must not have a value in column 0 (ID) as that is autoincrement.   AND you can’t have duplicate ID# in PID field.

It is, in fact better, to write a mapping script that reads the source data and writes to the mySQL DB using SQL statements.   I tend to do something very similar to the icd9 and cpt import scripts that can be found in the contrib/utils dir (without the web screen scraping parts).   That is, read in CSV write out SQL and import the result after checking it.  That way you can ignore field order (map by field name) and column counts and only load the required data.   Take a look at the format of the sample patient data in the sql/ dir.

Tony

keithlofstrom wrote on Monday, March 12, 2012:

First, apology to blankev.   The part about “video game” (followed by ////// representing a botched deletion) is an American sarcasm, about using GUI interfaces to do tasks that are better handled on the command line, without using a mouse to open windows and move cursors.    Some tasks are much better handled with automation, and text tools are easier to automate.    For example, I would much rather write this message with vi than in a Sourceforge text box.   GUI interfaces are reminiscent of video games, which hide what the computer is doing under layers of flashy distracting animation.  “Lipstick on a pig” is another sarcastic Americanism; like a GUI, it makes the pig prettier, but you still don’t want to kiss it. :frowning:

Second, a big thanks to Tony.  The Perl scripts you suggest tell me almost all I need to know.  The rest I can easily figure out.  I will start with something like those scripts.  I will add a test mode, which can dump the incoming data lines formatted as verbose multiline “cards”, and also look at the database and for potential overwrites.   Power tools are quick, but can make a big mess in a hurry.  When I have something functional, and get the data imported, I will share my own Perl scripts with Tony, who can decide whether to add them to contrib/utils, or plead with me to delete them and never use them again. :-/

While I am curious as to why I got the original message that I started this thread with, I would just as soon create a series of command line procedures and perl scripts to import the data, usable by a capable sysadmin but not tempting to the average GUI user. 

Keith

blankev wrote on Tuesday, March 13, 2012:

No apology needed! I just can’t figure why you have so much problems with the CSV upload. Me a complete stranger to Perl script uses the CSV mode and had after initial start no problems what so ever and the suggestion Tony was used several time with great succes…… Don’t know why you have the same problem over and over again. I uploaded more than 3000 lines of CSV adding to a list of 1500 in one of the past OpenEMR versions….

Once your pearl script is working, you might tell me the reason why your CSV upload didn’t work…. keep smiling ;-)) the solution is around the corner, the question remains what corner.

keithlofstrom wrote on Tuesday, March 13, 2012:

Found the error,  GREP -R IS MY FRIEND.  It is in the file openemr-4.1.0/phpmyadmin/libraries/import/csv.php, line 65:
    
   } elseif (strlen($csv_escaped) != 1) {

$csv_escaped is either length 0 (no contents) or 2 ( \\ ) which is “escaped escaped” and how it gets passed the escape character from the GUI (which shows only a single escape).  I edited the php file and changed the number to zero.  I took the quotes out of my csv file.   I called the import with parameters   (no quotes)   and imported 1000 records successfully.  We have moved from dead in the water to merely limping.  YAY!

I expect that " will get the same treatment, and turned into  and generate an error in the same block of code, if I try to use quotes.

I still have more records to import, and will frob up something in Perl for future imports.  I hate tweaking code in this ignorant way.  But someone more skilled with PHP than I (that is about 6 billion people) should find out what is really happening and fix it properly.    And PUT IN A USEFUL ERROR MESSAGE, dammit.

Perhaps I just have a bad version (2.11.10) of phpmyadmin.  That is what comes with my linux distro ( an RHEL6 clone ).

keithlofstrom wrote on Tuesday, March 13, 2012:

Gads, I wish this dumb sourceforge forum GUI would permit me to preview.

… I edited the csv.php file and changed the number to 2 (not zero) …