Re: Reading tab delimited string data (from MySQL)

[Andy Loughe <loughe(at)fsl.noaa.gov>]

Martin Schultz wrote:
> off the top of my head I can only think of using structures. You can
> dynamically create a template structure (just as you set up your
> column vectors) with Create_Struct, e.g. template =
> Create_Struct(id:0L, 'date','', 'height','', 'coverage','', $
>              'yy',0L, 'yn',0L, 'ny',0L, 'nn',0L )
> 
> Then you have two options:
> (1) if you don't know the number of rows coming out of your query
> (well, this applies more to results stored in a file), you can either
> use a linecount program and use approach 2 afterwards, or you still
> loop line by line and use a "block data" technique to enlarge your
> structure array by e.g. 100 elements at a time.
> 
> (2) if you know how many lines to expect, you should be able to "read"
> your data as
>     result = replicate(template, 5)
>     reads, data, result

Martin,

Actually, this works if I use "reads" with a format statement, 
but I also had to issue the mysql call so that it returns the dump
in "table" format (spawn, 'mysql -t sql_script', data), rather than in
TAB delimited format.  The strlen() of each record was different for 
the TAB delimited records, but identically the same for table formatted
output (which is "|" delimited).

Thanks for the great nudge in the right direction!
"reads" with a data structure is certainly the way to go.
I read everything in as a string, and specify through a keyword
which variables need to be converted to long or float on output.

The 66980 records with 10 variables per record are now read into IDL
in only 15 seconds, versus the 39 seconds of my first attempt.  
424,000 records with 10 variables per record (10 columns or fields 
in the database) are read into IDL in only 104 seconds.  Not too bad.

And now I do not have to purchase Dataminer and attempt to get it
to work on my Linux box with MySQL!

-- 
Andrew Loughe =====================================================
NOAA/OAR/FSL/AD   R/FS5  |  email: loughe@fsl.noaa.gov
325 Broadway             |  wwweb: www-ad.fsl.noaa.gov/users/loughe
Boulder, CO  80305-3328  |  phone: 303-497-6211   fax: 303-497-6301