[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Data Validation

majewski wrote:
> Hi
> Is it possible to speed up this process...
> I have a system of instruments which feed their output into a data taker (15
> second interval, for ~a week at a time). The data taker outputs a comma
> delimited ascii file which I then process in IDL.
> Occasionally the data taker, due to the harsh conditions it's under (in the
> hull of a boat), drops characters - rendering the line corrupt.
> I am trying to ensure that the data passed to the processing routine is
> error free.
> Currently my system does a str_sep(data_string, ',') to determine the number
> of terms on a line. if its too few or too many (#<14<#) the line is dropped.
> I then have an error check of each of the terms to see if they are floats
> (and also in the valid ranges).
> onerror, readnext
>     convert to float, etc
> readnext: ;-> returns to start of loop
> anyway if it is possible to speed up this process (or I've gone about it the
> wrong way) I'd like to know
> leon Majewski

You could read in a line as a string first, then use reads to convert it
to floats. Provided with proper error checking, this may be faster than
str_sep. Here is a "demo":

function parse_line,line,data,nval

;  line is the input string, data the returned float vector with nval
; return value of the function is either 1 (successs) or 0 (failure)
   catch, theError    ; regards to David ;-)
   if theError ne 0 then begin
     catch, /Cancel
     Message,' Error parsing data line ',/Continue
     return, 0

   data = fltarr(nval)
   reads, line, data
   return, 1

pro readdata, filename, data

   ; this is the main routine
   ; you should actually have another catch handler here ...

   nval = 10    ; number of values on a data line

   if n_elements(filename) eq 0 then begin
      Message,' Usage: readdata, filename, data'

   openr, ilun, filename, /GET_LUN

   count = 0L   ; make sure this is a long!
   while not eof(ilun) do begin
      line = ''
      if not parse_line(line,dataline,nval) then begin
         ok = 0
         ; here you can call a routine for error correction
         ; which then uses str_sep for example and sets ok to 1
         ; if the error could be corrected
      endif else ok = 1
      if ok then begin
         if count eq 0 then data = transpose(dataline) $
         else data = [ data, transpose(dataline) ]
      count = count + 1
   free_lun, ilun

Good luck,

[[ Dr. Martin Schultz   Max-Planck-Institut fuer Meteorologie    [[
[[                      Bundesstr. 55, 20146 Hamburg             [[
[[                      phone: +49 40 41173-308                  [[
[[                      fax:   +49 40 41173-298                  [[
[[ martin.schultz@dkrz.de                                        [[