Re: point_lun is slow

[Craig Markwardt <craigmnet(at)cow.physics.wisc.edu>]

George McCabe <george.mccabe@gsfc.nasa.gov> writes:
> 
> reading from a data file at regularly spaced byte locations, 2 bytes at
> a time using point_lun - my program is abnormally slow.  i don't have
> enough experience to guess whether the poor performance is inherent to
> point_lun & readu approach or if there are options which are affecting
> execution adversely.  
> 

As David mentions, you don't want to be doing lots of POINT_LUN calls
inside a large loop.  I have found that one of best ways to do sparse
reads is to read a large chunk of data into memory, and then operate
on it from there.  Who cares if you read a little too much data.
That's really what the underlying operating system has to do anyway,
but you can avoid doing lots of small READU calls.

buflen = 65536L
buffer = bytarr(buflen)
readu, unit, buffer, transfer_count=cc

You will have to do error checking here if cc is less than buflen.
You need to decide whether the entire file can fit in memory at once,
or if you need to do it in chunks.  

At this point, you have a big chunk of memory and can operate on it
without doing any more reads.  For example, if you want every other
byte, you could do something like this:

buffer = reform(buffer, 2, cc/2, /overwrite)  ;; This is fast!
result = buffer(0, *)                         ;; Gets the first of two bytes

By the way, this also answers the question of the fastest way to get
ever other element of an array.

If you need to walk some more complicated data structure, that's
harder.  You probably won't be able to do that without a FOR loop.
Hopefully your data file structure has a natural block size, and you
can read a whole number of blocks at once.

I do chunking like this all the time, and get excellent boosts in
performance.

Craig

P.S.  I'm in Building 2 at Goddard.  Stop by if you want. :-)

-- 
--------------------------------------------------------------------------
Craig B. Markwardt, Ph.D.         EMAIL:    craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
--------------------------------------------------------------------------