Re: Huge datasets under Win2000

[david(at)dfanning.com (David Fanning)]

Joe Means writes:

>  have to read in over 13,000,000 observations [eventually over 
> 33,000,000] with the structure
> dataShort = {dataShort, returnNum:0L, x:0D,y:0D,z:0D}
> 
> At first, either when reading in or processing, I ran out of memory with 
> a message about IDL not being able to allocate memory to make the 
> array.  So I read this into an Assoc variable which works fine.  Now , 
> however, when I try to access parts of this data [dataS is the 
> Associated structure array]:
> datSvX = (dataS.y-yll)/cellsize
> I get the error: 
> % File expression not allowed in this context: DATAS.

Joe, sorry. I was distracted this morning what with
playing tennis and watching that Henman/Federer match
at Wimbledon. In fact, I don't know why in the world
I was even reading the IDL newsgroup! :-(

But I don't think this error has anything at all to
do with reading large data sizes. I think it has to 
do with the way you are accessing the associated
variable. (Although I admit I've never seen the
error message before.)

If I understand your message correctly, DATAS is the
associated variable. Is this right?

   dummy = {dataShort, returnNum:0L, x:0D,y:0D,z:0D}
   OpenU, lun, 'mydatafile.dat', /Get_Lun
   datas = Assoc(lun, {dataShort})

It is really incorrect to think of this as an
array of structures, which I think is what you may
be doing. In other words, this syntax doesn't work
with associated variables:

    b = datas(4:6)

With associated variables, you are going to have
to pull each structure out to work with it:

   struct1 = datas(5)
   struct1.x = 5.5
   datas(5) = struct1

I can't really tell what you are doing with this
syntax:

   datSvX = (dataS.y-yll)/cellsize

But, I'm pretty sure this is the source of your
error.

Cheers,

David

-- 
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155