[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

VARRAY, memory & extracting subarrays

I'm processing large (105 MB) arrays of images, and I've been running
into memory problems.  (Not surprisingly, right?)  I've started using
Eric Korpela's VARRAY routine, which has helped, but I still can't
manage to extract a subarray without using all of the available memory. 
Theoretically I have 1 GB of memory, and we've tried maximizing every
system variable that we can, but I'm still crashing ("Unable to allocate
memory to create array") when I try to run my image processing program. 

I can declare the main array and subarray using VARRAY, and I can read
images into the main array.   However, once I try to put anything from
main array in the subarray, e.g. subarray=main_array[i:j,k:l,*], or even
subarray=main_array, I start chewing up memory.  

Two not-very-good workarounds are:
 - using subarray=temporary(main_array), but then I lose main_array. I
suppose I could read the main array into main_array and then into some
temporary variable, so that I could do subarray=temporary(tmp_array),
but that involves more I/O
 - writing the subsection of main_array to a file, then reading it
directly into subarray - this is slow

I've attached sample code below, along with the results of
memory(/current) along the way.

Any assistance with VARRAY would be greatly appreciated.  


Kristine Hensel                              
Environmental Systems & Services      Phone: +61-3-9864-5300
405 Toorongo Rd                         FAX: +61-3-9822-8028    
Hawthorn East, VIC 3123 Australia    e-mail: kristine@esands.com

pro sector_noaa
  filename = '/home/meteor/mapped_data/200105171839N14.map'

  openr, lun, filename, /get_lun, error=err
  readu, lun, nlon, nlat, nchannel
  readu, lun, lons, lats
                                ; declare main images array:
  file_delete, '/home/meteor/mapped_data/images.dat', /quiet

*** Memory in use: 71 kB

  images = varray("/home/meteor/mapped_data/images.dat",byte(0), $

*** Memory in use: 71 kB
                                ; read in images array:
  readu, lun, images

*** Memory in use: 71 kB

  free_lun, lun

                                ; declare subarray:
  file_delete, '/home/meteor/mapped_data/sector_images.dat', /quiet

*** Memory in use: 70 kB

  sector_images = varray("/home/meteor/mapped_data/sector_images.dat", $

*** Memory in use: 70 kB
                                ; write out subarray:
  openw, lun, filename+'_sector', /get_lun, error=err
  writeu, lun, images[0:nlon-1,0:nlat-1,*]
  free_lun, lun
                                ; read in sector images:
  openr, lun, filename+'_sector', /get_lun, error=err
  readu, lun, sector_images
*** Memory in use: 70 kB
  free_lun, lun

                                ; try to copy directly:
  sector_images = images
*** Memory in use: 10531 kB