Re: Large File Plotting

["Mark Hadfield" <m.hadfield(at)niwa.cri.nz>]

"Mark Przeslawski" <MarkPrzeslawski@ncodeinc.com> wrote in message
st78offfa3gm3b@corp.supernews.com">news:st78offfa3gm3b@corp.supernews.com...
> Does anyone have an example of plotting
> very large time history data files. I tried
> creating a 1 million point file and it plotted
> okay. I tried a 10 million point file with
> various plotting routines and it hung my
> Windows 2000 machine (PIII, 700MHz,
> 256Mb RAM). I tried plot, live_plot,
> and some of the coyote plotting
> routines to no avail.

That's interesting. On my machine (not too different from yours: NT 4.0,
PII, 400 MHZ, 256 MB RAM) I find that if I can create the array I can plot
it:

IDL> plot, findgen(1000000)
IDL> plot, findgen(10000000)
IDL> plot, findgen(20000000)
IDL> plot, findgen(50000000)
% Unable to allocate memory: to make array.
% Execution halted at:  $MAIN$

Of course, speed is an issue: about 3 s per million floating-point values on
my machine.

> ... What is needed is
> something that can detect file size and
> only display to the pixel resolution of the
> screen.

That's pretty much what the Direct Graphics plotting commands do. They just
write pixels to the screen as they go; they don't keep a copy of the data or
the plotting primitives. This is why IDL Direct Graphics is so good for
plotting really large datasets, and why it is so frustratingly limited in
some ways.

When plotting such large datasets you should avoid Object Graphics
(including LIVE_PLOT) and also Direct Graphics wrapper systems (like David's
XWINDOW) that copy the data. You will also need to be careful about
processing the data to avoid redundant copies, e.g. this generates two
10000000-element arrays

a = findgen(10000000)
b = a

but this generates only one

a = findgen(10000000)
b = temporary(a)

People other than me are more competent to advise you on handling large
datasets.

---
Mark Hadfield
m.hadfield@niwa.cri.nz  http://katipo.niwa.cri.nz/~hadfield/
National Institute for Water and Atmospheric Research
PO Box 14-901, Wellington, New Zealand

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!