[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Large File Plotting

"Mark Przeslawski" <MarkPrzeslawski@ncodeinc.com> wrote in message
> Does anyone have an example of plotting
> very large time history data files. I tried
> creating a 1 million point file and it plotted
> okay. I tried a 10 million point file with
> various plotting routines and it hung my
> Windows 2000 machine (PIII, 700MHz,
> 256Mb RAM). I tried plot, live_plot,
> and some of the coyote plotting
> routines to no avail.

That's interesting. On my machine (not too different from yours: NT 4.0,
PII, 400 MHZ, 256 MB RAM) I find that if I can create the array I can plot

IDL> plot, findgen(1000000)
IDL> plot, findgen(10000000)
IDL> plot, findgen(20000000)
IDL> plot, findgen(50000000)
% Unable to allocate memory: to make array.
% Execution halted at:  $MAIN$

Of course, speed is an issue: about 3 s per million floating-point values on
my machine.

> ... What is needed is
> something that can detect file size and
> only display to the pixel resolution of the
> screen.

That's pretty much what the Direct Graphics plotting commands do. They just
write pixels to the screen as they go; they don't keep a copy of the data or
the plotting primitives. This is why IDL Direct Graphics is so good for
plotting really large datasets, and why it is so frustratingly limited in
some ways.

When plotting such large datasets you should avoid Object Graphics
(including LIVE_PLOT) and also Direct Graphics wrapper systems (like David's
XWINDOW) that copy the data. You will also need to be careful about
processing the data to avoid redundant copies, e.g. this generates two
10000000-element arrays

a = findgen(10000000)
b = a

but this generates only one

a = findgen(10000000)
b = temporary(a)

People other than me are more competent to advise you on handling large

Mark Hadfield
m.hadfield@niwa.cri.nz  http://katipo.niwa.cri.nz/~hadfield/
National Institute for Water and Atmospheric Research
PO Box 14-901, Wellington, New Zealand

Hi! I'm a .signature virus! copy me into your .signature file to help me