[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Large data sets
- Subject: Re: Large data sets
- From: "Liam E. Gumley" <Liam.Gumley(at)ssec.wisc.edu>
- Date: Fri, 04 May 2001 08:47:27 -0500
- Newsgroups: comp.lang.idl-pvwave
- Organization: Space Science and Engineering Center, UW-Madison
- References: <9ctvb7$hf1$1@news.kodak.com>
- Xref: news.doit.wisc.edu comp.lang.idl-pvwave:24804
Neil Talsania wrote:
> I am new to IDL, and everything that I have done so far works great on
> small data sets. But, I dont know how to use it for a large (>2 gig)
> dataset, as I obviously just cant load that into an array. The data is on
> disk and can either be tiled so that it is stored in 1024x1024 segments, or
> it is stripped which is one complete row at a time. Even a complete row is
> large (>80000 samples).
>
> Any thoughts as to how I would be able to use this data?
>
> Another question: is there thread support in IDL? I have a machine with 18
> processors, so it sure would be nice to process this large dataset in 18
> parallel threads. Any suggestions?
The following may be of interest:
http://sag-www.ssl.berkeley.edu/~korpela/mmap/
Cheers,
Liam.
http://cimss.ssec.wisc.edu/~gumley/