Re: I have the craziest idea...

["tbowers" <tbowers(at)nrlssc.navy.mil>]

> Excellent idea, but from glancing over it, I am not sure that the
> structure is general and flexible enough. I started with a similar

That's exactly why I begged for comments. I knew special types like
netCDF and HDF would be different, but I've never worked with
either and I need input from people who know about those types.

> Unfortunately I haven't gotten too far yet (because people don't let
> me spend so much time on IDL programming here), but I do have a

I know the feeling. I'm doing this on my own time. Just thought it'd
be very useful to us all.

> A few more general issues in terms of such a project: If you really
> want a tool that almost everyone on this newsgroup can use, you need
> to make sure that you can access portions of the data in the file upon
> request. For example, here we often deal with model output on the
> order of 2GB (and it could be larger were there not the Linux
> filesystem limitations), and you certainly don't want to read the
> complete file before starting to work with these data. Now, this
> requires a program structure where you analyse the headers first, let
> the user select what he/she wants to read, and then go out and grab

Yes. I have a similar situation. I use a text file format which has
h lines of header followed by n columns of scattered data where the columns
have1 or more of position/time data (eg. Lat Lon depth) and then 1 to 100's
of columns of parameter data (temp, salinity, etc.) but you never know
which column contains what because there's no standard, depth could
be in column 1 or 21 or 101. So, I quickly wrote a little front end widget
that first presents to the user cw_field()'s that prompt for the data to
extract.
Similar to:

Number of Header Lines: ___
X Axis Data Column: ___
Y Axis Data Column: ___
Z Axis Data Column: ___
T Axis Data Column: ___
Variable Data Column: ___
  _____________
  |Accept | Cancel|
  ============

The user types in what he wants to extract, clicks Accept button, and
that data is read in. The dimensionality of the data is determined by
how many X,Y,Z,T fields the user filled out.

> ... glancing over this again, I must be out of my mind to give you
> such a hard time when you are proposing such a good thing. I guess, at

Absolutely not. Honest constructive criticism (with helpful suggestions,
of course ;) ) is what I'm looking for. Thanks. I'm kinda surprised
more people don't have an opinion on this.

Since I'm a netCDF/HDF moron, I don't think I could easily decide how
to handle/integrate your data reader. It may be easy, but I don't know.
How about I send you my code and you send me yours. I'll look at it this
weekend. Maybe together we can determine how easy/difficult this will
be.

I also don't want to discourage others. It's easy to see that netCDF/HDF
are special cases. I'm sure there are alot of file formats that could very
easily integrate into this code (e.g. FITS, Surfer grid, Sigma Plot grid,
medical image formats, etc.) formats which I have no experience with but
others do. Also, not all netCDF/HDF files are so general. E.g., SeaWiFs
satellite imagery comes in an HDF file for each image. This, it seems, could
be returned rather easily as a standard 2D image type.

Thanks Martin. Your comments and advice is greatly appreciated.
Todd