[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: I have the craziest idea...



tbowers wrote:
> 
> I've written a widget program that I've tried to design as a general data
> import interface. What I mean by general is that it's a wrapper widget to
> import various types of data and present it back to the user in some kind of
> standardized format for easy use. E.g., I have some 3D gridded data in ASCII
> text files of one format, 2D scattered data in ASCII text files of another
> format, etc. etc. What I did was write the widget as a function call so all
> I have to do is call it and I always have access to ALL data files that I
> know how to read. And, more importantly, the data is passed back to me in a
> format that's consistent and easy to process and visualize in IDL.
> 

Dear Todd,

thats not easy and might be impossible. May be this idea of a
datastructure
is useful for your job.

We have defined a general datastructure to handle all kinds of different
dataformats. It's specially used by instruments and models.

dataset normally having
1) a global description block
2) a parameter description block
3) the parameters itselfs one to n-dimensional

At the first level of the structure you can select between
the global attributes and the parameters.
The global tags are defined to start with a "!".

All others are parameter sub structures.

e.g. 
<img17.gif>

Each parameter has a tag "short_name" where the name is stored
to save the parameter in e.g. HDF or netCDF. So the
parameter tag name needn't be written the same as the short_name name.

The tag "param" holds the data. All other tags are parameter
attributes.
You can have as many parameters in this structure as you want.

All attribute-tags are defined in a definition file to get them correct 
written to a HDF or netCDF Format. An other reason is that's
all people are using the same attribute for the same thing.

more details in my publication:
http://www.fz-juelich.de/zb/text/publikation/juel3786.html

If I got a new format I have to set once a definition table or a filter.

If I have to write a filter I am using a routine to generate a
template.

write_icgspro,'template.pro',short=['time','value'],/small

see script at the end.

We are having a lot of filters from fileformats to this structure,
e.g. NASA Ames, HDF, netCDF and several other ASCii formats.

There are  many routines are available which uses this structure 
as input or output.

e.g.

; NAME:
;   icgs_timeshift
;
; PURPOSE:
;   This function computes the linear correlation between two time
series
;   as a function of a time shift between the two data sets.
;

; NAME:
;       icgs_time_slice
;
; PURPOSE:
;       This function returns a subset of an ICG structure wich is
defined by
;       a certain time interval.

; NAME:
;       icgs_time_filter
;
; PURPOSE:
;       This function returns an index into an ICG structure based on
certain time criteria
;

; NAME:
;   icgs_extract
;
; PURPOSE:
;   This procedure is used to extract parameters and their coordination
parameters
;   and the global settings from a given ICG Structure
;

; NAME:
;       icgs_correlate
;
; PURPOSE:
;       This function computes the Pearson linear correlation and linear
regression coefficients
;       for two parameters in an ICG structure.
;

; NAME:
;   icgs_concat
;
; PURPOSE:
;   This function returns the concatination of two icg_structures

; NAME:
;       icgs_calc
;
; PURPOSE:
;       This function evaluates an expression based on ICG structure
;       short names, math operators, and arbitrary function calls. The
result
;       may be stored as a new parameter in the ICG structure.


; NAME:
;   icg_ts_sync
;
; PURPOSE:
;   This function adds new time synchronized entries to the master
structure


; NAME:
;   icg_struct_idx
;
; PURPOSE:
;   The result of this function is a subset of the icg_struct by index.
Index could be used to sort ;   icg_struct param too.




pro template
   
   ; *********************************************
   ; CALLED BY:
   ; write_icgspro,'template.pro',short=['time','value'],/sma
   ; *********************************************
   
   extra= { $ 
      SHORT:['time',$
              'value'] ,$
      SMA:       1 $
          }
   
   struct=gen_icgs(_extra=extra)
   
   ;    *********************************************
   ;       FILL IN YOUR DATA
   ;    *********************************************
   
   *STRUCT.!GLOBAL.!DEF.FILE='\idl_source\idl_work\rb_lib\def_icg.pro'
   *STRUCT.!GLOBAL.!DEF.TIME='2001-01-18 11:24:52'
   *STRUCT.!GLOBAL.PI.NAME='UNDEFINED'
   *STRUCT.!GLOBAL.PI.ORGANISATION='UNDEFINED'
   *STRUCT.!GLOBAL.DATASET.TITLE='UNDEFINED'
   *STRUCT.!GLOBAL.DATASET.DATA_CATEGORY='UNDEFINED'
   *STRUCT.!GLOBAL.DATASET.EXPERIMENT='UNDEFINED'
   *STRUCT.!GLOBAL.DATASET.TYPE_OF_DATA='UNDEFINED'
   *STRUCT.!GLOBAL.PLATFORM.TYPE='UNDEFINED'
   *STRUCT.TIME.UNITS='UNDEFINED'
   *STRUCT.TIME.LONG_NAME='UNDEFINED'
   *STRUCT.TIME.FILL_VALUE='UNDEFINED'
   *STRUCT.TIME.FLAG='NONE'
   *STRUCT.TIME.PARAM='UNDEFINED'
   *STRUCT.TIME.SHORT_NAME='time'
   *STRUCT.VALUE.UNITS='UNDEFINED'
   *STRUCT.VALUE.LONG_NAME='UNDEFINED'
   *STRUCT.VALUE.FILL_VALUE='UNDEFINED'
   *STRUCT.VALUE.FLAG='NONE'
   *STRUCT.VALUE.PARAM='UNDEFINED'
   *STRUCT.VALUE.SHORT_NAME='value'


   ; *********************************************
   ; FILL IN YOUR COMMANDS
   ; e.g.
   ; struct=chk_struct(struct)  ; to remove UNDEFINED attributes
   ; write_ncdf,'file.nc',struct ; /ignore_test ; /remove_previous ;
/overwrite
   ; struct=ptr_struct2struct(struct,/free)
   ; *********************************************

   free_ptr_struct,struct

end


regards

Reimar

-- 
Reimar Bauer 

Institut fuer Stratosphaerische Chemie (ICG-1)
Forschungszentrum Juelich
email: R.Bauer@fz-juelich.de
http://www.fz-juelich.de/icg/icg1/
=============================================
a IDL library at ForschungsZentrum Jülich
http://www.fz-juelich.de/icg/icg1/idl_icglib/idl_lib_intro.html

http://www.fz-juelich.de/zb/text/publikation/juel3786.html

GIF image