GTOOL: I/O Library and Analysis Tool for Gridded Data

Eizi TOYODA^*1, Shin-Ichi Takehiro^*2, Masaki Ishiwatari^*3, and Yoshi-Yuki Hayashi^*4

Introduction

• As the capacity of computer and network grows, scientists face to need of more efficient handling of vast and multi-source data.
• Making common data format and sharing software will reduce efforts of such kind.
• The authors report on "gtool", a software environment for data handling in geofluid (e.g. atmosphere, ocean, or inner earth) simulation.

History

First Generation

• 1989: development of AGCM started by A. Numaguti and colleagues
• 1995: GTOOL3 documentation

Second Generation

• 1999: gtool4 project started
• 2001: latest version of gtool4 NetCDF Conventions
• 2003: latest version of gtool4 Fortran90 Tools/Library

GTOOL3 Overview

• GTOOL3 software = library + application tools
• written in FORTRAN 77
• Library/Application structure:

GTOOL3 Data Structure

File Format

• FORTRAN sequential unformatted file

  • good: portable program
  • good: can be splitted into many files (was important for magnetic tapes on mainframe)
  • NG: system-dependence

• File is a list of record pairs

  • "header": size 64 array of CHARACTER*16
  • "data": three-dimensional array

• Record pairs can be written in any order

  • sequential access always: dataset larger than memory can be handled

Header Record

• 64 named header fields
• Numeric value formatted - portable
• Library provides "query by name" from header

Unification of Data and Axes

• Three dimensions of data is not necessarily spatial (e.g. longitude, latitude, elevation)

  • domain knowledge abstracted

• Axes specified by external file reference

  • "axis name" header fields
  • "grid number range" header fields

• Axis file

  • file name derived (not just header value for portability) from axis name
  • same format as data file

Visualization Headers

• Control visual appearance of data/axis

  • The idea "the data file is the image file"

GTOOL3 Software Structure

Software Overview

• Applications: simulation models, pre/postprocessing, visualization
• 27,000 lines code, 376 subroutines (including entries)
• Modular design

module structure of GTOOL3 library.

Modular Coding Style

Adopted from DCL

• Namespace: first two letter of subroutine name shared

• Internal data:

  • in SAVEd variable, not in COMMON block

  • query by name

    SUBROUTINE GTPSET(HPARA, IPARA)
      CHARACTER HPARA*(*)
      INTEGER   IPARA, INDX
      INTEGER   INTRNL({number of parameters})
      SAVE      INTRNL
      INDX = {parameter index associated to HPARA}
      INTRNL(INDX) = IPARA
    ENTRY GTPGET(HPARA, IPARA)
      INDX = {parameter index associated to HPARA}
      IPARA = INTRNL(INDX)
    END

GTOOL3 Progress

• GHTOOL header field access: "query by name" from multiple instances
• GPTOOL math operators: abstract data type, destructive/non-destructive methods

Interface Design

GFTOOL File input/output

• Record pair access through GFOPEN/GFREAD/GFWRIT/GFCLSE

  • FORTRAN unit number allocated automatically
• Record pair search by one header field (usually TIME) value

GPTOOL: header-data pair as an data object

• Array operation

  • Elemental (array-array) or array-scalar arithmetic operations
  • Subarray extraction

• Wrapper to DCL array operation subroutines

  • No additional array size parameter needed: all parameters extracted from header

GGTOOL: header-data pair as an graphic object

• typically GGCNTR (draw contours) and GGAXES (draw axes) called
• No additional metadata needed: title, units, axis legend, and viz parameters extracted from header

Commandline Tool Support

• generic commandline argument parser: use namelist
• quasi-pipeline: input/output filename standardized to gtool.out
• abnormal exit status using abort()

gtool4 project

• Enable wider applications
• Improve GTOOL3 limitations
• Deploy new infrastructure

Generalization vs. Specialization

• Conflicting requirement

  • Generalize for wider application
  • Specialize for easier use or better performance
  • → Our task: "fill the growing gap"

• Layered interface design solution

  • Upper: specialized to application domain
  • (fill the gap with our domain knowledge)
  • Lower: generalized

• Slogan: "data file that can be visualized with a double-click"

  • That means: we firstly examined our everyday work on visualization

GTOOL3 Limitations

• File format portability

• Axis file management

  • axis file hard to create
  • data file meaningless without axis file

• Data structure extensibility

  • number of dimensions up to 3
  • room for additional or long attributes
  • domain knowledge dependency (e.g. difference between handling of time and spatial dimensions)

• Software construction

  • subroutine name readability
  • data access subroutine looks verbose
  • manipulation of visualization (such as composition of figures) not supported

New Infrastructure Employed

• Fortran 90

  • New: module, derived data type, generic procedure, ...

• NetCDF

  • Portable, extensible, and self-descriptive file format framework
  • Structure: variable, dimension, attribute
  • Access interface provided

overview of gtool4 software products

gtool4 Data Structure

Data structure defined by "gtool4 NetCDF Conventions"

• Mainly usage of NetCDF attribute

  • NetCDF attribute

    • Property of "variable" (or file)
    • Identified by name
    • Usage of name left to user conventions

• Data Representation Attributes

  • mainly based on NCAR CSM conventions

    • not to depend on GTOOL3 limitations (ex. dimension number)
  • GTOOL3 compatibility considered
• Visualization Attributes

Visualization Attributes

• Traditional Style

  • Visualization attributes: property of data variable
  • Used when the data visualized
  • Complicated figure (ex. two contours composition) cannot be described

• Structured Style

  • Based on object-oriented concepts

  • Structure variable and "has-a" relation between variables

    • "structure variable" has "data variable"

  • Enhance the idea "the data file is the image file"

    • original data of the figure can be extracted

schematic illustration of structure variable

Visualization Class Structure

• Visible components (contours, vectors, line, text)

  • Abstracted from DCL graphics interface
  • "shade" and "marker" withdrawn: implementational reasosns

visualization class in gtool4 NetCDF Conventions.

Compatibility to Existing Conventions

• Considered:

  • COARDS (1995)
  • NCAR CSM (Eaton, 1997)
  • GDT (Gregory et al., 1999)

• Import problem

  • Non-gtool4 data used for gtool4-compliant application
  • Most attributes: safely ignored if unknown
  • Non-negligible attr: effect harmful if ignored

• Export problem

  • gtool4 data used for existing application
  • New attributes: safely ignored if no name coincidence
  • Mandatory attr in existing application: should be mandatory also in gtool4

Technique to assure interoperability

• Attribute Namespace

  • Name of new attributes start with "gt_" to avoid coincidence

• Attribute Classification

  • Most attributes classified as "recommended"
  • Mandatory: Conventions, gt_version, history, long_name, title, units
  • Non-Negligible: scale_factor, valid_max, valid_min

gtool4/Fortran90 Implementation

Objectives

• Theoretical: verify above-described data structure
• Practical: construct software environment
• Biproduct: establish practical Fortran 90 coding style

Modular design

• Explicit module support of Fortran 90

• Layered design

  • Modules under error handler called "lower layer"
  • C-like utility functions: regular expression, error number, printf, string manipulation, system dependence

module structure of gtool4 library.

Data I/O Interface

• Example of layered design: specialized to generic

• First example: simulation model model output

  • gtool_history: specialized to history (time series) output of n-dimensional array
  • gtdata_generic: slice-scan API
  • gt_map: subarray or transposed view
  • an_generic: text-numeric conversion for attributes, implicit dimension create, and implicit file open
  • raw netCDF interface

gtool_history interface

Simpler, more intuitive, thus more specialized than GTOOL3

• header array no longer needed

• interface suitable to typical access pattern:

  • time incremented by HistorySetTime
  • variable specified at each HistoryPut(varname=) call

Visualization Interface

• Graphical object

  • load from/save to structural variable in NetCDF file

schematic illustration of structure variable

• GUI Frontend written in Visual Basic

  • Examine structure variables and visualize them on Windows

Screenshot of gtool4 GUI Frontend (left) and gtview contoring/shading result (right)

Practical Fortran 90 Coding Style

Object-oriented module

• class → derived type
• instance → variable
• method → subroutine

• static polymorphism → generic subroutine

  • subroutine OPEN for various argument types
• dynamic polymorphism → branching in subroutine (Decyk, 1997)

Header module

• Chained module reference makes some compiler impractically slow

  • This seems to be due to module procedure (subroutine)

• Solution: splitting them into external procedure

  • Similar to relation of header and library in C
  • Use integer handle as the derived type: move member data into inside module

Variable-length character string

• Data model difference between C/Fortran character string

  • Fortran CHARACTER

    • fixed length, no terminator character
    • trailing space ignored in comparison

  • C char *

    • variable length, terminated by NUL
    • trailing space significant in comparison
  • NetCDF data usually handled by program in C: potential confusion

• ISO/IEC 1539-2:1994 specifies the ISO_VARYING_STRING module

  • Memory leak in original version
  • Upper-limit version developed

• But ...

  • Most Fortran 90 compilers have problem around overriding of //, LEN or CHAR
  • Modified interface is used currently

Regular expression in Fortran

• Objective: order to parse units and time string
• Minimal implementation without or operator

Summray

• Self-descriptive data representation of GTOOL3 is successfully translated into gtool4 on NetCDF platform
• Limitations on portability, extensibility, and code readability successfully solved
• Data access interface improved (more intuitively)

• Awaiting work

  • Practice of new concepts, such as structure variables
  • Practice of interoperability
  • Application development

References

^*1Numerical Prediction Division, Japan Meteorological Agency
^*2Faculty of Sciences, Kyushu University
^*3Graduate School of Environment Earth Sciences, Hokkaido University
^*4Graduate School of Science, Hokkaido University