Submitted to the First Symposium on Integrated Observing Systems, 2-7 February 1997.
Paper number 1.15.
The author recently began asking colleagues how they define the term "data integration." Answers were mostly along the lines of "displaying data side by side" and "displaying two or more kinds of data overlaid on each other." But an integrated display is a small step toward true data integration, as the integration must still be done subjectively in someone's head.
In particular, the integration of geostationary satellite and WSR-88D data for quantitative applications, such as severe weather warning algorithms and forecasting techniques, is not a trivial problem. Each observing platform provides unique information, and has distinct time and space resolution, scanning strategies, and differing degrees of accessibility. The primary focus of this paper/ presentation is to define and discuss the importance of integrating satellite and radar data in the context of the complexities inherent in that effort. My goal, as well as that of my colleagues in the National Severe Storms Laboratory's Data Integration Team, is to approach the integration of geostationary satellite, WSR-88D, and other types of data specifically to alleviate severe weather diagnosis and short term prediction problems faced by the National Weather Service.
Satellite and radar data provide very different information about the character of precipitating clouds. Infrared sensors on geostationary satellites are passive radiometers which provide information about the infrared radiative and visible properties, such as the thermal emission and solar reflection, of the tops of clouds. Users generally think of satellite images as a snap-shot, even though they are built by scanning along an east/west parallel line, and are accustomed to looping imagery to see motion. Data quality control is done line by line at a centralized location as data is collected and then disseminated via the satellite as data collection continues. Users with ground stations receive data via the satellite itself, and others receive data via Internet- type connections. [Note: It was sometimes necessary for users to adjust navigation in early GOES-8 and GOES-9 imagery.]
The strengths of satellite data are clear, and justify the effort of worrying about the various caveats listed below. Geostationary satellite data provide nearly constant resolution within and beyond a mesoscale region, such as that of a National Weather Service Forecast Office (NWSFO) area of responsibility. In that same size region, WSR-88D reflectivity data resolution varies from under 1 km (beam diameter) within 50km of the radar to almost 4 km at 230 km. The second main strength of geostationary satellite data is that consistent cloud top height information may be inferred from the top-down look at clouds that satellite data provide, even for semi-transparent clouds (Nieman et al., 1993). WSR-88D hardware and software limitations do not allow forecasters to accurately sample the cloud height with radar. Volumetric sampling can cause uncertainties of over 3,000 meters in cloud echo top determination (Howard et al., 1996).
Caveats to consider when dealing with satellite data may include, depending on the wavelength and application: inability to see new storm growth due to optically thick high cloud, viewing angle, angle of solar incidence, and scanning methods. For severe storm diagnosis and short term prediction, very short time and space scales are important. The author suggests that uncertainties on the time scale of 5 minutes and space scale of 10 km as significant for severe storm diagnosis and prediction algorithms (i.e. see Shenk, 1974).
The radiometers on GOES-I/M are infrared radiometers (Menzel and Purdom, 1994). Since most clouds are opaque in the infrared, GOES-8 and GOES-9 cannot "see" below clouds. Satellite data may be quite useful in alerting forecasters of developing storms (most recently shown in Roberts, 1996), but will not always be able to give advance warning of developing storms since new storm growth could be obscured by higher cloud or anvils of other storms. This caveat is unavoidable due to the design of the instruments.
If precise earth location of high cloud is needed, satellite data must be adjusted for viewing angle. The parallax error, or apparent earth location, of a 15 km high storm over Norman, Oklahoma (35N, 97W), is 16.1 km approximately to the northwest if viewed by GOES-8 (75W), and 23.1 km approximately to the northeast if viewed by GOES-9 (135W). The closer to the satellite subpoint (i.e. 0N, 75W for GOES-8), the smaller the parallax error. When working to co-locate an individual storm (spatial scale of 5-10 km) on satellite and radar, the parallax error could cause mis-matching if not considered. [For the complete, unadulterated parallax calculation, see Weiss (1978).]
For precise time co-location of satellite and radar, the time of scans over a particular location must be known. In GOES data, the image is assigned the time of the beginning of the first scanline. Images take anywhere from 1 minute (super rapid scan sector) to over 26 minutes (full disk) to scan. Depending on location in the United States, the time differential between first scan and scan over a particular location can be as much as 5 to 8 minutes, meaning the data may be better correlated with a later volume scan. How best to approach time correlation of satellite and radar images may depend on the usage. For low altitude information, such as outflow boundary location, the satellite may be best time correlated with the first elevation scan of the WSR-88D. If, however, the cloud top is of interest, the problem becomes complicated, as several radar scans may intersect a storm anvil. In assessing the utility of satellite data in severe storm warnings, lead time is an issue and the time lag will greatly affect statistics.
Angle of incident solar reflection is important when the shortwave infrared (3.9 mm) channel of the GOES- I/M satellites is used. This channel is sensitive to reflected solar radiation, as well as to emitted thermal radiation (Scorer, 1989). The angle of solar incidence is also important when using shadows of clouds in visible imagery for cloud height determination.
Digital geostationary satellite data are not yet widely accessible to NWS forecasters. Most NWSFOs have access to at least reduced resolution images though SWIS or Micro-SWIS, but this does not allow forecasters to use the digital information in the imagery. A few forecast offices are fortunate to have NESDIS's Regional and Mesoscale Meteorology Branch (RAMM) Advanced Meteorological Satellite Demonstration and Interpretation System (Schrab et al., 1996), and receive high quality digital satellite data in near real time.
The author notes that satellite or integrated algorithm testing opportunities are few, as NWSFOs do not yet routinely receive satellite data in the full resolution and digital form necessary for algorithm use.
Whereas satellite data provide only a top down look at the infrared radiative properties of clouds, radar data contain information about the internal structure, such as the reflectivity and radial velocity, of hydrometeors in precipitation and clouds. Data are collected on-site or disseminated to secondary users via NIDS vendors. Primary users are accustomed to editing data, running quantitative algorithms, and displaying cross sections through storms derived from interpolated sweeps. Radar data are inherently complicated to interpret due to changing altitude and volume (i.e. resolution) with distance from the radar. With volumetric sampling around a radar site, the WSR-88D builds a three dimensional "picture" of hydrometeors in storms.
Caveats that must be considered when dealing with WSR-88D data include implications from fixed Volume Coverage Patterns (VCPs), range ambiguity, varying resolution with range, non-standard atmospheric refraction, and spurious effects from hydrometeor distribution and behavior.
WSR-88D data are gathered in spherical coordinates defined by particular scanning strategies, which were chosen to minimize tradeoffs between spatial and temporal resolution and meet the needs of the Department of Transportation (i.e. Federal Aviation Administration) and the Department of Defense, as well as those of the Department of Commerce, which includes the National Weather Service. Individual radars cannot be controlled beyond selection of particular predefined Volume Coverage Patterns. Forecasters are no longer able to use the radar to obtain vertical sweeps through storms, resulting in poor cloud top sampling, particularly in VCP 21 (Howard et al., 1996). Integration of cloud top height from geostationary satellites can alleviate this caveat.
Because of the beam geometry, azimuthal resolution and elevation vary drastically in relatively short distances; whereas the radial dimension of the observing volume remains constant with range. NWS forecasters and radar algorithms must take into account the varying implication of a return value of energy as a function of range. In addition, any particular radar sweep is displayed as though it and the earth were flat, when in fact the elevation above the ground is changing, sometimes radically, with range, and the earth is curving away.
Range ambiguity is also a problem unique to the radar observing platform. Because radar is an active radiometer, return of energy must be assigned a distance from the radar. When energy is returned after a second pulse is emitted, it is impossible, without a secondary source of information, to know if the return is from the first or second pulse emitted by the radar.
The hydrometeor distribution within a cloud can significantly affect the characteristics measured by a radar. In addition, the size, state, and shape also play a role. These can cause such problems as hail contamination and bright banding, and can then cause radar data to be misleading if not taken into account.
There are two challenges to doing data integration. The first is to actually do data integration: to realize how various combinations of disparate data combine to provide new and useful information. The second, and more immediate challenge, is to find software with which to do applied research and algorithm development. Most software available today was designed to view and manipulate either satellite or radar data, and was developed independently of considerations for the other. The result is that no software platform handles the attributes of both well. In addition, software developed for operations has a very different set of priorities and functionality than software required for research. When neither exists, new development must make a priority choice as to which kind of use to develop for (or develop first).
To the author's knowledge, no software packages, proprietary or otherwise, presently exist which contain a full suite of tools popularly utilized in both radar and satellite research. This is likely the result of the datasets being so disparate, as well as how each dataset is traditionally used. Thus, applied research has been mostly single sensor based.
The National Severe Storms Laboratory's Data Integration Team's (DIT) mission is to perform exploratory, applied research to determine whether various physical characteristics of severe weather environments revealed in the enhanced, multi-platform operational data stream have predictive value, and should be the subject of operational algorithm development. This will require sophisticated visual and statistical analysis of disparate data, including detailed research case studies; directed modelling studies; and a cutting-edge understanding of the underlying physical phenomena. DIT has spent the last several months addressing software requirements for truly integrated applied research.
After much discussion of the benefits, as well as the problems inherent in remapping data to common projections, the Data Integration Team concluded that we need to work with data in both native coordinates as well as common geometry. Remapping of data will destroy the original quality, and we must have the ability to assess the damage done by remapping. We also concluded that we need software with research tools to facilitate quick exploration of new ideas. Our requirements for exploratory software are very similar to, but go beyond, those articulated by Botts and Phillips (1996) regarding the development principles of the VisAnalysis Systems Technologies (VAST) software under development at the University of Alabama- Huntsville for NASA's Mission to Planet Earth.
The following abilities are essential: 1) all datasets must be accessed by time, 2) data must be stored and accessed in native coordinates, 3) remapping tools are required for display and manipulation of data on common coordinates, 4) tools for easy access to subsections of datasets, 5) tools for statistics, and 6) flexible graphing capability (such as for algorithm output). The emphasis of exploratory software must be on tools for easy interrogation of data.
To our knowledge, the above requirements are not met in any existing software package. As recognized above, other groups, such as NASA's Mission to Planet Earth, have also recognized the need for truly integrated software, and are investing in creating a much needed tool. NSSL is working toward this goal on a smaller scale by reworking the concept of the WDSS to be based on time (rather than radar volume scan) and to include a more complete set of tools.
In a series of meetings, therefore, the Data Integration Team not only defined our group requirements, but conducted a survey of current software (commercial, freeware, and in-house) . We identified three lists of potential support software. The first list is our current hypothesis of the best software for individual datastreams (includes more data than is covered by this paper; all software packages are identified in Section 8):
| Data Type | Software |
|---|---|
| satellite data | McIDAS |
| gridded model output, surface data, raobs, and profiler data | GEMPAK |
| lightning data | IDL or GEMPAK |
| WSR-88D data | HiRes, K-Edit, WDSS, and IRAS |
| any set of dense multi-dimensional data | Vis-5D, AVS, and NIH Image |
The following list is our hypothesis for best overall software for displaying data of multiple sources:
Though neither is complete, these two commercial packages compliment each other well to provide many intrinsic tools to assist exploratory applied research. In both cases, a minimal amount of effort will be necessary to add additional datasources by programming or obtaining shareware programs. The third and final is a list of software with necessary and valuable utilities that may not exist in the previously mentioned software packages:
| Software | Capability |
|---|---|
| REORDER/CEDRIC | user combined "primitives" |
| PV-WAVE | similar to IDL with statistics and graphing |
| CODIAC | to locate data sources |
| RDSS | to peruse and edit universal format radar data |
| a2io | library of Level II radar data reading routines |
| a2 | reformat Level II radar data to universal format |
The Data Integration Team will piece together these packages to do exploratory work. The resulting applications development ideas will be tested in available software as opportunities permit, with direct feedback from National Weather Service forecasters. Finally, we will make recommendations to the National Weather Service according to the usefulness of forecasting and diagnosis techniques developed and tested.
The National Severe Storms Laboratory has made preliminary steps toward integrating satellite data to the Warning Decision Support System (Eilts et al., 1996). Visible, infrared, and watervapor satellite data, along with a set of basic algorithms, were implemented in WDSS this past spring. Testing opportunities were few, due to limited availability of digital satellite data at current WDSS testing opportunities. Most testing was done in-house on case study data.
WDSS is now capable of displaying satellite and WSR- 88D data in a common projection (see Figure 1). Each type of data is displayed in a separate frame and can be linked together for common zoom and recentering. Other features of WDSS include map overlays of counties, towns, highways and the polar radar grid. Severe storm algorithm output, such as cell ids, cell tracks, and microburst, tornado, and mesocyclone detection symbols can also be overlaid, as can cloud- to-ground lightning strike and surface data. Severe storm analysis trends may be graphed. Cell, mesocyclone, and tornado tables list occurrences of such ranked according to severity.