wiki:WorldWindChallenge/2014

Context Navigation

In short
In depth
1. Datasets
2. Demo Features
  1. Area and Time of interest: data access and analysis
  2. AOT cross-comparison: WCPS in action
Credits

NASA WORLD WIND Europa Challenge 2014

Our team (in collaboration with MEEO Srl has accepted the World Wind (WW) Europa challenge 2014 with the following project: "Cross-coverage atmosphere data analytics with the WCPS query language".

Official website of the project: http://worldwind.flanche.net/
Source code: git clone http://git.flanche.net/vlad/wcpscrosscovanalysis.git
L-SIS internal project wiki: http://kahlua.eecs.jacobs-university.de/trac/wiki/wiki/WorldWindChallenge/2014

In short

This demo will experiment with support and performance of pixel data fusion done at server levels, leveraging the OGC Web Coverage Service (WCS) processing extension and its relevant operators —- a.k.a. the Web Coverage Processing Service (WCPS) grammar.

We investigate by implementing examples of well-known functions and parameters that are extracted from datasets for environmental analysis (aerosol optical properties). The datasets used in the experiments are gridded rectified products, which are best suited for the open source implementation of WCPS used.

All information shown during the demo is entirely retrieved from a WCPS server. Our chosen implementation is provided by the open-source rasdaman array DBMS (http://rasdaman.org). WCPS service endpoint is provided by MEEO Srl (http://www.meeo.it/).

In depth

The new paradigm introduced by this deluge of data that is hitting us every day from satellite and air-borne sensors is changing the workflow of environmental data analysts and modellers. Web geo-services play now a fundamental role, and no longer they are needed to preliminary offline data download and storage but rather they interact in real-time with GIS applications.

Due to the very Big amount of data that is curated and made available by web services, it is crucial to deploy smart solutions for i) optimizing network bandwidth by enabling download of high-entropy data to the user ii) reduce duplication of data that can be otherwise efficiently computed on-the-fly by the server, moving the processing to the data.

Our WW demonstrations acts as a showcase for the OGC Web Coverage Service potential: what can a client get from a WCS server today? The most exciting aspect of WCS is represented by its processing extension, a.k.a. the OGC Web Coverage Processing Service (WCPS) standard. A WCPS-compliant service allows a client to attach a processing query: by exploiting a full grammar for multidimensional-array algebra, several different kinds of information can be retrieved from one or more datasets together: scalar condensers, cross-sectional profiles, comparison maps and plots, etc.

The purpose of our demo is to evaluate and show the application of WCPS queries for analysis and cross-comparison of atmospheric datasets with different spatial and temporal resolutions. Our data provider at MEEO Srl offers maps of columnar Aerosol Optical Thickness (AOT — 550 nm channel) from different sources, namely the MODIS radiometer, ECMWF forecasts and ESA CCI products. The characteristics of each time-series of AOT data are presented in the next section.

Datasets

MOD08: The MOD08 (http://modis.gsfc.nasa.gov/data/dataprod/dataproducts.php?MOD_NUMBER=08) are MODIS Level 3 Atmosphere Products, each covering a different temporal scale: Daily, 8-Day, and Monthly. Each of these Level 3 products contains statistics derived from over 100 science parameters from the Level 2 Atmosphere products: Aerosol, Precipitable Water, Cloud, and Atmospheric Profiles. Statistics are aggregated to a 1° × 1° equal-angle global grid. See also: http://modis.gsfc.nasa.gov/data/dataprod/pdf/MOD_08.pdf MOD08

spatial resolution: 1x1 degrees²
temporal resolution: 1 day (24h sample space too)
# of bands: 1 (MEAN/MIN/MAX bands are extracted to separate 1-band coverages)
data type: float (NODATA: -9999⇒inversion_fail; 0⇒missing)
spacetime bbox: year 2008; world-wide

MACC: (http://apps.ecmwf.int/datasets/data/macc_reanalysis/) The European Centre for Medium-Range Weather Forecasts (ECMWF) develops and operates global models and data-assimilation systems for the dynamics, thermodynamics and composition of the Earth's fluid envelope and interacting parts of the Earth-system. Aerosols from MACC Reanalysis (http://apps.ecmwf.int/datasets/data/macc_reanalysis/) consists of global maps with 1.125° x 1.125° spatial resolution and 3-hourly temporal frequency. See also http://www.gmes-atmosphere.eu/services/aqac/

spatial resolution: 1.125x1.125 degrees²
temporal resolution: 3 hours (3h sample space too)
# of bands: 1
data type: float (NODATA: 0⇒missing)
spacetime bbox: 2007-Dec-12 to 2009-Jan-05; world-wide

ESA CCI: The primary products concerned in the aerosol_cci project are level 2 (daily 10km and 50km pixel products) and level 3 (aggregated monthly gridded datasets) multi-spectral AOD and associated probabilities of predefined aerosol types for a number of European satellite instruments (ATSR-2, AATSR, MERIS, POLDER, GOME, SCIAMACHY, OMI, GOME-2, AVHRR/3); stratospheric aerosols are observed with GOMOS (and tested for SCIAMACHY). Freely available ESA CCI Aerosol products (http://www.esa-aerosol-cci.org/?q=node/148) are available.

spatial resolution: 1x1 degrees²
temporal resolution: 1 day (24h sample space too)
# of bands: 1 (MEAN/STDEV bands are extracted to separate 1-band coverages)
data type: float (NODATA: -9999⇒inversion_fail; 0⇒missing)
spacetime bbox: year 2008; world-wide

WCPS can hide the complexities derived by:

a proper handling of two different NODATA values from MOD and CCI products,
synchronization of data with different spatial and temporal resolution

The presented features are just a sample of how to exploit the WCPS query language to get a variety of information from geo-services, without any need of further maths on the client side.

Demo Features

In this section we describe the intended flow of analysis of our demo.

Area and Time of interest: data access and analysis

As the WW globe appears, the user has to choose its Region of Interest (RoI):

click on the 'Select interest area' button
draw the RoI (rectangular bounding box) on the globe

It is recommended to select a relatively big RoIs: the scale of available datasets are meant for global analysis. By means of the Clear selection button, the initial state of the demo can be set again and RoI can be re-drawn.

Once the RoI is correctly defined, the user can click on the 'Start analysis of selected area' button to trigger the retrieval of the three AOT products (MODIS, CCI, and MACC) over those areas: the frame of analysis of the demo is now spatially defined, but the date of analysis still needs to be selected.

By default, the demo Day of Interest (DoI) is set to 1st of January 2008 (first available date), and the correspondent maps are visualized in the left-side panel 'Time slices preview'. The user can then move the analysis window along the time by visualizing so-called Hovmoeller time-latitude and time-longitude profiles along selected orthogonal cross-sections which the used can define by single mouse clicks on the WW globe, within the RoI. As this is done, Hovmoeller maps will appear in the bottom panel of the same name.

When an interesting AOT situation is recognized on the Hovmoeller maps, the calendar in the bottom-left corner can be used to move the time DoI to a different date: the preview of the time slices are then consequently refreshed in the left-side panel. Note that the data from MACC forecasts are averaged on the 24H period of a day, to synchronize it with MOD and CCI temporal support.

To aid the user on spotting an interesting DoI, by clicking on a product preview, the image is overlaid on the WW globe and an additional panel called 'Statistics' shows, not surprisingly, some sample statistics on the data:

sample mean
sample standard deviation
a the full plot of temporal evolution of AOT mean values on the RoI (the vertical blue line identifies the current DoI)

After RoI/DoI spatio-temporal window is set, the use can start using the tools for cross-comparing the three products with each other.

AOT cross-comparison: WCPS in action

The demo offers 3 different features for aid the numerical and visual comparison of two separate AOT products: A In all the cases, the user first needs to select two out of the three available products (although WCPS has no limitations on the number of datasets that can be used in a request):

right-click on the thumbnail image of the 'Time series preview' panel
enter the 'Send To' menu
click on one of the tools shown

You will see the correspondent bottom-panels being filled as you send products: as the second one is selected, the proper WCPS processing query is sent to the server and the result is visualized.

The 3 features currently available are:

Scatterplot: 2D plots that compare each co-located RoI pixel of the products is shown here: AOT is the target variable in both the dataset, so scatterplots can help showing the degree of similarity between the two sources by the degree of linearity along the x=y line of the cartesian space.

Delta Map: While scapperplots analyze the accordance of the two products along the feature space (what is the degree of accordance for high or low AOT loadings?), delta maps maintain the geo-referencing along its axes by showing the difference in AOT estimation on each pixel of the RoI and on the 24H of the selected DoI. Hot spots in the delta map (high relative positive or negative differences) can give hints on where such differences happen: whether its on land or ocean, which type of land use seems to be the cause, etc.

Merged Map: After scatterplots and maps are visualized in the bottom-panel, a third feature is available to visualize on the globe the merged map of the two products being compared. This is an other precious tool for researchers: at a global scale different models or sensors might show better predictive power under certain peculiar circumstances, and sometimes the best answer is actually given by a weighted mix of the various output. While no chance is currently provided for a manual insertion of weighting coefficients, a pairwise merging average can be overlaid on the globe by selecting this feature.

After visualizing the merged product on the WW globe, the demo is meant to finish. A new tour of analysis can always be undertaken by wither changing date on the reference calendar (bottom-left corner) or by clearing the RoI selection and start it over.