wiki:PetascopeDevGuide

Version 26 (modified by pcampalani, 4 years ago) (diff)

--

Petascope Developer's Documentation

This page serves as an introduction on petascope 's development.
An introductory presentation on the whole system can be downloaded here.

Servlets endpoints

Currently there are several, service specific endpoints that could be unified into a more rational and smaller set (see ticket)

Draft endpoints:

  • /coverages : serving WCS, WCPS and WCS-T (WPS to be exmined but I'd say it has its own interface root that is different from covs)
  • /rasdaman : direct rasql
  • /maps : serving wms (leaving room for other map services)

Tracked here:

#270
Deprecate old servlets in petascope

IDE

Petascope can be opened directly as a web project in NetBeans (version 7.2 recommended at the moment).
In NetBeans, go to File -> Open Project, and open $RASDAMAN/applications/petascope, where $RASDAMAN is path to the rasdaman source tree.

Petascope database schema

Coverage metadata is stored in database tables. The database schema documentation can be downloaded from here.

This schema documentation can be regenerated with this script script. The script accepts as arguments the location of the Schema Spy jar file, the location of the PostrgreSQL driver jar file (can be found at $RASDAMAN/applications/petascope/lib/postgresql-...jar) and the location of the petascope.properties settings file ($RMANHOME/etc/petascope.properties).

Old database schema

Initially the database schema required storing all details about the coverage, like coverage axis names/types, CRS details, etc. Now we're in the process of moving to complete use of CRS URLs and definitions for this kind of metadata, so the database schema will be simplified a lot (see below).

New database schema

Note: this is still under development (Piero Campalani), see here for more insights.

There are several GML-related aspect to be solved before these modifications can be applied, but meanwhile I can post the relevant modifications to the petascopedb schema, so that some implementation can get started, in case. Any comment and suggestion is appreciated, of course!

  • NEW TABLES
    • ps_irrSeries : explicit mapping of each domain-value of an irregularly spaced axis to its cell index.
       psql#> \d ps_irrseries
                                Table "public.ps_irrseries"
        Column |  Type   |                         Modifiers
       --------+---------+-----------------------------------------------------------
        id     | integer | not null default
       nextval('ps_irrseries_id_seq'::regclass)
        axis   | integer | not null
        value  | text    | not null
        cell   | integer | not null
       Indexes:
           "ps_irrseries_pkey" PRIMARY KEY, btree (id, axis, value, cell)
       Foreign-key constraints:
           "ps_irrseries_axis_fkey" FOREIGN KEY (axis) REFERENCES ps_domain(id)
       ON DELETE CASCADE
      
      The axis field links the tuple to a certain axis in ps_domain, then cell and value map the association to the rasdaman index. For instance, in case of an irregular time-series of, say, three images:
       psql#> select * from ps_irrSeries where axis=23;
        id | axis |       value      | cell |
       ----+------+------------------+------+
         1 |   23 | 2012-01-01 12:00 |   0  |
         2 |   23 | 2012-01-05 16:30 |   1  |
         3 |   23 | 2012-01-18 22:05 |   2  |
      
    • ps_rasgeo : where rasimport can store metadata -- replacing nm_neta table (ticket #169).
      CREATE TABLE ps_rasgeo (
              oid            integer REFERENCES ps_coverage(oid) ON DELETE CASCADE,
              pixeltype      character varying(50),
              attrtable_name character varying(64),
              primary key (oid)
      );
      
      
    • ps_metadata : for storing extra metadata for WCS 2.0
      CREATE TABLE ps_metadata (
              id serial NOT NULL,
              coverage int NOT NULL,
              binaryMetadata bytea, 
      	primary key (id),
      	foreign key (coverage) references ps_coverage (id)
      );
      
      
  • MODIFIED TABLES
    • ps_coverage : crs field turned to foreign key (as it was meant to be) imported from ps_crs; this field, previously left empty, will now be used as unique CRS reference of the coverage, independently of its number of axes: a unique compound URI will be used in case. The URI must be stored in ps_crs and referenced by ps_coverage.
       ALTER TABLE ps_coverage DROP COLUMN     crs; -- cast integer from text is
       not possible: need to drop the column.
       ALTER TABLE ps_coverage ADD  COLUMN     crs integer;
       ALTER TABLE ps_coverage ADD  CONSTRAINT ps_coverage_crs_fkey FOREIGN KEY
       (crs) REFERENCES ps_crs (id) MATCH FULL;
      
      Optional information regarding the single image OIDs and visibility (stub) is also added to the table (ticket #169):
      ALTER TABLE ps_coverage ADD COLUMN oid     integer  UNIQUE;
      ALTER TABLE ps_coverage ADD COLUMN rasName text     UNIQUE;
      ALTER TABLE ps_coverage ADD COLUMN visible boolean DEFAULT true;
      
      rasName refers to the collection name in rasdaman, and oid to the unique object identifier of the rasdaman object; these two allow to uniquely connect the petascope coverage to the rasdaman MDD, and are necessary when the rasName collection in rasdaman contains more than one MDDs. The code in petascope will be adapted to make use of this information when provided, by modifying the generated rasql from a WCPS query to be in this form:
      select ... from rasName as c where oid(c) = oid
      
      visible determines whether a coverage is advertised in the GetCapabilities response, and whether it is at all available in DescribeCoverage and GetCoverage, as well as in WCPS requests.
      NOTE: a new boolean field, isIrregular, is added (self explanatory). In case this is true, the axis will have tuples in ps_irrSeries to map each point in the series to the corresponding cell index.
    • ps_domain : removed redundant metadata present in the GML definition of a CRS (i.e. uom, name, type) and rely on a generic alphanumeric value for the extents (now called minValue and maxValue), whether it is a numeric coordinate or a timestamp.
      psql#> \d ps_domain
                                  Table "public.ps_domain"
          Column    |  Type   |                       Modifiers
      --------------+---------+--------------------------------------------------------
       id           | integer | not null default
      nextval('ps_domain_id_seq'::regclass)
       coverage     | integer | not null
       i            | integer | not null
       minvalue     | text    |
       maxvalue     | text    |
       isirregular  | boolean | not null default false
      Indexes:
          "ps_domain_pkey" PRIMARY KEY, btree (id)
          "ps_domain_coverage_key" UNIQUE, btree (coverage, i)
      Foreign-key constraints:
          "ps_domain_coverage_fkey" FOREIGN KEY (coverage) REFERENCES
      ps_coverage(id) ON DELETE CASCADE
      Referenced by:
          TABLE "ps_irrseries" CONSTRAINT "ps_irrseries_axis_fkey" FOREIGN KEY
      (axis) REFERENCES ps_domain(id) ON DELETE CASCADE
      
      NOTE-1: the i-order is now the only metadata of an axis, and it must reflect the order which an axis is stored in rasdaman. Moreover the easting component should always be stored relatively before the northing axis. In case the direction is not clear (e.g. antarctic imagery?), the axis defined as x (or longitude, etc.), in the GML definition should be put first. This is to make metadata handling less prone to misunderstandings, see http://docs.geotools.org/latest/userguide/library/referencing/order.html for explanations.
      NOTE-2: In case a collection has a compound CRS (CCRS), the i-order must also be reflected in the order of concatenation of the single atomic CRSs: e.g. in case of a spatio-temporal collection where the t dimension is stored as first axis in rasdaman, the ISO:8601 URI (e.g. http://.../def/crs/ISO/0/8601) must be put before the spatial CRS in the compound URI.
    • ps_format : Added new columns: gdalid and description for mapping the GDAL id to the corresponding MIME type.
      ALTER TABLE ps_format ADD COLUMN gdalid varchar(64);
      ALTER TABLE ps_format ADD COLUMN description text;
      
  • DROPPED TABLES
    • ps_crsDetails : really harmful table, limiting the bboxes to 2D and providing redundant information already visible in ps_domain.
    • ps_cellDomain : the cell-domain extents are now retrieved by Petascope via RasQL queries (sdom).
    • ps_crsSet : the CRS URI is now unique for a coverage, and does not need to be linked to single axes.
    • ps_axisType: this kind of metadata is now in the GML definitions of the CRSs and in Petascope.
  • INTEGRITY ISSUES
    • ps_coverage : nulldefault, interpolationtypedefault, nullresistancedefault, type and crs should be NOT NULL (?)
    • ps_domain : minvalue and maxvalue should be NOT NULL.
    • ps_metadata: metadata should be NOT NULL (?)
    • ...

Discussions

Tickets

Summary Reporter Owner Ticket
WMS style management dmisev #1491
Support wcpsTransformQuery in WMS InsertStyle requests dmisev dmisev #1490
Petascope_Use WCPS 1.5 when process WCPS Request in translated XML format bphamhuu #1357
Petascope_Create correct base type with range fields when create collection bphamhuu bbell #1342
WCS_CRS_Extension Support reproject on distort coverage. bphamhuu #1316
Petascope_More meaniningful error from Rasql projection() bphamhuu #1308
3D coverages in WMS dmisev dmisev #1304
Rasql_WCPS_Support interpolation argument in project() and crsTransform() functions bphamhuu dmisev #1303
Petascope ODMG - Wrong return from single boolean value in Rasql query bphamhuu #1273
wcst_import concurrent will have error in petascope bphamhuu #1152
WCST Import should write the result of the ingestion to a log file mdumitru vfrasineanu #1143
WCS-T doesn't support CInt16 dmisev dmisev #1095
wcst_import: more meaningful type names bphamhuu bphamhuu #1029
Inserting slices in middle of existing irregular timeseries dmisev vmerticariu #936
Move coverage's grid origin when domain is extended vmerticariu vmerticariu #913
Evaluate the use of fixed number of decimal places in numeric representations pcampalani vmerticariu #758
SECORE support proj4 to support transformation to customized CRS pcampalani dmisev #708
Customizable handling of sample size of a coverage point pcampalani vmerticariu #680
Support for gml:CompoundCRS pcampalani dmisev #679
WCPS1.5_WCPS Interval expression to actually support mathematical expressions pcampalani vmerticariu #596
petascopedb to fully support SWE Quantities pcampalani vmerticariu #582
ORM in petascope with Spring Boot pcampalani vmerticariu #502
Petascope streaming results dmisev dmisev #268

Attachments (8)