This is a high-level feature summary of rasdaman. For details refer to Why rasdaman??, Technology?, the documentation, the rasdaman Wikipedia entry and the scientific publications of the Large-Scale Scientific Information Systems Research Group at Jacobs University which is coordinating the rasdaman community project.
Technically, rasdaman is a domain independent Array DBMS, which makes it suitable for all applications where raster data management is an issue. The documentation petascope component of rasdaman adds on geo semantics; it is a servlet package implementing OGC standard interfaces WCS, WCPS, WCS-T, and WPS; see matrix below for details and EarthLook for a kaleidoscope of hands-on interactive demos.
Features described on this page below make up the rasdaman community version. Add-ons with performance boosters (such as just-in-time compilation and further storage types) are available with rasdaman enterprise from rasdaman GmbH. The feature table below summarizes the differences.
Historically, rasdaman has pioneered the field of Array Databases, being the first (and currently only fully implemented) system of this kind. The rasdaman technology has been developed over a series of EU funded prjects and then marketed by rasdaman GmbH, a research spin-off dedicated to its commercial support, since 2003. In 2008/2009, the company has teamed up with Jacobs University for a code split to establish rasdaman community (encompassing a complete Array DBMS) as an open-source project. The original rasdaman code remains as rasdaman enterprise. Both are kept in sync at any time, and both rasdaman GmbH and Jacobs University contribute actively to the open-source project.
Contributions to the rasdaman community code come from a worldwide team of collaborators. Notably, a significant extent of the fixes and new functionality is coming from rasdaman GmbH (such as WMS recently). All community contributions submitted are made available in rasdaman community immediately after checking them for correctness and coherence (eg, with the code guide); no contribution whatsoever goes into rasdaman enterprise first. The only action the company undertakes is to keep both rasdaman variants in sync by merging the rasdaman community tree into rasdaman enterprise, which typically occurs upon release of versions so as to keep both in sync. Aside from that, rasdaman enterprise is developed exclusively by the company and does not contain any community code that rasdaman community does not contain. So rest assured that your valuable contributions are to the benefit of the worldwide user community.
Finally, for the legalese see Imprint and Disclaimer.
Summary: rasdaman community versus rasdaman enterprise
The following table shows feature availability in the open source version, rasdaman community, and the proprietary variant, rasdaman enterprise.
Note that both rasdaman versions are complete, fully operational Array Database Systems, without reservation. As a rough indicator, rasdaman enterprise mainly adds performance boosters plus some specific feature extensions and maintenance tools.
|rasql||X||X||every community query will run on both versions|
|C++ API||X||X||C++ clients using community functionality will run on both versions|
|Java API||X||X||Java clients using community functionality will run on both versions|
|dynamic data definition||X||X||new cell, array, and collection types can be added at runtime|
|null values||X||X||released into open source by [http://www.rasdaman.com rasdaman GmbH]|
|storage layout language||X||X|
|input/output data formats||X||X|
|in-situ data access||X||register data instead of importing them, allows reading from any preexisting raster archive structure|
|query result cache||X|
|parallel server processes||X||X|
|peer-to-peer workload balancing||X||X||"inter-query parallelization"|
|distributed query processing||X||"intra-query parallelization"|
|GPU query evaluation||X|
|via command line||X||X|
|via Web browser||X|
|WMS 1.1||X||X||thanks to volunteers from the community porting the code from enterprise (provided by rasdaman GmbH to community|
|WMS 1.3||X||X||thanks to rasdaman GmbH, funded by EU through project PublicaMundi Project|
|WMS client||X||performance optimized for multiple raster layers; but any other client (such as OpenLayers) will work as well, of course|
|WCS 1.1||X||X||deprecated by OGC, unmaintained by rasdaman|
|WCS Core 2.0||X||X||WCS 2.0.1 based on GMLCOV 1.0.1 (both OGC's current version)|
|WCS Encoding Extension: GeoTIFF 0.x.x<||X||X||note that all GDAL-supported formats are available in rasdaman|
|WCS Encoding Extension: JPEG2000 0.x.x||X||X||Same as above; also includes GMLJP2|
|WCS Service Extension: Range Subsetting 1.0.1||X||X|
|WCS Service Extension: Transaction 1.1.4||X||X|
|WCS Service Extension: Processing 1.0.0||X||X||This is the adapter linking WCPS into WCS protocols|
|WCS Service Extension: Scaling 0.x.x||X||X|
|WCS Service Extension: Interpolation 0.x.x||X||X|
|WCS Service Extension: CRS 0.x.x||X|
|WCS Protocol Extension: GET/KVP 1.0.1||X||X|
|WCS Protocol Extension: POST/XML 1.0.0||X||X|
|WCS Protocol Extension: SOAP 1.0.0||X||X|
|WCS Protocol Extension: REST 0.x.x||X||X||In the OGC adoption process|
|WCPS 1.0||X||X||CRS transforms in rasdaman enterprise only|
|Base DBMSs supported:|
|tape archive integration||X|
Array data model
Arrays are determined by their extent ("domain") and their cell ("pixel", "voxel"). Extents are given by a lower and upper bound taken from the integer domain (so negative boundaries are possible as long as the lower bound remains below the upper bound). For the cells, all base and composite data types allowed in languages like C/C++ (except for pointers and arrays) can be defined as cell types, including nested structs.
Over such typed arrays, collections (ie, tables - ODMG style, again) are built. Collections have two columns (attributes), a system-maintained object identifier (OID) and the array itself. This allows to conveniently embed arrays into relational modeling: foreign keys in conventional tables allow to reference particular array objects, in connection with a domain specification even parts of arrays.
As such, rasdaman is prepared for the forthcoming ISO SQL/MDA ("Multi-Dimensional Arrays") standard, which actually is crafted along rasdaman array model and query language. This standard will define arras as new attribute types, following an "array-as-an-attribute" approach for optimal integration with relations (as opposed to an "attribute-as-table" approach - as pursued, e.g., by SciDB and SciQL - which has some remarkable shortcomings in practice).
The rasdaman query language, rasql, offers raster processing formulated through expressions over raster operations in the style of SQL. Consider the following query: "The difference of red and green channel from all images from collection LandsatImages where somewhere in the red channel intensity exceeds 127". In rasql, it is expressed as
select ls.red - ls.green from LandsatImages as ls where max_cells( ls.red ) > 127
Rasql is a full query language, supporting select, insert, update, and delete. Additionally, the concept of a partial update is introduced which allows to selectively update parts of an array. In view of the potentially large size of arrays this is a practically very relevant feature, e.g., for updating satellite image maps with new incoming imagery.
Query formulation is done in a declarative style (queries express what the result should look like, not how to compute it). This allows for extensive optimization on server side. Further, rasql is safe in evaluation: every valid query is guaranteed to to terminate in finite time.
C++ and Java API
Client development is supported by the C++ API, raslib, and the Java API, rasj; both adhere to the ODMG standard. Communication with a rasdaman database is simple: open a connection, send the query string, receive the result set. Iterators allow convenient acecss to query results.
Once installed, go into the share/rasdaman/examples subdirectory to find sample code.
On server side, arrays are stored inside a standard database. To this end, arrays are partitioned into subarrays called tiles; each such tile goes into a BLOB (binary large object) in a relational table. This allows conventional relational database systems to maintain arrays of unlimited size.
A spatial index allows to quickly locate the tiles required for determining the tile set addressed by a query.
The partitioning scheme is open - any kind of tiling can be specified during array instantiation. A set of tiling strategies is provided to ease administrators in picking the most efficient tiling.
Query evaluation in the server follows the principle of tile streaming. Each operator node processes a set of incoming tiles and generates an output tile stream itself. In many cases this allows to keep only one database tile at a time in main memory. Query processing becomes very efficient even on low-end server machines.
A rasdaman server installation can consist of an arbitrary number of rasdaman server processes. A dynamic scheduler, rasmgr, receives incoming connection requests and assigns a free server process. This server process then is dedicated to the particular client until the connection is closed. This allows for highly concurrent access and, at the same time, increases overall safety as clients are isolated against each other.
OGC geo service standards support
While rasdaman itself is domain agnostic and supports any array application, the petascope servlet, as part of rasdaman, adds in geo semantics, such as dealing with geo coordinates. To this end, rasdaman implements the Open Geospatial Consortium standards for gridded coverages, i.e., multi-dimensional raster data. The OGC service interfaces supported are
- Web Map Service: for rendering coverage data into maps which can be displayed with a wide range of open-source and commercial clients,
- Web Coverage Service: a versatile, modular suite for accessing and server-side processing of coverages,
- Web Coverage Processing Service: OGC's declarative raster query language,
- Web Processing Service: invoking geospatial processing services, such as polygon overlay, as a Web service.
The Princial Architect of rasdaman, Peter Baumann, is chair of the OGC WCS Standards Working Group (WCS.SWG) and editor of coverage model (GMLCOV), WCPS, and most of the WCS specifications, rasdaman naturally has become Reference Implementation for several of these standards and usually implements them first and way ahead of other systems, even before final adoption. Likewise, any changes to coverage-related specifications usually are verified in rasdaman first and, hence, become available early. The same holds for the OGC conformance testing of coverage services where rasdaman code contributors have a lead. In summary, rasdaman can be considered the most comprehensive and best tested implementation of the OGC coverage standards.