Export GeoTIFF/JPEG2000 with WCPS

[Dimitar Misev]

I have an old version of petascope from Michael, which adds geo-spatial information to TIFF files, converting them to GeoTIFF.

This should be integrated into the current petascope code.

[Dimitar Misev]

There are two ways to fix this converstion:

1. Allow setting of tags in the tiff convertor itself, so that you can do e.g.

   select tiff(c,"TIFF_GEOTAG1=value1,TIFF_GEOTAG2=value2,...") from COLL as c
   

   This would be done in rasdaman (source:conversion/tiff.cc), and then accordingly use the tiff function with the right GeoTIFF tags from petascope. In this way you can get a GeoTIFF straight from rasdaman.

2. The other way is as Michael has done it: get a regular TIFF file from rasdaman (so just tiff(c) without any parameters), and then convert it to GeoTIFF in petascope. In this case you'd just need to incorporate the attachment:relevant_sources.tgz​ in the current petascope.

[Dimitar Misev]

Replying to dmisev:

and then convert it to GeoTIFF in petascope

Note that convert to GeoTIFF means just updating the TIFF file metadata (in-memory), so it's not some expensive operation to do.

[Dimitar Misev]

How can you test your fix: run a WCPS query

for c in (coverage)
return
encode(c, "tiff")

And then verify with gdalinfo if the result file has the expected geo-information.

[p.baumann@…]

note that "in-memory operation" assumes we can hold the result in main memory, which may not be the case - in other words, we limit ourselves in file size. Further, re-reading a file might cause additional overhead, while rasdaman can do it on the fly while generating the file, hence more efficiently. In the extreme case, this might mean a factor of 2 in performance.

[p.baumann@…]

discussion:
pro 1: general mechanism - this will be useful, eg, once we want to set further storage parameters, such as tiling, compression, or JPEG quality
pro 1: scalable, as it does not touch the file a second time (after rasdaman generation)
pro 3: scalable beyond server main memory limits
pro 2: done already, and GeoTIFF is geo and, hence, petascope domain.
My favourite is 1, but I'm ready to listen.

[Dimitar Misev]

The mechanism for specifying conversion parameters is already there (QL guide, section 9.8), e.g. for TIFF you have these possible parameters:

comptype=[none|ccittrle|ccittfax3|ccittfa
x4|lzw|ojpeg|jpeg|next|ccittrlew|packbits
|thunderscan|pixarfilm|pixarlog|deflate|d
cs|jbig]
quality=%i (default:80)

So we just need to add a parameter to set specified tags, e.g. something like

tifftags=TAG_NAME1:TAG_VALUE1;TAG_NAME2:TAG_VALUE2;..

Not sure if it's always possible to fit a tag value easily in a string though. Also there are things like TIFF directories, besides the tags.

GeoTIFF documentation btw: ​http://www.remotesensing.org/geotiff/spec/contents.html

[Dimitar Misev]

Alireza what's the status, have you looked at this?

[Dimitar Misev]

Obsolete, will be done using the new encode function.

[Dimitar Misev]

Ok, encode function has been implemented, what is needed now is to just pass the georeferencing information to it.

This should be done when the Geotiff/JPEG2000 format extensions in petascope are invoked by a request. When a WCS request is translated to WCPS, these additional parameters need to be passed to the WCPS encode function (which is now same as the rasql one): xmin,ymin,xmax,ymax,crs

On the WCPS level, the encode function should be modified to directly invoke rasql's encode, instead of png(), tiff(), or similar.

I will fix it in WCPS, and I think it's best to assign the WCS level to Piero.

[Piero Campalani]

Ok, agreed.

[Dimitar Misev]

Replying to dmisev:

On the WCPS level, the encode function should be modified to directly invoke rasql's encode, instead of png(), tiff(), or similar.

This is done and already in the repo.

[Peter Baumann]

what's the status of this? Several people are waiting for this…

[Piero Campalani]

I've been working on other tasks, but I can prioritize it now.

[Piero Campalani]

Dimitar,
on the WCPS level, should the user explicitly set the format parameters like:

for c in (mean_summer_airtemp)
  return encode(c, "GTiff", "xmin=XX;ymin=-YY;...")

so that only on the WCS level this would be automatically done?
I guess that at a second step we should also let the WCPS call rasql encode function with automatic filling of some parameters like bbox and CRS.

[Piero Campalani]

OGC 12-100 (pending):
—-
Requirement 5 /req/geotiff-coverage/mime-type-identifier:
GeoTIFF encoding of a coverage shall be indicated by the following MIME type
identifier: image/tiff
—-

[Dimitar Misev]

For now let's make it at WCS because that's easy, at WCPS level it would be quite hard I think because we'll have to investigate the WCPS query tree in order to determine the geo-bounds of the result.

[Piero Campalani]

Patch submitted.
Tested queries:

• No subset ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=mean_summer_airtemp&format=image/tiff
• One subset ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=mean_summer_airtemp&subset=x(120,130)&format=image/tiff
• Two subsets ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=mean_summer_airtemp&subset=x(120,130)&subset=y(-30,-20)&format=image/tiff
• Slice away t on eobstest ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=eobstest&subset=t(1601-01-02)&subset=x(30,40)&subset=y(60,70)&format=image/tiff
• 3 bands (RGB) ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=NIR&subset=x(0,100)&subset=y(0,200)&format=image/tiff
• ERROR: output dimension 1 ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=mean_summer_airtemp&subset=x(120,130)&subset=y(-20)&format=image/tiff
• ERROR: output dimension 3 ​http://localhost:8080/petascope/wcs2?service=WCS&version=2.0.0&request=GetCoverage&coverageid=eobstest&subset=t(1601-01-02,1601-01-03)&subset=x(30,40)&subset=y(60,70)&format=image/tiff

Next step: extend behavior to JPEG2000 format.

[Piero Campalani]

Patches accepted, we can fix this.

1. WCS to set proper geo-paramters for GTiff encoding (ticket #164). 
2. WCS to set proper geo-paramters as well for JP2 encoding (ticket #164). 

[Dimitar Misev]

Reopening so we can fix this for WCPS as well. A preliminary, rudimentary patch: attachment:wcps.patch​

[Piero Campalani]

The patch you suggest Dimitar however sets the overall geo-bounds, and not those of possible trims.
I'll move the discussion on the dev-list.

[Piero Campalani]

To fetch the bounds of the request it was necessary to store children nodes at the parsing stage, to then enable fetching trims and slice back in the WCPS XML tree.

@devs: now an additional hierarchy level of the petascope.wcps.server.core classes is inserted via abstract class (AbstractRasNode.java), to host methods shared by all the nodes.

Patch is ready to be submitted.

[Dimitar Misev]

Tested, works in changeset:1022398 !

[Piero Campalani]

Some recent patch seems to have broken the functionality at the WCPS level: geo-bounds are not filled at all (anymore), apparently.

for c in (mean_summer_airtemp) return encode (c, "JPEG2000")

for c in (mean_summer_airtemp) return encode (c, "GTiff")

The WCPS test created at the time are *52* and *53*: the GTiff now does not work either anymore, while the JPEG2000 does.
This was not detected since the tests do not check the content (but this is on-going activity, see #363).

for c in (eobstest) return encode(                                                                                                                           
  c[  
    t(1),
      x:"http://www.opengis.net/def/crs/EPSG/0/4326" (20:35),
      y:"http://www.opengis.net/def/crs/EPSG/0/4326" (10:15)]), 
--EITHER    "GTiff")     --
--OR        "JPEG2000")  --

[Peter Baumann]

on a side note: until v9.0 we should move to using MIME type identifiers, as specified in WCS 2.0.1.

[Piero Campalani]

WCPS does not use GDAL ids, but WCPS formats (ps9_format.name, so "tiff" and "jpeg2000"), thus comment:26 must be ignored, the functionality is not broken (my fault).

[Peter Baumann]

FYI, WCPS 2.0 (under work by me) I plan to harmonize with WCS so that MIME types are used to idenfity formats, and "extra Parameters" can be used as defined in the GMLCOV format profiles.

[Jelmer Oosthoek]

I filled in the following JPEG2000 export query in ​http://planetserver.jacobs-university.de:8080/petascope/wcps:

for data in ( frt0000b5a3_07_if165l_trr3_1_01 ) return encode( (char) (255 / (max((data.106 / (0.1*data.12 + 0.1*data.32 + 0.4*data.50 + 0.4*data.71 ) ) - 1) - 0)) * (((data.106 / (0.1*data.12 + 0.1*data.32 + 0.4*data.50 + 0.4*data.71 ) ) - 1) – 0), "JPEG2000" )

The result named ‘wcps’ I renamed to wcps.jp2 and I did gdalinfo:

Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: wcps.jp2
Size is 788, 741
Coordinate System is `'
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  741.0)
Upper Right (  788.0,    0.0)
Lower Right (  788.0,  741.0)
Center      (  394.0,  370.5)
Band 1 Block=788x741 Type=Byte, ColorInterp=Undefined

So unfortunately no CRS info for JPEG2000. Should this work?

[Dimitar Misev]

I submitted a patch that refactors a bit this stuff, and makes it work for cases when extend is used, besides trim/slice. Probably your issue will be fixed by this patch Jelmer.

[Jelmer Oosthoek]

Thanks. When will this patch be out? The above JPEG2000 export query currently doesn't give a JP2 with CRS info yet.

[Dimitar Misev]

It's already out. Does GeoTIFF work?

[Jelmer Oosthoek]

Indeed using GTiff works! gdalinfo doesn't show a CRS, but the origin and resolution are the correct values. The exported wcps.tiff was added to ArcGIS and is located at the correct location.

[Dimitar Misev]

Could be some issue with the GDAL JPEG2000 driver, as there's no difference to how GTiff and JPEG2000 are handled in rasdaman.

[Jelmer Oosthoek]

Ok, for PS GTiff is good enough. Thanks.