Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#164 closed defect (fixed)

Export GeoTIFF/JPEG2000 with WCPS

Reported by: dmisev Owned by: pcampalani
Priority: critical Milestone: 8.4
Component: petascope Version: 8.3
Keywords: gtiff jpeg2000 geo Cc: pbaumann, mantovani
Complexity: Medium

Description (last modified by abeccati)

I have an old version of petascope from Michael, which adds geo-spatial information to TIFF files, converting them to GeoTIFF.

This should be integrated into the current petascope code.

Attachments (2)

relevant_sources.tgz (3.4 KB) - added by dmisev 5 years ago.
wcps.patch (2.0 KB) - added by dmisev 5 years ago.

Download all attachments as: .zip

Change History (40)

Changed 5 years ago by dmisev

comment:1 Changed 5 years ago by arezaeim

  • Owner changed from dmisev to arezaeim
  • Status changed from new to assigned

comment:2 follow-up: Changed 5 years ago by dmisev

There are two ways to fix this converstion:

  1. Allow setting of tags in the tiff convertor itself, so that you can do e.g.
    select tiff(c,"TIFF_GEOTAG1=value1,TIFF_GEOTAG2=value2,...") from COLL as c
    
    This would be done in rasdaman (source:conversion/tiff.cc), and then accordingly use the tiff function with the right GeoTIFF tags from petascope. In this way you can get a GeoTIFF straight from rasdaman.
  1. The other way is as Michael has done it: get a regular TIFF file from rasdaman (so just tiff(c) without any parameters), and then convert it to GeoTIFF in petascope. In this case you'd just need to incorporate the attachment:relevant_sources.tgz in the current petascope.

comment:3 in reply to: ↑ 2 Changed 5 years ago by dmisev

Replying to dmisev:

and then convert it to GeoTIFF in petascope

Note that convert to GeoTIFF means just updating the TIFF file metadata (in-memory), so it's not some expensive operation to do.

comment:4 Changed 5 years ago by dmisev

How can you test your fix: run a WCPS query

for c in (coverage)
return
encode(c, "tiff")

And then verify with gdalinfo if the result file has the expected geo-information.

comment:5 Changed 5 years ago by p.baumann@…

note that "in-memory operation" assumes we can hold the result in main memory, which may not be the case - in other words, we limit ourselves in file size. Further, re-reading a file might cause additional overhead, while rasdaman can do it on the fly while generating the file, hence more efficiently. In the extreme case, this might mean a factor of 2 in performance.

comment:6 Changed 5 years ago by p.baumann@…

discussion:
pro 1: general mechanism - this will be useful, eg, once we want to set further storage parameters, such as tiling, compression, or JPEG quality
pro 1: scalable, as it does not touch the file a second time (after rasdaman generation)
pro 3: scalable beyond server main memory limits
pro 2: done already, and GeoTIFF is geo and, hence, petascope domain.
My favourite is 1, but I'm ready to listen.

comment:7 Changed 5 years ago by dmisev

The mechanism for specifying conversion parameters is already there (QL guide, section 9.8), e.g. for TIFF you have these possible parameters:

comptype=[none|ccittrle|ccittfax3|ccittfa
x4|lzw|ojpeg|jpeg|next|ccittrlew|packbits
|thunderscan|pixarfilm|pixarlog|deflate|d
cs|jbig]
quality=%i (default:80)

So we just need to add a parameter to set specified tags, e.g. something like

tifftags=TAG_NAME1:TAG_VALUE1;TAG_NAME2:TAG_VALUE2;..

Not sure if it's always possible to fit a tag value easily in a string though. Also there are things like TIFF directories, besides the tags.

GeoTIFF documentation btw: http://www.remotesensing.org/geotiff/spec/contents.html

comment:8 Changed 5 years ago by dmisev

Alireza what's the status, have you looked at this?

comment:9 Changed 5 years ago by dmisev

  • Owner changed from arezaeim to dmisev

Obsolete, will be done using the new encode function.

comment:10 follow-up: Changed 5 years ago by dmisev

  • Owner changed from dmisev to pcampalani

Ok, encode function has been implemented, what is needed now is to just pass the georeferencing information to it.

This should be done when the Geotiff/JPEG2000 format extensions in petascope are invoked by a request. When a WCS request is translated to WCPS, these additional parameters need to be passed to the WCPS encode function (which is now same as the rasql one): xmin,ymin,xmax,ymax,crs

On the WCPS level, the encode function should be modified to directly invoke rasql's encode, instead of png(), tiff(), or similar.

I will fix it in WCPS, and I think it's best to assign the WCS level to Piero.

comment:11 Changed 5 years ago by pcampalani

Ok, agreed.

comment:12 in reply to: ↑ 10 ; follow-up: Changed 5 years ago by dmisev

Replying to dmisev:

On the WCPS level, the encode function should be modified to directly invoke rasql's encode, instead of png(), tiff(), or similar.

This is done and already in the repo.

comment:13 Changed 5 years ago by pbaumann

what's the status of this? Several people are waiting for this...

comment:14 Changed 5 years ago by pcampalani

I've been working on other tasks, but I can prioritize it now.

comment:15 Changed 5 years ago by pcampalani

Dimitar,
on the WCPS level, should the user explicitly set the format parameters like:

for c in (mean_summer_airtemp)
  return encode(c, "GTiff", "xmin=XX;ymin=-YY;...")

so that only on the WCS level this would be automatically done?
I guess that at a second step we should also let the WCPS call rasql encode function with automatic filling of some parameters like bbox and CRS.

comment:16 Changed 5 years ago by pcampalani

OGC 12-100 (pending):
---
Requirement 5 /req/geotiff-coverage/mime-type-identifier:
GeoTIFF encoding of a coverage shall be indicated by the following MIME type
identifier: image/tiff
---

comment:17 in reply to: ↑ 12 Changed 5 years ago by dmisev

For now let's make it at WCS because that's easy, at WCPS level it would be quite hard I think because we'll have to investigate the WCPS query tree in order to determine the geo-bounds of the result.

comment:19 Changed 5 years ago by pcampalani

  • Resolution set to fixed
  • Status changed from assigned to closed

Patches accepted, we can fix this.

1. WCS to set proper geo-paramters for GTiff encoding (ticket #164). 
2. WCS to set proper geo-paramters as well for JP2 encoding (ticket #164). 

comment:20 Changed 5 years ago by dmisev

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening so we can fix this for WCPS as well. A preliminary, rudimentary patch: attachment:wcps.patch

Changed 5 years ago by dmisev

comment:21 Changed 5 years ago by abeccati

  • Description modified (diff)
  • Milestone set to 8.4

comment:22 Changed 4 years ago by dmisev

  • Summary changed from TIFF to GeoTIFF in petascope to Export GeoTIFF with WCPS

comment:23 Changed 4 years ago by pcampalani

The patch you suggest Dimitar however sets the overall geo-bounds, and not those of possible trims.
I'll move the discussion on the dev-list.

comment:24 Changed 4 years ago by pcampalani

To fetch the bounds of the request it was necessary to store children nodes at the parsing stage, to then enable fetching trims and slice back in the WCPS XML tree.

@devs: now an additional hierarchy level of the petascope.wcps.server.core classes is inserted via abstract class (AbstractRasNode.java), to host methods shared by all the nodes.

Patch is ready to be submitted.

comment:25 Changed 4 years ago by dmisev

  • Resolution set to fixed
  • Status changed from reopened to closed

Tested, works in changeset:1022398 !

comment:26 Changed 4 years ago by pcampalani

  • Complexity set to Medium
  • Keywords gtiff jpeg2000 geo added
  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Summary changed from Export GeoTIFF with WCPS to Export GeoTIFF/JPEG2000 with WCPS

Some recent patch seems to have broken the functionality at the WCPS level: geo-bounds are not filled at all (anymore), apparently.

for c in (mean_summer_airtemp) return encode (c, "JPEG2000")
for c in (mean_summer_airtemp) return encode (c, "GTiff")

The WCPS test created at the time are *52* and *53*: the GTiff now does not work either anymore, while the JPEG2000 does.
This was not detected since the tests do not check the content (but this is on-going activity, see #363).

for c in (eobstest) return encode(                                                                                                                           
  c[  
    t(1),
      x:"http://www.opengis.net/def/crs/EPSG/0/4326" (20:35),
      y:"http://www.opengis.net/def/crs/EPSG/0/4326" (10:15)]), 
--EITHER    "GTiff")     --
--OR        "JPEG2000")  --

comment:27 Changed 4 years ago by pcampalani

  • Priority changed from major to critical
  • Type changed from enhancement to defect

comment:28 Changed 4 years ago by pbaumann

on a side note: until v9.0 we should move to using MIME type identifiers, as specified in WCS 2.0.1.

comment:29 Changed 4 years ago by mantovani

  • Cc mantovani added

comment:30 Changed 4 years ago by pcampalani

  • Resolution set to fixed
  • Status changed from reopened to closed

WCPS does not use GDAL ids, but WCPS formats (ps9_format.name, so "tiff" and "jpeg2000"), thus comment:26 must be ignored, the functionality is not broken (my fault).

comment:31 Changed 4 years ago by pbaumann

FYI, WCPS 2.0 (under work by me) I plan to harmonize with WCS so that MIME types are used to idenfity formats, and "extra Parameters" can be used as defined in the GMLCOV format profiles.

comment:32 Changed 4 years ago by joosthoek

I filled in the following JPEG2000 export query in http://planetserver.jacobs-university.de:8080/petascope/wcps:

for data in ( frt0000b5a3_07_if165l_trr3_1_01 ) return encode( (char) (255 / (max((data.106 / (0.1*data.12 + 0.1*data.32 + 0.4*data.50 + 0.4*data.71 ) ) - 1) - 0)) * (((data.106 / (0.1*data.12 + 0.1*data.32 + 0.4*data.50 + 0.4*data.71 ) ) - 1) – 0), "JPEG2000" )

The result named ‘wcps’ I renamed to wcps.jp2 and I did gdalinfo:

Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: wcps.jp2
Size is 788, 741
Coordinate System is `'
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  741.0)
Upper Right (  788.0,    0.0)
Lower Right (  788.0,  741.0)
Center      (  394.0,  370.5)
Band 1 Block=788x741 Type=Byte, ColorInterp=Undefined

So unfortunately no CRS info for JPEG2000. Should this work?

comment:33 Changed 4 years ago by dmisev

I submitted a patch that refactors a bit this stuff, and makes it work for cases when extend is used, besides trim/slice. Probably your issue will be fixed by this patch Jelmer.

comment:34 Changed 4 years ago by joosthoek

Thanks. When will this patch be out? The above JPEG2000 export query currently doesn't give a JP2 with CRS info yet.

comment:35 Changed 4 years ago by dmisev

It's already out. Does GeoTIFF work?

comment:36 Changed 4 years ago by joosthoek

Indeed using GTiff works! gdalinfo doesn't show a CRS, but the origin and resolution are the correct values. The exported wcps.tiff was added to ArcGIS and is located at the correct location.

comment:37 Changed 4 years ago by dmisev

Could be some issue with the GDAL JPEG2000 driver, as there's no difference to how GTiff and JPEG2000 are handled in rasdaman.

comment:38 Changed 4 years ago by joosthoek

Ok, for PS GTiff is good enough. Thanks.

Note: See TracTickets for help on using tickets.