condense max and min bug

[Dirk Daems]

Doesn't seem to work in the pre-release of v9.0.
Got the following exception using the last WCPS query listed in this ticket:

WcpsError: Invalid coverage Expression, next node: condense - Invalid coverage Expression, next node: condense
        at petascope.wcps.server.core.CoverageExpr.<init>(CoverageExpr.java:171)
        at petascope.wcps.server.core.ConstructCoverageExpr.<init>(ConstructCoverageExpr.java:87)
        at petascope.wcps.server.core.CoverageExpr.<init>(CoverageExpr.java:94)
        at petascope.wcps.server.core.EncodeDataExpr.<init>(EncodeDataExpr.java:77)
        at petascope.wcps.server.core.XmlQuery.startParsing(XmlQuery.java:120)
        at petascope.wcps.server.core.ProcessCoveragesRequest.<init>(ProcessCoveragesRequest.java:102)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:118)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:112)
        at petascope.wcps.server.servlet.WcpsServlet.doPost(WcpsServlet.java:181)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:722)

System tests involving condense min / max also fail.

[cj]

there is no such operator max or min (i.e. writing max(1,2) will also not be recognized). So the problem is not in the condenser per se but in the grammar. Closing the ticket.

[Peter Baumann]

the documentation in the QL Guide indeed lists these operations, so they should be supported.

[Piero Campalani]

Hi there,

in the meantime, a workaround for WCPS condense expressions with min and max — leading to this unsupported RasQL condense operations — is achieved by either using a direct min() operator call, or either, in case of a condense operation nested into a coverage constructor, replacing the nested condensation with a min(<coverageConstr>).
E.g.

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   min(
        coverage mincov
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

Instead of:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

[Dimitar Misev]

Seems like it was fixed in rasql but not yet in WCPS.

[Dimitar Misev]

Replying to pcampalani:

Instead of:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

Actually it's working in WCPS too, note that condense should have using, not values. I've added a test in the WCPS tests.

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        using    c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

[Piero Campalani]

Hi Dirk,

here I get the query working, e.g.:

for c in (mr)
  return encode(
     coverage mincov
     over     $px x (50:61)
     values   (
        condense max
        over     $py y (0:100)
        using    c[i:"CRS:1"($px), j:"CRS:1"($py)]
     )
  ), "csv")

Which query did you run?
Did you use CRS axis labels? (mind i and j in the query)

[Dirk Daems]

Replying to pcampalani:

Hi Dirk,

here I get the query working, e.g.:

for c in (mr)
  return encode(
     coverage mincov
     over     $px x (50:61)
     values   (
        condense max
        over     $py y (0:100)
        using    c[i:"CRS:1"($px), j:"CRS:1"($py)]
     )
  ), "csv")

{84,83,89,87,85,85,85,87,80,80,74,80}

Which query did you run?
Did you use CRS axis labels? (mind i and j in the query)

Piero,

I used the query in this ticket:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        using    c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

[Piero Campalani]

Hi Dirk,
that's the point: that query is not valid in v9.0 since the axis labels are wrong. OGC:Index2D CRS defines two axes named i and j, not x and y.
Please checkout the user guide for further explanations, and close the ticket if you can run the same query I put in comment:13, thanks.

[Dirk Daems]

Replying to pcampalani:

Hi Dirk,
that's the point: that query is not valid in v9.0 since the axis labels are wrong. OGC:Index2D CRS defines two axes named i and j, not x and y.
Please checkout the user guide for further explanations, and close the ticket if you can run the same query I put in comment:13, thanks.

Piero,

Even when replacing 'x' and 'y' by 'i' and 'j' I get the same error. I also tried your query in comment 13, but with the exact same result. Yesterday I downloaded and compiled the latest master (9.0.0-rc1 experienced some issues according to Dimitar) so I have a recent Rasdaman version. FYI: systemtest 84 which also uses a 'condense min' statement also fails.
What I find strange is that your query still uses "CRS:1" while this is not present in the ps_crs table → shouldn't that be e.g. "​http://<servername>:8080/def/crs/OGC/0/Index2D" then?

Regards,
Dirk

[Piero Campalani]

Hi Dirk,
I cannot reproduce your problem unluckily, since here both query in comment:13 and WCPS systemtest n.84 work fine.

I suggest we move to the m-list to further inspect the issue (refer to the ticket number in the subject), I'd need your server logs and the HEAD of your repo.

Regarding CRS:1: this is (still) our way to access internal grid coordinates. Index CRSs are external indexed coordinate systems which we assign to non-georeferenced coverages, but there can be a difference there.

If you look in out systemtest import functions, you see we attach Index CRSs to mr and rgb. The orientiation of such CRSs is not the same as the internal (rasdaman) CRS:1 since the latter starts from UL corner of the image, whereas the CRS sets its origin (0,0) to the LL corner, which is more natural: increasing vertical coordinates go upwards.

In practice this means that if you ask subset=j(0) in a WC*S request you get the bottom line (CRS:1 index = hi).

One can always sync the external Index CRS to have the same orientation as the internal rasdaman space: this is done for irr_cube_1, where the CRS origin is placed exactly where the rasdaman origin is, so UL. irr_cube_1 is a fake image, with no meaningful values, but with a photo for example, doing so would mean that you see the photo flipped upside down.

[Dirk Daems]

Piero,

I forgot to change the axisnames x and y to Long and Lat resp.
The following query seems to work:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px i (100:101)
     values   (
        condense min
        over     $py j (300:301)
        using    c[Long:"CRS:1"($px), Lat:"CRS:1"($py)]
     )
  ), "csv")

Please note that the errormessage ("WCPS Error: Invalid coverage Expression, next node: condense - Invalid coverage Expression, next node: condense") didn't guide me to change the axis names. Seems like a valuable improvement to me.

Regards,
Dirk

[Peter Baumann]

Dirk, you are right - insufficient error message details are a known issue. Actually, an alpha version of WCPS 2 is already in the repo, with a substantially improved error handling. So this will change soon.

[Piero Campalani]

Thanks for the feedback Dirk, indeed we are waiting for WCPS 2.0 for a thorough improvement of error handling. There has been effort on this in the past (#206), but in the end the task could not be achieved.

Meanhwile you might have a look at ​this discussion I started in the m-list: you can find some general suggestion on how to deal with WCPS query construction.

[Piero Campalani]

While the WCPS runs without error, the retrieved values are wrong: it seems that the parametrized indexes gets lost when a condense RasQL expression is placed inside a coverage contructor.

The query proposed by Dirk in comment:17 correctly translates to:

select csv(
  marray i_i in [100:101] values
  condense min over i_j in [300:301] using (c) [i_i[0],i_j[0]]
) from mean_summer_airtemp AS c 

But instead of {90,91} (see equivalent simplified query below), the extracted cells are {0,0}, which probably means that '(c) [i_i[0],i_j[0]]' is internally translated to '(c) [0:0,0:0]'.

Simplified query (workaround when condense is not strictly necessary):

for c in (mean_summer_airtemp)
return encode(
   coverage mincov
   over     $px i (100:101)
   values   min(c[Long:"CRS:1"($px), Lat:"CRS:1"(300:301)]
)), "csv") 

Note: this ticket holds for current development too (pre-9.0 release).

[Piero Campalani]

More flexible workaround is in comment:5, which for release 9.0 translates to:

for c in (mean_summer_airtemp)
return encode(
   coverage mincov
   over     $px x (100:101)
   values   min(
      coverage mincov
      over     $py y (300:301)
      values   c[Long:"CRS:1"($px), Lat:"CRS:1"($py)]
   )
), "csv") 

[Dimitar Misev]

There is a bug in the min itself:

rasql -q 'select csv(marray i in [-1:1] values condense min over j in [3:4] using i) from mr' --out string
  Result object 1: {0,0,0}

max works as expected:

rasql -q 'select csv(marray i in [-1:1] values condense max over j in [3:4] using i[0]) from mr' --out string
  Result object 1: {-1,0,1} 

[Dimitar Misev]

Not even in the min:

rasql -q 'select 5 min 2 from mr' --out string
  Result element 1: 2

[Dimitar Misev]

patch submitted.