Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#359 closed defect (invalid)

Error translating to RasQL for complex WCPS query

Reported by: pcampalani Owned by: dmisev
Priority: critical Milestone: 8.4.2
Component: qlparser Version: 8.4
Keywords: Cc: dirk.daems@…
Complexity: Hard

Description

(issued discovered by Dirk Daems)

PREFACE: currently the specific cause of the hereby discussed error is not known: as soon as it will be is identified, the ticket title will change accordingly.

Briefly: starting from a valid (and huge) add() clause, it cannot resolve to a valid RasQL query if wrapped by e.g. a sqrt() operation.


Working query:

for c in (NDVI_SPOT_VGT)
  return
     add(
       coverage averagesOverTime
       over     $T t(0:14)
       values(
        (
          (
      ((1/(count((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T*36)]) * ((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250))) )) * 0.004 - 0.08)
          -
          avg(coverage averagesOverTime2
                over $T2 t(0:14)
                values(((1/(count((c[x(15:20),y(0:5),t(2+$T2*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T2*36)]) * ((c[x(15:20),y(0:5),t(2+$T2*36)]) <= 250))) )) * 0.004 - 0.08)
          )
        )
        *
        (
          (((1/(count((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T*36)]) * ((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250))) )) * 0.004 - 0.08)
          -
          avg(coverage averagesOverTime3
                over $T3 t(0:14)
                values(((1/(count((c[x(15:20),y(0:5),t(2+$T3*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T3*36)]) * ((c[x(15:20),y(0:5),t(2+$T3*36)]) <= 250))) )) * 0.004 - 0.08)
          )
        )
      )
    )/15)

..which is translated into the following RasQL query:

select add_cells(marray i_q in [0:14] values ((((((1)/(count_cells((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_s in [0:14] values ((((1)/(count_cells((((c) [(2)+((i_s[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_s[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_s[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))))*((((((1)/(count_cells((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_q[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_t in [0:14] values ((((1)/(count_cells((((c) [(2)+((i_t[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_t[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_t[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))))) from NDVI_SPOT_VGT AS c

Previous WCPS query with an additional external sqrt() call:

for c in (NDVI_SPOT_VGT)
  return
    sqrt(
     add(
      coverage averagesOverTime
      over $T t(0:14)
      values(
        (
          (((1/(count((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T*36)]) * ((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250))) )) * 0.004 - 0.08)
          -
          avg(coverage averagesOverTime2
                over $T2 t(0:14)
                values(((1/(count((c[x(15:20),y(0:5),t(2+$T2*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T2*36)]) * ((c[x(15:20),y(0:5),t(2+$T2*36)]) <= 250))) )) * 0.004 - 0.08)
          )
        )
        *
        (
          (((1/(count((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T*36)]) * ((c[x(15:20),y(0:5),t(2+$T*36)]) <= 250))) )) * 0.004 - 0.08)
          -
          avg(coverage averagesOverTime3
                over $T3 t(0:14)
                values(((1/(count((c[x(15:20),y(0:5),t(2+$T3*36)]) <= 250)) * add(((c[x(15:20),y(0:5),t(2+$T3*36)]) * ((c[x(15:20),y(0:5),t(2+$T3*36)]) <= 250))) )) * 0.004 - 0.08)
          )
        )
      )
    )/15))

...which is translated into the following RasQL query:

select sqrt((add_cells(marray i_y in [0:14] values ((((((1)/(count_cells((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_{ in [0:14] values ((((1)/(count_cells((((c) [(2)+((i_{[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_{[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_{[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))))*((((((1)/(count_cells((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_y[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_| in [0:14] values ((((1)/(count_cells((((c) [(2)+((i_|[0])*(36)),3696:4256,3696:4256])<=(250)))))*(add_cells((((c) [(2)+((i_|[0])*(36)),3696:4256,3696:4256])*((((c) [(2)+((i_|[0])*(36)),3696:4256,3696:4256])<=(250)))))))*(0.004))-(0.08))))))/(15)) from NDVI_SPOT_VGT AS c

...and which produces the following error:

Caused by: org.odmg.QueryException: Parsing error 300 in line 1, column 294: Unexpected name {.
        at rasj.odmg.RasOQLQuery.execute(RasOQLQuery.java:249)
        at petascope.util.ras.RasUtil.executeRasqlQuery(RasUtil.java:111)
        ... 18 more

It looks like the name used in the RASQL array constructor (... marray i_{ in ...) is causing the error.

Change History (11)

comment:1 Changed 4 years ago by pcampalani

I tried to reproduce the error with some similar but much shorter WCPS query.
But they all worked:

  • sqrt() of a reduce expression:
    for c in (mean_summer_airtemp)
      return sqrt(
         add(
            coverage averagesOverTime
            over     $T t(0:4)
            values   $T
         )
      )
    
  • sqrt() of two nested reduce expressions:
    for c in (mean_summer_airtemp)
      return sqrt(
         add(
            coverage outerCov
            over $T t(0:4)
            values max(
               coverage innerCov
               over     $T t(0:5)
               values   $T
            )
         )
      )
    
  • sqrt() of a reduce expression with a coverage expression to be reduced:
    for c in (mean_summer_airtemp)
      return sqrt(
         max(
            coverage innerCov
            over     $T t(0:5)
            values   c[x:"CRS:1"($T),y:"CRS:1"(300)]
         )
      )
    
  • sqrt() of a nested reduce expression with an inner coverage expression to be reduced:
    for c in (mean_summer_airtemp)
      return sqrt(
         add(
            coverage outerCov
            over $T t(0:4)
            values 1 + max( 
               coverage innerCov
               over $T t(0:5)
               values c[x:"CRS:1"($T),y:"CRS:1"(300)]
            )
         )
      )
    

comment:2 Changed 4 years ago by dmisev

  • Component changed from catalogmgr to qlparser

comment:3 Changed 4 years ago by pcampalani

Dimitar,
I believe this is a ticket for Petascope: the problem is that it creates a variable called '"i_{"', which is not accepted by the qlparser:

$ rasql -q "SELECT add_cells(marray i_{ in [0:1] values 1c) FROM mean_summer_airtemp AS c" --out string
rasql: rasdaman query tool v1.0, rasdaman v8.4.0rc.2-10-g96e9534 -- generated on 08.04.2013 16:58:10.
opening database RASBASE at localhost:7001...ok
Executing retrieval query...rasdaman error 300: Parsing error 300 in line 1, column 27: Unexpected name {.
aborting transaction...ok
rasql done.

comment:4 Changed 4 years ago by pcampalani

Domain-info on the dataset used in the complex query above:

     name      | name | axistype | i |    numlo    |   numhi    |                    name
---------------+------+----------+---+-------------+------------+--------------------------------------------
 NDVI_SPOT_VGT | t    | t        | 0 |           0 |        540 | CRS:1
 NDVI_SPOT_VGT | x    | x        | 1 | -18.0044643 | 52.0044643 | http://www.opengis.net/def/crs/EPSG/0/4326
 NDVI_SPOT_VGT | y    | y        | 2 | -36.0044643 | 38.0044643 | http://www.opengis.net/def/crs/EPSG/0/4326
(3 rows)

comment:5 follow-up: Changed 4 years ago by dmisev

  • Status changed from new to accepted

I can't reproduce it, can you try with the latest petascope Dirk?

I tested with this query:

for c in (eobstest)
return sqrt(
(
(
  add(
    coverage averagesOverTime
    over     $T t(0:2)
    values
    (
      (((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250))) )) * 0.004 - 0.08)
      -
      avg(coverage averagesOverTime2
       over $T2 t(0:2)
       values(((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) <= 250))) )) * 0.004 - 0.08))
    )
    *
    (
      (((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250))) )) * 0.004 - 0.08)
      -
      avg(coverage averagesOverTime3
            over $T3 t(0:2)
            values(((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) <= 250))) )) * 0.004 - 0.08)
      )
    )
  )
)) / 15)

result rasql is

select sqrt((add_cells(marray i_i in [0:2] values ((((((1)/(count_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])*((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_j in [0:2] values ((((1)/(count_cells((((c) [((1)*(i_j[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_j[0]))+(1),15:20,0:5])*((((c) [((1)*(i_j[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))))*((((((1)/(count_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])*((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_k in [0:2] values ((((1)/(count_cells((((c) [((1)*(i_k[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_k[0]))+(1),15:20,0:5])*((((c) [((1)*(i_k[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))))))/(15)) from eobstest AS c

comment:6 Changed 4 years ago by dmisev

I added a test to the systemtest in changeset:ae213f5 which demonstrates this works fine for the eobstest coverage in the tests.

comment:7 in reply to: ↑ 5 Changed 4 years ago by dirk.daems@…

Dimitar,

Can you let me know how I can upload the eobstest coverage? I can't find it in the repository.

Thanks,
Dirk

Replying to dmisev:

I can't reproduce it, can you try with the latest petascope Dirk?

I tested with this query:

for c in (eobstest)
return sqrt(
(
(
  add(
    coverage averagesOverTime
    over     $T t(0:2)
    values
    (
      (((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250))) )) * 0.004 - 0.08)
      -
      avg(coverage averagesOverTime2
       over $T2 t(0:2)
       values(((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T2 + 1)]) <= 250))) )) * 0.004 - 0.08))
    )
    *
    (
      (((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T + 1)]) <= 250))) )) * 0.004 - 0.08)
      -
      avg(coverage averagesOverTime3
            over $T3 t(0:2)
            values(((1/(count((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) <= 250)) * add(((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) * ((c[x:"CRS:1"(15:20),y:"CRS:1"(0:5),t(1 * $T3 + 1)]) <= 250))) )) * 0.004 - 0.08)
      )
    )
  )
)) / 15)

result rasql is

select sqrt((add_cells(marray i_i in [0:2] values ((((((1)/(count_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])*((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_j in [0:2] values ((((1)/(count_cells((((c) [((1)*(i_j[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_j[0]))+(1),15:20,0:5])*((((c) [((1)*(i_j[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))))*((((((1)/(count_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_i[0]))+(1),15:20,0:5])*((((c) [((1)*(i_i[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))-(avg_cells(marray i_k in [0:2] values ((((1)/(count_cells((((c) [((1)*(i_k[0]))+(1),15:20,0:5])<=(250)))))*(add_cells((((c) [((1)*(i_k[0]))+(1),15:20,0:5])*((((c) [((1)*(i_k[0]))+(1),15:20,0:5])<=(250)))))))*(0.004))-(0.08))))))/(15)) from eobstest AS c

comment:8 Changed 4 years ago by pcampalani

Hi Dirk,

the dataset is in "systemtest/testcases_petascope/test_wcps/testdata/eobs.nc", but you can get is automatically inserted if you run the WCS or WCPS tests in "systemtest/testcases_petascope/".

See: wiki:RasdamanTestSuites#Petascopetests

Let me know if that works.

Last edited 4 years ago by pcampalani (previous) (diff)

comment:9 Changed 4 years ago by dirk.daems@…

The problem seems to be fixed. Works for both the eobs test coverage and our NDVI_SPOT_VGT coverage.

comment:10 Changed 4 years ago by dmisev

  • Resolution set to invalid
  • Status changed from accepted to closed

Yes, I think you just had a bit older petascope. Closing the ticket then.

comment:11 Changed 4 years ago by abeccati

  • Milestone changed from 9.0 to 8.4.2
Note: See TracTickets for help on using tickets.