Use wcst_import in OGC systemtest

[Dimitar Misev]

We should switch to wcst_import for importing the testdata.

[Dimitar Misev]

So you should open source:systemtest/util/petascope.sh and there you have the various import_* scripts that ingest the data used in the petascope (WCS/WCPS/..) tests.

In these functions we should replace the calls to rasql and psql with wcst_import.sh and an ingredient file for each coverage (rgb, mr, mean_summer_airtemp, etc).

[Bang Pham Huu]

Ok, Dimitar. Now, I know how to do this and could manually inject my script to petascope.sh. However, it looks like something error with WCST when I try to import rgb with recipe (map_mosaic). Why it need to -0.5 in i and j axis?

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=rgb1&subset1=i(0.5,399.5)&inputCoverageRef=file:///tmp/9154dae2_b18a_4c12_8bea_5516a0b71d11.gml&subset2=j(344.5,-0.5)
Error Code: RuntimeError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>Runtime error while processing request</ows:ExceptionText>
    </ows:Exception>
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>petascope.wcps2.error.managed.processing.UnorderedSubsetException : Invalid subsetting coordinates: 344.5:-0.5 for axis j.</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

[Dimitar Misev]

Do you use crs Index2D?

[Bang Pham Huu]

Have a look in here Dimitar: ​http://rasdaman.org/attachment/ticket/941/rgb_import.json, I think there are some other problem because with this recipe I could ingest another files .Tiff.

[Dimitar Misev]

Hm not sure if subset_correction should be false?

[Alex Dumitru]

Yes, it should be false, that should be enabled only for extra small offset vectors.

[Bang Pham Huu]

Thanks guys. I've passed over this error and I could ingest RGB to Rasdaman by Petascope. Once I also have a trouble with this ingestion due to the permit of RASBASE (I've changed it again for rasdaman user instead of only root can write). So I think the exception could be add some information like ("Check the permission of RASBASE" as suggestion will help users).

Initialization
Collected files: ['/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png']...

Validation
The recipe has been validated and is ready to run.
Recipe: map_mosaic
Coverage: png
WCS Service: http://localhost:8080/rasdaman/ows
Operation: INSERT
Subset Correction: False
Mocked: False
Track files: True
WMS Import: False

A couple of files have been analyzed and valid to import in below. Please check the axis subsets are correct.

Slice 1: {Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=InsertCoverage&coverageRef=file:///tmp/b03ad296_56e5_4728_963b_f5d7d8f76b25.gml&tiling=ALIGNED [0:100, 0:100]&pixelDataType=Byte&useId=existing
Error Code: RasdamanRequestFailed
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RasdamanRequestFailed">
        <ows:ExceptionText>Error evaluating rasdaman query: 'CREATE COLLECTION png RGBSet</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

[Bang Pham Huu]

Hi again. I have messed up with the permission so I tried to reinstall Rasdaman and retry with the sample data. But this time with "subset_correction": false, it has an error same to the old error.

Slice 1: {Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=rgb&subset1=i(0.0,400.0)&inputCoverageRef=file:///tmp/f242a470_d35c_4be8_b451_46b1502cf4ec.gml&subset2=j(344.0,0.0)
Error Code: RuntimeError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>Runtime error while processing request</ows:ExceptionText>
    </ows:Exception>
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>petascope.wcps2.error.managed.processing.UnorderedSubsetException : Invalid subsetting coordinates: 344.0:0.0 for axis j.</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

Configuration file:

{

  "config": {

    "service_url": "http://localhost:8080/rasdaman/ows",

    "tmp_directory": "/tmp/",

    "crs_resolver": "http://localhost:8080/def",

    "default_crs": "http://localhost:8080/def/crs/OGC/0/Index2D",

    "mock": false,

    "automated": false,

    "subset_correction": false

  },

  "input": {

    "coverage_id": "rgb", 

    "paths": [

      "/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png"

    ]

  },

  "recipe": {

    "name": "map_mosaic", 
    "options": {
 	  "tiling": "ALIGNED [0:100, 0:100]"
    }

  }

[Dimitar Misev]

Yes 344.0:0.0 doesn't look correct, the j is 344 pixels so I'd expect the subset to be 343:0

[Bang Pham Huu]

ok, so Dimitar, you know I'm stucking at this problem but I think when I still could ingest a tif file so let consider it an example of 'rgb' and I will finish the rest now (I assume it ingest data good) and now what I consider is check that coverage ID is existed on Petascope and if it existed then just ignore importing.

I think I should keep the way to check Coverage ID is existed by Postgresql (because when I try with "wget" to Petascope with DescribeCoverage service (even it is existed or not) it only return 500 and stuck.

What about your idea? Should we just import by Petascope and keep check exist and delete by PSQL?

[Dimitar Misev]

It would be good to get away from using low-level postgresql checks as much as possible.

If DescribeCoverage fails for you then something is wrong in your petascope.

[Bang Pham Huu]

Ok, I got the idea and you also agree like this. I want to reconfirm that petascope has no error when I could DescribeCoverage on Web browser ​http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb it return the GML with full detail and no problem (even when the coverage ID is not existed), like this

<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" version="2.0.0" xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd">
<ows:Exception locator="rgb2" exceptionCode="NoSuchCoverage">
<ows:ExceptionText>
One of the identifiers passed does not match with any of the coverages offered by this server
</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>

However, we need to check as you said that the Coverage is existed without PSQL so the only way is access HTTP Request by Terminal and the most common too lis "wget" return about the error of Server. Please view below and yes the reason is of course from Server but it is quite hard to find out the reason when WebBrowser works good and Terminal stop.

wget http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb2
[1] 18364
[2] 18365
[3] 18366
[2]-  Done                    version=2.0.1
[3]+  Done                    request=DescribeCoverage
[rasdaman@gonzo test_wcps]$ --2015-10-14 16:55:07--  http://localhost:8080/rasdaman/ows?service=WCS
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... connected.
HTTP request sent, awaiting response... 500 Server Error
2015-10-14 16:55:07 ERROR 500: Server Error.

[Dimitar Misev]

Can you try

wget "http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb2"

[Dimitar Misev]

In general, always quote arguments in bash.

[Bang Pham Huu]

ok, it is the key, it works with error 404 (NOT found) if coverage is not existed and download the GML file if it is exist. Thanks.

[Bang Pham Huu]

Ok, it looks good now and import also success with check exist before importing. Do you see anywhere we need deleveCoverage? I don't see because if coverage is exist then it just ignore or if not exist then it will insert.

I ask this because I see some old idea before:

  if [ $? -ne 0 ]; then
      #drop_colls $COLL
      #drop_petascope $COLL
      log "  Coverage ID: $COLL is not exist, now importing $COLL... 

and not really sure why if coverage is not exist, still have to delete it Petascope.

[Bang Pham Huu]

I could not know why not only PNG file but I could not import file eobs.nc and mean_summer_airtemp.tif. It is the same errors with the source domain is not fit with target domain (I've change coverage id to new id everytime and also subset_correction: true).

I could only ingest irr_cube_2 files by time_serries_irregular recipe (and with my test rgb and mr_1 (tif not png file because I have problem as previous comments)).

Can I get from Alex also, what should I do to import PNG (rgb, mr) and mean_summer_airtemp and eobs with WCST-Import?

Here is error with mean_summer_airtemp and eobs

wcst_import.sh '/home/rasdaman/ingredient.txt'
Initialization
Collected files: ['/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif']...

Validation
The recipe has been validated and is ready to run.
Recipe: map_mosaic
Coverage: ads8
WCS Service: http://localhost:8080/rasdaman/ows
Operation: INSERT
Subset Correction: True
Mocked: False
Track files: True
WMS Import: False

A couple of files have been analyzed and valid to import in below. Please check the axis subsets are correct.

Slice 1: {Axis Subset: i(111.975,156.275) j(-44.525,-8.975) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(111.975,156.275) j(-44.525,-8.975) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=ads8&subset1=i(112.0,156.25)&inputCoverageRef=file:///tmp/f7066bf8_18fa_482d_be05_c257af718a01.gml&subset2=j(-44.55,-8.95)
Error Code: RasdamanError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RasdamanError">
        <ows:ExceptionText>rasdaman error 967: Update error 967 in line 1, column 1, near token UPDATE: Source MDD domain is not within target update MDD domain.
</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

[Bang Pham Huu]

ok, as ticket 1018 has fixed, now I could ingest all the images without georeferenced with axis (integer point). But still stuck at error 967 when try images with float points in each corners.

After few hours of debugging I can conclude there are something wrong with "ungeoreferenced images" that origins are float value).

But I could not wait until have an answer (for example wait until this is fixed correctly), so I just using gdal tool to set georeference to 2 test files: eobs.nc (netCDF) and mean_summer_airtemp (TIF) and now I could impost by WCST-Tool to Petascope.

So now, I assume everything is working and will try to finish this task soon.

[Bang Pham Huu]

This ticket could not finished soon because:

+ WCST_import could not decode 1D (I think should add function to import CSV file for float_1d).
+ There are differences cases between WCST_Import and test_oracle (query by rasql) with non-georefenced like rgb, mr as ticket 1021 has shown.
+ eobstest also has big different with test oracles and WCST_Import (in test oracles it is a 3D cube (Lat, Long, Time - each band is 1 time anchor) but import by WCST_Import it is 1 2D with 5 bands).
+ Need effor to create float_4d and irr_cub_1.

So this can only change in long term, even need to modify a lot of test oracles (a new test suite can be created) as Vlad suggested like eobstest is deprecated.

[Alex Dumitru]

Good overview Bang. I propose the following things:

• Separate datasets needed for rasql from the ones needed for petascope. WCSTImport is meant to deal with geodata. While it is possible to ingest things like non-georeferenced images, it shouldn't be our goal. So datasets meant for rasql tests should still be imported using plain rasql.
• For petascope tests, we need to tie it to WCSTImport even if that means to change the oracles. We also need to test with real data (and we have plenty of it) and not with manually created datasets like eobtest or rgb. I would suggest the following datasets:
  • 1D: ​http://flanche.com:9090/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=DescribeCoverage&coverageId=NN3_1
  • 2D: ​http://flanche.com:9090/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=DescribeCoverage&coverageId=BlueMarble - scaled version as this is quite huge.
  • 3D: ​http://flanche.com:9090/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=DescribeCoverage&coverageId=AvgLandTemp - first 12 slices or so
  • 4D: ​http://flanche.com:9090/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=DescribeCoverage&coverageId=Temperature4D - t(0:2), h(0:1)

@Dimitar: What do you think?

[Dimitar Misev]

Agreed; it would be good to have a floating point dataset as well.

eobstest is a couple of slices of a ​real dataset btw and is in netcdf format, I think it's not bad to have it as well.

[Alex Dumitru]

@Dimitar: Didn't know that about eobs. I see on gdal that 6 bands are listed and 2 dimensions, while ncdump lists 3 dimensions and 4 variables. It seems we should provide a NetcdfMetadataProvider besides a GdalMetadataProvider in WCSTImport as NetCDF support seems abysimal in GDAL. Maybe we can setup a meeting and discuss what we can do for NetCDF support, I'm not that familiar with the format?

[Dimitar Misev]

Yes GDAL is doing some workarounds to fit the netcdf model in their framework I guess, which doesn't really work out that well. There is ​python API for netcdf, which could be used in wcst_import?