RasJ_Cannot send query with metadata containing ampersand (&) from Petascope to rasservers

[Bang Pham Huu]

Import this netCDF file ​https://rsg.pml.ac.uk/shared_files/olcl/for_bang/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OCx-199709-fv3.1.nc with this ingredient ​https://pastebin.com/GWw0fTVq

Then do this simple GetCoverage request with slicing in both lat, long axes

http://localhost:8080/rasdaman/ows?&SERVICE=WCS&VERSION=2.0.1&REQUEST=GetCoverage&COVERAGEID=OCCCI_V3_1_chlor_a_monthly_metadata_test_specific&SUBSET=Lat(-89.99999733271590645982866404)&SUBSET=Long(-180.0000033337191804607014701)&SUBSET=ansi(%221997-09-04T00:00:00.000Z%22)&FORMAT=image/png

From Petascope logs the query is executed in RasUtil.java

 RasUtil@93: Executing rasql query: SELECT encode(c[0,4319,0], "png" ,
 "{\"metadata\":{\"comment\":\"See summary attribute\",\"time_coverage_resolution\":\"P1M\",\"geospatial_vertical_max\":\"0.0\",\"product_version\":\"3.1\",\"geospatial_lat_units\":\"decimal degrees north\",\"geospatial_lon_units\":\"decimal degrees east\",\"references\":\"http://www.esa-oceancolour-cci.org/\",\"Metadata_Conventions\":\"Unidata Dataset Discovery v1.0\",\"keywords\":\"satellite,observation,ocean,ocean colour\",\"number_of_optical_water_types\":\"14\",\"id\":\"ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OCx-199709-fv3.1.nc\",\"naming_authority\":\"uk.ac.pml\",\"spatial_resolution\":\"4km nominal at equator\",\"number_of_files_composited\":\"19\",\"geospatial_lat_max\":\"90.0\",\"title\":\"ESA CCI Ocean Colour Product\",\"standard_name_vocabulary\":\"NetCDF Climate and Forecast (CF) Metadata Conventions Version 1.6\",\"platform\":\"Orbview-2,Aqua,Envisat,Suomi-NPP\",\"sensor\":\"SeaWiFS,MODIS,MERIS,VIIRS\",\"start_date\":\"01-SEP-1997 00:00:00.000000\",\"processing_level\":\"Level-3\",\"time_coverage_duration\":\"P1M\",\"keywords_vocabulary\":\"none\",\"stop_date\":\"30-SEP-1997 23:59:00.000000\",\"geospatial_lat_min\":\"-90.0\",\"geospatial_lat_resolution\":\"0.04166666666666666666\",\"time_coverage_start\":\"199709010000Z\",\"creator_email\":\"help@esa-oceancolour-cci.org\",\"date_created\":\"Tue Aug 23 09:00:23 2016\",\"creation_date\":\"Tue Aug 23 09:00:23 2016\",\"institution\":\"Plymouth Marine Laboratory\",\"geospatial_lon_max\":\"180.0\",\"geospatial_lon_min\":\"-180.0\",\"geospatial_lon_resolution\":\"0.04166666666666666666\",\"creator_name\":\"Plymouth Marine Laboratory\",\"license\":\"ESA CCI Data Policy: free and open access. When referencing, please use: Ocean Colour Climate Change Initiative dataset, Version <Version Number>, European Space Agency, available online at http://www.esa-oceancolour-cci.org. We would also appreciate being notified of publications so that we can list them on the project website at http://www.esa-oceancolour-cci.org/?q=publications\",\"time_coverage_end\":\"199709302359Z\",\"summary\":\"Data products generated by the Ocean Colour component of the European Space Agency Climate Change Initiative project. These files are daily composites of merged sensor (MERIS, MODIS Aqua, SeaWiFS LAC & GAC, VIIRS) products. MODIS Aqua and MERIS were band-shifted and bias-corrected to SeaWiFS bands and values using a temporally and spatially varying scheme based on the overlap years of 2003-2007. VIIRS was band-shifted and bias-corrected in a second stage against the MODIS Rrs that had already been corrected to SeaWiFS levels, for the overlap period 2012-2013. VIIRS and SeaWiFS Rrs were derived from standard NASA L2 products; MERIS and MODIS from a combination of NASA's l2gen (for basic sensor geometry corrections, etc) and HYGEOS Polymer v3.5 (for atmospheric correction). The Rrs were binned to a sinusoidal 4km level-3 grid, and later to 4km geographic projection, by Brockmann Consult's BEAM. Derived products were generally computed with the standard SeaDAS algorithms. QAA IOPs were derived using the standard SeaDAS algorithm but with a modified backscattering table to match that used in the bandshifting. The final chlorophyll is a combination of OC4, Hu's CI and OC5, depending on the water class memberships. Uncertainty estimates were added using the fuzzy water classifier and uncertainty estimation algorithm of Tim Moore as documented in Jackson et al (2017).\",\"Conventions\":\"CF-1.6\",\"project\":\"Climate Change Initiative - European Space Agency\",\"cdm_data_type\":\"Grid\",\"source\":\"NASA SeaWiFS L2 R2014.0 LAC and GAC, MODIS-Aqua L1A, MERIS L1B 3rd reprocessing inc OCL corrections, NASA VIIRS L2 R2014.0.1 (data identical to R2014.0.2)\",\"tracking_id\":\"659b397a-953c-4814-a3e2-460d6218fcfa\",\"geospatial_vertical_min\":\"0.0\",\"creator_url\":\"http://esa-oceancolour-cci.org\",\"netcdf_file_type\":\"NETCDF4_CLASSIC\",\"history\":\"Source data were: ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970904-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970906-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970909-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970910-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970915-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970916-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970918-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970919-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970920-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970921-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970922-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970923-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970924-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970925-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970926-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970927-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970928-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970929-fv3.1.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OCx_QAA-19970930-fv3.1.nc; netcdf_compositor_cci composites Rrs_412, Rrs_443, Rrs_490, Rrs_510, Rrs_555, Rrs_670, water_class1, water_class2, water_class3, water_class4, water_class5, water_class6, water_class7, water_class8, water_class9, water_class10, water_class11, water_class12, water_class13, water_class14, atot_412, atot_443, atot_490, atot_510, atot_555, atot_670, aph_412, aph_443, aph_490, aph_510, aph_555, aph_670, adg_412, adg_443, adg_490, adg_510, adg_555, adg_670, bbp_412, bbp_443, bbp_490, bbp_510, bbp_555, bbp_670, chlor_a, kd_490, chlor_a_log10_bias, Rrs_412_bias, Rrs_443_bias, Rrs_490_bias, Rrs_510_bias, Rrs_555_bias, Rrs_670_bias, aph_412_bias, aph_443_bias, aph_490_bias, aph_510_bias, aph_555_bias, aph_670_bias, adg_412_bias, adg_443_bias, adg_490_bias, adg_510_bias, adg_555_bias, adg_670_bias, kd_490_bias with --mean, chlor_a_log10_rmsd, Rrs_412_rmsd, Rrs_443_rmsd, Rrs_490_rmsd, Rrs_510_rmsd, Rrs_555_rmsd, Rrs_670_rmsd, aph_412_rmsd, aph_443_rmsd, aph_490_rmsd, aph_510_rmsd, aph_555_rmsd, aph_670_rmsd, adg_412_rmsd, adg_443_rmsd, adg_490_rmsd, adg_510_rmsd, adg_555_rmsd, adg_670_rmsd, kd_490_rmsd with --root-mean-square, and SeaWiFS_nobs, MODISA_nobs, MERIS_nobs, VIIRS_nobs, total_nobs - with --totaln1471940520 Subsetted from standardised_geo/ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1M_MONTHLY_4km_GEO_PML_OCx_QAA-199709-fv3.1.nc to only include variables MERIS_nobs_sum,MODISA_nobs_sum,SeaWiFS_nobs_sum,VIIRS_nobs_sum,chlor_a,chlor_a_log10_bias,chlor_a_log10_rmsd,crs,lat,lon,time,total_nobs_sum\"},\"nilValues\":[{\"value\":\"-999999\",\"reason\":\"The nil value represents an error in the sensor.\"},{\"value\":\"-999999\",\"reason\":\"The nil value represents an error in the sensor.\"},{\"value\":\"-999999\",\"reason\":\"The nil value represents an error in the sensor.\"}],\"nodata\":[-999999,-999999,-999999]}")
 FROM OCCCI_V3_1_chlor_a_monthly_metadata_test_specific AS c

However, it is stuck in RasJ at RasRasnetImplementation.java

StreamedHttpQueryRepl streamedHttpQueryRepl = 
                    this.getRasServerService().beginStreamedHttpQuery(
                            beginStreamedHttpQueryReq);

and this RASQL query cannot be sent to rasserver (no query is added in rasserver's log).

[Dimitar Misev]

The problem seems to be that the metadata contains '&' characters. I'm not sure why this is a problem specifically with rasj yet (rasql/raslib does not have this problem).

[Dimitar Misev]

The problem is that http requests (for rasj) seem to be composed of several parameters split by a '&' (see the strtoks [here browser:server/rasserver_entry.cc#L426]).

So when a query contains a '&' like in this case it splits the parameters incorrectly.

@Peter any suggestions on how to fix this? I think a reasonably practical (but not fully correct fix) would be:

1. Get token for QueryString up to the next '&'
2. Get next token
   • if it's not a valid command then likely it is still a part of the QueryString so it needs to be appended to it

[Peter Baumann]

hm, I see two potential culprits:

• ampersand sign
• comma: if I look at the rasql request it seems incorrect as encode() has many more paramters than allowed, separated by comma

First, the protocol should be documented somewhere, including hints on what special characters need particular treatment. If that documentaiton is missing I cannot understand how this passed code review.

Second, maybe the commas inside the metadata have to be escaped just like other special chars.

Third, as obviously http syntax is used internally, maybe all special chars (as per http) need to be rewritten into entities.

my 2 cents.

[Dimitar Misev]

This http code is really old as far as I know. Not sure who has written/reviewed it, and I haven't seen any documentation.

For now I fixed it so that ampersands in queries will be correctly handled. Only ampersands are relevant actually in the request parsing with strtok.

[Peter Baumann]

ah, you are right, my comma concern was wrong - oversaw the enclosing quotes in the example.

[Dimitar Misev]

Patch is applied with appropriate test in the systemtest.