Petascope_UploadFileHandler from WCST_import uses filePaths in rasql query

[Dimitar Misev]

The format parameters of decode allow specifying filepaths as follows:

  // Absolute path to input file(s). This improves ingestion performance if the data 
  // is on the same machine as the rasdaman server, as the network transport is bypassed;
  // It is possible that a format could have multiple files associated to each other,
  // so this argument is an array of filepaths.
  // Note: supported for netCDF, GRIB, and GDAL formats.
  "filePaths": [ "/path/to/file.tif", ... ],

It would be good to make use of this option in wcst_import when the conditions are right (data on same machine as rasserver): we'll save quite some data reading / copying / writing.

Example query:

rasql --user rasadmin --passwd rasadmin -q 
     'UPDATE test_mr SET test_mr[0:255,0:210] ASSIGN shift(decode(<[0:0] 1c>, 
      "GDAL", "{\"filePaths\":[\"/home/rasdaman/mr_1.png\"]}"), [0,0]) WHERE oid(test_mr) = 6145'

[Dimitar Misev]

I think this should be default behaviour actually, it's very rare to ingest data to a remote server? wcst_import can't even ingest to a remote server if I'm not wrong?

[Vlad Merticariu]

The blocker is in petascope, but you're right, currently importing is only supported locally.

[Dimitar Misev]

Ok, so we should just use the filePaths when WCST generates the decode format parameters.

[Dimitar Misev]

Raising priority, it's a simple fix but will have quite some impact.

[Bang Pham Huu]

I think by default, it already used this feature as no file was uploaded to a temporary folder and then running rasql update from this folder, have a look at a updated slice from UpdateCoverageHandler.

rasql —user rasadmin —passwd rasadmin -q 'UPDATE test_ansidate_different_crs_origin_5 SET test_ansidate_different_crs_origin_5[0:0,0:4318,0:8640] ASSIGN shift(decode($1, "NetCDF", "{\"variables\": [\"MERIS_nobs_sum\", \"chlor_a\"]}"), [0,0,0]) WHERE oid(test_ansidate_different_crs_origin_5) = 1025'

—file '/home/rasdaman/trask/pixel_shiftted/Chloro/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-201003-fv2.0.nc

[Dimitar Misev]

You'd still run rasql but without the —file option now. Instead the file is specified in the format parameters like this:

rasql —user rasadmin —passwd rasadmin -q 'UPDATE test_ansidate_different_crs_origin_5 SET test_ansidate_different_crs_origin_5[0:0,0:4318,0:8640] ASSIGN shift(decode($1, "NetCDF", "{\"variables\": [\"MERIS_nobs_sum\", \"chlor_a\"], \"filePaths\": [ \"/home/rasdaman/trask/pixel_shiftted/Chloro/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-201003-fv2.0.nc\" ]}"), [0,0,0]) WHERE oid(test_ansidate_different_crs_origin_5) = 1025'

[Bang Pham Huu]

sorry, I could not see Dimitar's comment as email was not fowarded, then closed the ticket.