decode() function in rasql

[Dimitar Misev]

In similar fashion to the rasql encode() function we could implement a decode() function for importing data by using GDAL.

Example for importing rgb for example:

rasql -q 'insert into rgb values decode($1, "png")' -f rgb.png ..

In this way we can deprecate most of the code in source:conversion as well as expand the capabilities to further formats that GDAL supports.

For inspiration see source:qlparser/qtencode.cc, a qtdecode.cc should be exact inverse of the encode and hence simple to implement.

[Dimitar Misev]

George, your patch is missing some tests. Best if you duplicate some of the tests in source:systemtest/testcases_mandatory/test_conversion/test.sh to use decode. Basically in the systemtest we could replace the inv_* references with decode:

systemtest$ grep -R inv_ *
testcases_mandatory/test_conversion/test.sh:local inv_fun=$2
testcases_mandatory/test_conversion/test.sh:local inv_ext=$4
testcases_mandatory/test_conversion/test.sh:log ----- $fun and inv_$fun conversion ------
testcases_mandatory/test_conversion/test.sh:insert_into test_tmp "$TESTDATA_PATH/$f.$inv_ext" "$extraopts" "inv_$inv_fun"
testcases_mandatory/test_conversion/test.sh:insert_into test_tmp "$TESTDATA_PATH/rgb.png" "" "inv_png"
testcases_mandatory/test_conversion/test.sh:################## jpeg() and inv_jpeg() #######################
testcases_mandatory/test_conversion/test.sh:################## tiff() and inv_tiff() #######################
testcases_mandatory/test_conversion/test.sh:################## png() and inv_png() #######################
testcases_mandatory/test_conversion/test.sh:################## bmp() and inv_bmp() #######################
testcases_mandatory/test_conversion/test.sh:################## vff() and inv_vff() #######################
testcases_mandatory/test_conversion/test.sh:################## hdf() and inv_hdf() #######################
testcases_open/test_select/test.sh:insert_into $TEST_GREY "$TESTDATA_PATH/mr_1.png" "" "inv_png"
testcases_open/test_select/test.sh:insert_into $TEST_GREY2 "$TESTDATA_PATH/mr2_1.png" "" "inv_png"
testcases_open/test_select/test.sh:insert_into $TEST_RGB2 "$TESTDATA_PATH/rgb.png" "" "inv_png"
util/petascope.sh:  $RASQL -q "insert into $c values ($c_basetype) inv_netcdf(\$1, \"vars=tg\")" -f "$TESTDATA_PATH"/eobs.nc > /dev/null || exit $RC_ERROR
util/petascope.sh:  $RASQL -q "insert into $c values ($c_basetype) inv_png(\$1)" -f "$TESTDATA_PATH"/rgb.png > /dev/null || exit $RC_ERROR
util/petascope.sh:  $RASQL -q "insert into $c values ($c_basetype) inv_png(\$1)" -f "$TESTDATA_PATH"/mr_1.png > /dev/null || exit $RC_ERROR
util/rasql.sh:  local inv_fun="$4"
util/rasql.sh:  local values="$inv_fun(\$1 $extraopts)"
util/rasql.sh:  if [ -z "$inv_fun" ]; then
util/rasql.sh:  insert_into $TEST_GREY "$TESTDATA_PATH/mr_1.png" "" "inv_png"
util/rasql.sh:  insert_into $TEST_GREY2 "$TESTDATA_PATH/mr2_1.png" "" "inv_png"
util/rasql.sh:  insert_into $TEST_RGB2 "$TESTDATA_PATH/rgb.png" "" "inv_png"

But the test_conversion ones should still keep the inv_* functions as well, it's good to have them tested for a while longer.

[Dimitar Misev]

Also Peter should document this new function in the QL guide, probably next to the encode.

[Peter Baumann]

indeed - George, please send me what you think I should know for describing.

[Dimitar Misev]

This segfault happens when inserting rgb:

[2014-03-14 14:37:05] request from 127.0.0.1
Request: 'insert into rgb values (char) decode(#MDD0#)'...parsing...checking semantics...QtTypeElement::setDataType() - MDD and complex types need to be specified further.


Segmentation fault caught, stacktrace:
[bt]: (1) /lib/x86_64-linux-gnu/libc.so.6 (??:0) - +0x324f0 [0x7fef7fc354f0]
[bt]: (2) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (mddbasetypecommon.cc:131) - MDDBaseType::getBaseType() const+0xc [0x6b17dc]
[bt]: (3) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (qtunaryinduce.cc:843) - QtCast::checkType(QtTypeTuple*)+0x91 [0x5d1cb3]
[bt]: (4) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (qtinsert.cc:488) - QtInsert::checkType()+0x44 [0x5ef0d0]
[bt]: (5) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (querytree.cc:120) - QueryTree::checkSemantics()+0x91 [0x605c75]
[bt]: (6) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (servercomm2.cc:2268) - ServerComm::executeInsert(unsigned long, char const*, ExecuteQueryRes&)+0x2bb [0x57b21b]
[bt]: (7) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rasserver_entry.cc:307) - RasServerEntry::compat_ExecuteInsertQuery(char const*, ExecuteQueryRes&)+0x30 [0x56e018]
[bt]: (8) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rnpservercomm.cc:769) - RnpRasDaManComm::executeInsertQuery()+0x68 [0x58db1a]
[bt]: (9) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rnpservercomm.cc:361) - RnpRasDaManComm::decodeFragment()+0x2ef [0x58cbf3]
[bt]: (10) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rnpservercomm.cc:132) - RnpRasDaManComm::processRequest(akg::CommBuffer*, akg::CommBuffer*, rnp::RnpTransport::CarrierProtocol, rnp::RnpServerJob*)+0x20d [0x58c49b]
[bt]: (11) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rnpcommunication.cc:415) - rnp::RnpServerJob::processRequest()+0x5d [0x593f6f]
[bt]: (12) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (akgnet_nbcomm.cc:483) - akg::NbCommunicator::processJobs()+0x81 [0x6dbfff]
[bt]: (13) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (akgnet_nbcomm.cc:453) - akg::NbCommunicator::mainLoop()+0xba [0x6dbf1c]
[bt]: (14) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (akgnet_nbcomm.cc:422) - akg::NbCommunicator::runServer()+0x52 [0x6dbe46]
[bt]: (15) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rnpserver.cc:99) - startRnpServer()+0x2fb [0x58b6cb]
[bt]: (16) /home/dimitar/projects/rasdaman/community/install/bin/rasserver (rasserver_main.cc:186) - main+0x2fc [0x52fb66]
[bt]: (17) /lib/x86_64-linux-gnu/libc.so.6 (??:0) - __libc_start_main+0xfd [0x7fef7fc21ead]
[bt]: (18) /home/dimitar/projects/rasdaman/community/install/bin/rasserver() [0x52f6d9]

To reproduce run the WCPS systemtest for example.

[George Merticariu]

Function signatures:

1. decode ( <file>) :
   • <file> - the image file to be imported
   • detects automatically the rasdaman type, so no need for format as extra param

2. decode (<file>, <format>, <gdal_params>):
   • <file> - the image file to be imported
   • <format> - the gdal format to be used (for the list of available format use "gdalinfo —formats")
   • <gdal_parameters> - format specific parameters (for the list of format specific parameters use "gdalinfo —format <format_name>")