RASQL_GRIB updates existing collection is really slow

[Bang Pham Huu]

I have a test case like this, a GRIB file (5 messages) in system test / test all wcst_import with 0.5 MB could be ingested as a new collection in 2 seconds, Run the same rasql query to update this collection will take 6 minutes.

The type of collections
#// array
typedef marray <double, [*:*,*:*,*:*,*:*]> test_grib_regular_5_messages_Array;
#// set
typedef set <test_grib_regular_5_messages_Array> null values [9999:9999] test_grib_regular_5_messages_Set;

[INFO] - 09/10/2017 11:03:42.302743: evaluating...
 [INFO] - 09/10/2017 11:03:42.302787: ok, result type 'set <marray <char, [0:93]>>', 1 element(s), total size 94 bytes.
 [INFO] - 09/10/2017 11:03:54.166550: Connected successfully to '/home/rasdaman/install/data/RASBASE'
 [INFO] - 09/10/2017 11:03:54.167140: Request: 'CREATE COLLECTION test_grib_regular_5_messages_pixel_is_point_new test_grib_regular_5_messages_Set'...
 [INFO] - 09/10/2017 11:03:54.167154: parsing...
 [INFO] - 09/10/2017 11:03:54.167232: checking semantics...
 [INFO] - 09/10/2017 11:03:54.167239: evaluating...
 [INFO] - 09/10/2017 11:03:54.167373: ok
 [INFO] - 09/10/2017 11:03:54.167381: ok
 [INFO] - 09/10/2017 11:03:54.223283: Connected successfully to '/home/rasdaman/install/data/RASBASE'
 [INFO] - 09/10/2017 11:03:54.224243: Request: 'INSERT INTO test_grib_regular_5_messages_pixel_is_point_new VALUES <[0:0,0:0,0:0,0:0] 0d> TILING ALIGNED [0:13, 0:10, 0:0, 0:10] TILE SIZE 4000000'...
 [INFO] - 09/10/2017 11:03:54.224262: parsing...
 [INFO] - 09/10/2017 11:03:54.224386: checking semantics...
 [INFO] - 09/10/2017 11:03:54.224398: evaluating...
 [INFO] - 09/10/2017 11:03:54.225199: ok, result type 'set<long>', 1 element(s).
 [INFO] - 09/10/2017 11:03:54.225231: ok
 [INFO] - 09/10/2017 11:03:54.284370: Connected successfully to '/home/rasdaman/install/data/RASBASE'
 [INFO] - 09/10/2017 11:03:54.285169: Request: 'SELECT oid(test_grib_regular_5_messages_pixel_is_point_new) FROM test_grib_regular_5_messages_pixel_is_point_new'...
 [INFO] - 09/10/2017 11:03:54.285188: parsing...
 [INFO] - 09/10/2017 11:03:54.285269: checking semantics...
 [INFO] - 09/10/2017 11:03:54.285449: evaluating...
 [INFO] - 09/10/2017 11:03:54.285650: ok, result type 'set<double>', 1 element(s).
 [INFO] - 09/10/2017 11:03:54.285670: ok
 [INFO] - 09/10/2017 11:03:55.95120: Connected successfully to '/home/rasdaman/install/data/RASBASE'
 [INFO] - 09/10/2017 11:03:55.95743: Request: 'UPDATE test_grib_regular_5_messages_pixel_is_point_new SET test_grib_regular_5_messages_pixel_is_point_new[0:4,0:0,0:719,0:360] ASSIGN shift(decode(<[0:0] 1c>, "GRIB", "{\"internalStructure\":{\"messageDomains\":[{\"msgId\":1,\"domain\":\"[4:4,0:0,0:719,0:360]\"},{\"msgId\":2,\"domain\":\"[3:3,0:0,0:719,0:360]\"},{\"msgId\":3,\"domain\":\"[2:2,0:0,0:719,0:360]\"},{\"msgId\":4,\"domain\":\"[1:1,0:0,0:719,0:360]\"},{\"msgId\":5,\"domain\":\"[0:0,0:0,0:719,0:360]\"}]},\"filePaths\":[\"/home/rasdaman/test_server/rasdaman/systemtest/testcases_services/test_all_wcst_import/test_data/grib_regular_5_messages_pixel_is_point/test5messages.grib2\"]}"), [0,0,0,0]) WHERE oid(test_grib_regular_5_messages_pixel_is_point_new) = 62465'...
 [INFO] - 09/10/2017 11:03:55.95755: parsing...
 [INFO] - 09/10/2017 11:03:55.95879: checking semantics...
 [INFO] - 09/10/2017 11:03:55.96007: evaluating...
 [INFO] - 09/10/2017 11:03:56.155001: ok
 [INFO] - 09/10/2017 11:03:56.155026: ok


 [INFO] - 09/10/2017 11:04:10.312154: Connected successfully to '/home/rasdaman/install/data/RASBASE'
 [INFO] - 09/10/2017 11:04:10.312849: Request: 'UPDATE test_grib_regular_5_messages_pixel_is_point_new SET test_grib_regular_5_messages_pixel_is_point_new[0:4,0:0,0:719,0:360] ASSIGN shift(decode(<[0:0] 1c>, "GRIB", "{\"internalStructure\":{\"messageDomains\":[{\"msgId\":1,\"domain\":\"[4:4,0:0,0:719,0:360]\"},{\"msgId\":2,\"domain\":\"[3:3,0:0,0:719,0:360]\"},{\"msgId\":3,\"domain\":\"[2:2,0:0,0:719,0:360]\"},{\"msgId\":4,\"domain\":\"[1:1,0:0,0:719,0:360]\"},{\"msgId\":5,\"domain\":\"[0:0,0:0,0:719,0:360]\"}]},\"filePaths\":[\"/home/rasdaman/test_server/rasdaman/systemtest/testcases_services/test_all_wcst_import/test_data/grib_regular_5_messages_pixel_is_point/test5messages.grib2\"]}"), [0,0,0,0]) WHERE oid(test_grib_regular_5_messages_pixel_is_point_new) = 62465'...
 [INFO] - 09/10/2017 11:04:10.312867: parsing...
 [INFO] - 09/10/2017 11:04:10.312995: checking semantics...
 [INFO] - 09/10/2017 11:04:10.313120: evaluating...
 [INFO] - 09/10/2017 11:10:10.110952: ok
 [INFO] - 09/10/2017 11:10:10.110979: ok

[Dimitar Misev]

The problem is with the tiling specification which results in ~21,000+ tiles of size 13kB being created. The final array size is 313MB, although it should actually be 10MB. With the corrected tiling scheme the issue is fixed.

Bottom line: do not put random tile copied from elsewhere, it can easily get out of control like this, when importing a simple 0.5MB file.. just leave the tiling out to default if it's not clear how it works.