Custom Query (2765 matches)

In the case of a growing archive of data (where new data is continuously being added), it would be convenient to automatically ingest this data in rasdaman as soon as it becomes available.

The idea is to support this through an option for wcst_import.sh:

--watch [interval (seconds)]

By default interval=3600 if none is specified.

When --watch is specified:

1. interval is as specified, or 3600 (1 hour) otherwise
2. --daemon is implied
3. The daemon runs the ingestion as usual and waits for it to finish
4. After the first execution of wcst_import.py finished, daemon waits for interval seconds, then runs wcst_import.py again, and so on in a cycle

Request: 'SELECT encode(c[6,0:17999,-3600:719], "json" ) FROM test_import_order_descending_irregular_time_netcdf AS c'... parsing... checking semantics... evaluating... 
[WARNING] - 2018-09-03 17:00:38,600, specified domain [6:6,0:17999,-3600:719] does not intersect with the MDD spatial domain [-6:0,0:17999,-3600:719], returning empty result.
 [INFO] - 2018-09-03 17:00:39,909: Segmentation fault caught, stacktrace:
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (1) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x649347]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (2) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x63c2ab]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (3) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x52140d]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (4) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x51889d]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (5) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x549ee7]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (6) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x57b69d]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (7) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x488c0b]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (8) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x471159]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (9) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x425eb8]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (10) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x70f8a8]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (11) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x733516]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (12) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x74ba94]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (13) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x769719]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (14) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x7b74de]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (15) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x7b993b]
 [INFO] - 2018-09-03 17:00:39,909: [bt]: (16) /home/rasdaman/rasdaman_community/install/bin/rasserver() [0x7b9a4c]
 [INFO] - 2018-09-03 17:00:40,049: [bt]: (17) /lib64/libstdc++.so.6 (??:0) - +0xb52b0 [0x7fcaa2e8c2b0]
 [INFO] - 2018-09-03 17:00:40,206: [bt]: (18) /lib64/libpthread.so.0 (??:?) - +0x7e25 [0x7fcaa3500e25]
 [INFO] - 2018-09-03 17:00:40,315: [bt]: (19) /lib64/libc.so.6 (??:0) - clone+0x6d [0x7fcaa25f434d]
 [INFO] - 2018-09-03 17:00:40,315: rasserver terminated.

Rather than running in the foreground as a typical program, optionally wcst_import should be able to run as a daemon process in the background; relevant: ​python-daemon. We could emulate this by running it with nohup, but this doesn't allow very good control.

Starting wcst_import.sh on the same ingredients file while the daemon is already running should exit with an error in this case, and the daemon keeps running.

wcst_import would have options for daemon control, e.g:

• wcst_import.sh ingredients.json —daemon status
• wcst_import.sh ingredients.json —daemon start
• wcst_import.sh ingredients.json —daemon stop

This feature is nice to have for running wcst_import in the background while watching for new files (#1870)