"watch" option in wcst_import

[Dimitar Misev]

In the case of a growing archive of data (where new data is continuously being added), it would be convenient to automatically ingest this data in rasdaman as soon as it becomes available.

One idea to support this is through an option in the ingredients file. This is pretty natural as the ingredients file fully specifies how the coverage is constructed. E.g.

"watch": True,
or
"watch": <interval in seconds>

With such an option in the ingredients:

1. User executes wcst_import.sh
2. wcst_import.sh ingests the currently available data as collected by the "paths"
3. Rather than exiting at the end, it continues running as a daemon (relevant: ​python-daemon) that regularly checks for new data. When new data becomes available, the ingestion is automatically done with the same ingredients file to update the coverage.
   • starting wcst_import.sh on the same ingredients file while the daemon is already running should exit with an error in this case, and the daemon keeps running

Some extra useful stuff:

1. Add options to wcst_import for daemon control (start/stop/status), e.g:

   wcst_import.sh ingredients.json --daemon status
   

2. Allow executing a script or program before or after ingesting each file. This would allow for preprocessing, e.g. doing gdalwarp automatically, and cleanup afterwards. A convention for calling scripts and what results they return will need to be established though; I think this would make sense:
   • input: the file(s) to be ingested
   • output: result file (or a list of files), which would replace the originally specified files in the ingestion.