WCST_Import register custom user defined recipes from '$RMANHOME/share/rasdaman/wcst_import/recipes_custom' folder

[Bang Pham Huu]

There is a feature in WCST_Import which allows user to create their own recipes and by using this class ReflectionUtil in wcst_import/util/ReflectionUtil.py, WCST_Import will scan all the possible recipes in wcst_import/recipes, below are the supported recipes currently:

drwxrwxr-x  2 rasdaman rasdaman 4096 Jun 12 11:48 general_coverage
drwxrwxr-x. 2 rasdaman rasdaman   80 Jun 12 11:48 map_mosaic
drwxrwxr-x. 2 rasdaman rasdaman   80 Jun 12 11:48 time_series_irregular
drwxrwxr-x. 2 rasdaman rasdaman   80 Jun 12 11:48 time_series_regular
drwxrwxr-x. 2 rasdaman rasdaman   80 Jun 12 11:48 wcs_extract

@Vlad: Should this feature be removed from WCST_Import? Dimitar and I both think that users will have no benefit to create their own recipes (no use cases up to now) and it is an unnecessary big constraint for changing underneath classes which can be used in recipes.

e.g: Dimitar has an outdated recipe file which was not removed when make install WCST_Import and ReflectionUtil throws an exception about the way it imports library (pygrib).

[Vlad Merticariu]

We can change the way recipes are registered in wcst_import. Right now, ReflectionUtil looks into the recipe directory as you noticed, but I have no problems with maintaining a python array of recipe names.

This means that if the user wants to create a new recipe, he has to add it to the array.

[Bang Pham Huu]

Replying to vmerticariu:

We can change the way recipes are registered in wcst_import. Right now, ReflectionUtil looks into the recipe directory as you noticed, but I have no problems with maintaining a python array of recipe names.

This means that if the user wants to create a new recipe, he has to add it to the array.

ok, so as Vlad wanted to maintain this feature. I think ReflectionUtil should scan recipes folder wcst_import/recipes and a path to external folder to store user's defined recipes by an environment variable (e.g: USER_RECIPES_PATH, default this variable is empty).

Then, when make install with CMake, it can delete the old installation WCST_Import folder and copy the newest one. Also, it doesn't touch the user's recipes folder if it exists in future (!!!). I'm supposing there is no user has ever created a recipe in WCST_Import installation folder.

[Dimitar Misev]

I think ReflectionUtil should scan recipes folder wcst_import/recipes

I disagree with this, you should explicitly import the python files in this internal directory.

and a path to external folder to store user's defined recipes by an environment variable (e.g: USER_RECIPES_PATH, default this variable is empty).

This sounds good.

[Vlad Merticariu]

What I meant above was to remove ReflectionUtil and instead have an array of the form:

ENABLED_RECIPES=["map_mosaic", "irregular_timeseries", …] in a config class. If the user adds a recipe, he has to add it to ENABLED_RECIPES.

[Bang Pham Huu]

Replying to vmerticariu:

What I meant above was to remove ReflectionUtil and instead have an array of the form:

ENABLED_RECIPES=["map_mosaic", "irregular_timeseries", …] in a config class. If the user adds a recipe, he has to add it to ENABLED_RECIPES.

I understood about recipes registry. The problem here is we want to remove old WCST_Import installation to clean and install new files. If we let user create recipes in WCST_Import folder, CMake will remove them as well. So I don't know who even use this user-created recipe feature but it shouldn't cause constraint to WCST_Import later. Best is we don't support this "fancy" feature.

[Vlad Merticariu]

An alternative for keeping track of existing recipes:

• at make install check if recipes dir exists
• if it exists, back it up
• after copying the new recipe dir in place, copy from the backup the files that exist in backup but not in the recipe dir

This allows us to update existing recipes, but in case somebody added something else it will be kept as is.

[Dimitar Misev]

To me it sounds more complex/less robust, and doesn't work for the issue this ticket started actually: there used to be some recipe X in wcst_import, which was removed from the sources but was still in my installation and causing troubles.

[Vlad Merticariu]

But was it a custom recipe? If the user adds a custom recipe, we definitely want it to be kept.

[Dimitar Misev]

It wasn't a custom recipe, see ​http://rasdaman.org/ticket/1726#comment:3

[Dimitar Misev]

My suggestion is to dedicate a directory like share/rasdaman/wcst_import/user_recipes that will not be removed during installation.

[Vlad Merticariu]

You're right, that's clean and easy.

[Bang Pham Huu]

Replying to vmerticariu:

You're right, that's clean and easy.

It is not easy, because:

• Which current custom recipes have ever created from user? How do you know?

You will end up with copy obsolete recipes (not listed from current repository) to this deficated folder.
And the user recipes if ever existed also would not work because it would be moved to different folder (error importing).

So keep in mind that, to support this fancy feature user ever programmed recipes are subject to changed from base codes without their concern, like, like before: import module A, now it is import module B, user will have to fix their created recipes manually.

[Vlad Merticariu]

Replying to bphamhuu:

Replying to vmerticariu:

You're right, that's clean and easy.

It is not easy, because:

• Which current custom recipes have ever created from user? How do you know?

Custom recipes = all recipes except the ones we provide (map_mpsaic, timeseries_irregular etc).

You will end up with copy obsolete recipes (not listed from current repository) to this deficated folder.

???

And the user recipes if ever existed also would not work because it would be moved to different folder (error importing).

So keep in mind that, to support this fancy feature user ever programmed recipes are subject to changed from base codes without their concern, like, like before: import module A, now it is import module B, user will have to fix their created recipes manually.

Quote from the ticket body: "Dimitar and I both think that users will have no benefit to create their own recipes (no use cases up to now)", so what are you talking about?

Please keep the discussion civil and constructive.