wiki:WCSTImportGuide

Version 7 (modified by dmisev, 23 months ago) (diff)

--

WCSTImport Guide

Description

WCSTImport is a utility application in the rasdaman software suite that allows importing of georeferenced datasets into a WCS service supporting the Transaction Extension (see wiki:WCSTGuide). Its primary functionality is allowing the ingestion of archives of georeferenced files. This utility introduces two concepts:

  • Recipe - A recipe is a class implementing the BaseRecipe that based on a set of parameters (ingredients) can import a set of files into WCS forming a well defined structure (image, regular timeseries, irregular timeseries etc)
  • Ingredients - An ingredients file is a json file containing a set of parameters that define how the recipe should behave (e.g. the WCS endpoint, the CRS resolver etc are all ingredients)

Running

wcst_import.sh path/to/my_ingredient.json

Recipes

As of now, three recipes are in the codebase:

  • Mosaic Map
  • Regular Timeseries
  • Irregular Timeseries

For each one of them there is an ingredients file under the ingredients folder, which contain an example of what parameters are available. Below, you can find a description of each ingredient file:

Mosaic Map

Well suited for importing a tiled map, not necessarily continuous, it will place all input files given under a single coverage and deal with their position in space. Parameters are explained below (please note that the comments syntax using "" is not valid json so remove them if you copy the parameters):

{
  "config": {
//The endpoint of the WCS service with the WCST extension enabled
    "service_url": "http://localhost:8080/rasdaman/ows",
//A directory where to store the intermediate results
    "tmp_directory": "/tmp/",
//A link to the crs resolver to be used, best to use one that is frequently updated
    "crs_resolver": "http://opengis.net/def/",
//A default 2D crs to be used when the given files do not have one
    "default_crs": "OGC/0/Index2D",
//If set to true, it will print the WCST requests and will not execute them. To actually execute them set it to false
    "mock": true,
//If set to true, the process will not require any user confirmation, use with care, useful for production environments when deployment is automated
    "automated": false
  },
  "input": {
//The name of the coverage, if the coverage already exists, we will update it with the new files
    "coverage_id": "MyCoverage",
    "paths": [
//Any normal full path or regex that would work with the ls command. You can add as many as you wish, separated by commas
      "/var/data/*"
    ]
  },
  "recipe": {
//The name of the recipe
    "name": "map_mosaic",
    "options": {
//The tiling that you want to be done in rasdaman
      "tiling": "ALIGNED [0:500, 0:500]"
    }
  }
}

Mosaic Map

Well suited for importing a tiled map, not necessarily continuous, it will place all input files given under a single coverage and deal with their position in space. Parameters are explained below (please note that the comments syntax using "" is not valid json so remove them if you copy the parameters):

{
  "config": {
//The endpoint of the WCS service with the WCST extension enabled
    "service_url": "http://localhost:8080/rasdaman/ows",
//A directory where to store the intermediate results
    "tmp_directory": "/tmp/",
//A link to the crs resolver to be used, best to use one that is frequently updated
    "crs_resolver": "http://opengis.net/def/",
//A default 2D crs to be used when the given files do not have one
    "default_crs": "OGC/0/Index2D",
//If set to true, it will print the WCST requests and will not execute them. To actually execute them set it to false
    "mock": true,
//If set to true, the process will not require any user confirmation, use with care, useful for production environments when deployment is automated
    "automated": false
  },
  "input": {
//The name of the coverage, if the coverage already exists, we will update it with the new files
    "coverage_id": "MyCoverage",
    "paths": [
//Any normal full path or regex that would work with the ls command. You can add as many as you wish, separated by commas
      "/var/data/*"
    ]
  },
  "recipe": {
//The name of the recipe
    "name": "time_series_regular",
    "options": {
//The starting date for the first slice
      "time_start": "2012-12-02T20:12:02",
//The format of the time provided above, auto will try to guess it, other wise use any combination of YYYY:MM:DD HH:mm:ss
      "time_format": "auto",
//The crs to be used for the time axis
      "time_crs": "http://opengis.net/def/crs/OGC/0/AnsiDate",
//The distance between each slice in time, granularity seconds to days
      "time_step": "2 days 10 minutes 3 seconds",
//The tiling that should be used for it
      "tiling": "ALIGNED [0:1000, 0:1000, 0:2]"
    }
  }
}
  }
}

Mosaic Map

Well suited for importing a tiled map, not necessarily continuous, it will place all input files given under a single coverage and deal with their position in space. Parameters are explained below (please note that the comments syntax using "" is not valid json so remove them if you copy the parameters):

{
  "config": {
//The endpoint of the WCS service with the WCST extension enabled
    "service_url": "http://localhost:8080/rasdaman/ows",
//A directory where to store the intermediate results
    "tmp_directory": "/tmp/",
//A link to the crs resolver to be used, best to use one that is frequently updated
    "crs_resolver": "http://opengis.net/def/",
//A default 2D crs to be used when the given files do not have one
    "default_crs": "OGC/0/Index2D",
//If set to true, it will print the WCST requests and will not execute them. To actually execute them set it to false
    "mock": true,
//If set to true, the process will not require any user confirmation, use with care, useful for production environments when deployment is automated
    "automated": false
  },
  "input": {
//The name of the coverage, if the coverage already exists, we will update it with the new files
    "coverage_id": "MyCoverage",
    "paths": [
//Any normal full path or regex that would work with the ls command. You can add as many as you wish, separated by commas
      "/var/data/*"
    ]
  },
  "recipe": {
//The name of the recipe
    "name": "map_mosaic",
    "options": {
//Information about the time parameter, two option possible, choose either of them
      "time_parameter": {
//Get the date for the slice from a tag that can be read by GDAL
        "metadata_tag": {
//The name of such a tag
          "tag_name": "TIFFTAG_DATETIME"
        },
//The format of the datetime value in the tag
        "datetime_format": "YYYY:MM:DD HH:mm:ss"
      },
      "time_parameter" :{
//Another option to extract the time. Use only one of the two!
        "filename": {
//The regex has to contain groups of tokens, separated by parentheses. The group parameter specifies which regex group to use for retrieving the time value
        "regex": "(.*)_(.*)_(.+?)_(.*)",
        "group": "2"
      },
}

//The crs of the time axis
      "time_crs": "http://opengis.net/def/crs/OGC/0/AnsiDate",
//The tiling to be used
      "tiling": "ALIGNED [0:10, 0:1000, 0:500]"
    }
  }
}

Attachments (4)

Download all attachments as: .zip