CONCAT should be supported in WCPS?

[Bang Pham Huu]

should I do it in Petascope?

[Dimitar Misev]

ping, should we implement CONCAT support in WCPS?

[Vlad Merticariu]

Pros: useful for creating larger coverages from smaller parts, already implemented in rasql.

Cons: not part of the WCPS standard.

If we go for the implementation we should follow the rasql model:

concat array1 with array 2 with … with arrayN along dimension

For WCPS we could have:

concat coverage1 with coverage2 with … with coverageN along Long

What we need to enforce:

• all involved coverages must have the same crs
• all involved coverages must have the same axis order (otherwise we can't concat along 1 axis)
• all involved coverages must have the same resolutions
• all involved coverages must have the same range fields
• probably there must be some restriction on the coefficients as well

The result coverage should have:

• the domain set resulting from concatenation: usually for axis i that is (min(lowerBound_axis_i_coverage1, lowerBound_axis_i_coverage2, …, lowerBound_axis_i_coverageN) : max(upperBound_axis_i_coverage1, …, upperBound_axis_i_coverageN)). When done along an irregular axis, the coefficients must be added to the result axis after being translated to the new axis origin.
• the range fields of the first coverage (not sure what to to with null values though)
• the extra metadata of the first coverage (or we could concatenate all, or even make something custom)
• the range set is resulting from the concat rasql

We will probably need this in the federation as well, when we will lift it at WCPS level. However I would only go for it once we have a clear need, or if Bang is running out of tasks

[Bang Pham Huu]

thanks Vlad for comment, I'm running out of task now. I can see the tricky problem from the coverage's metadata in WCPS when concatenating along the irregular axis, for example: a 3D coverage A (time (2008-08-01:2008-08-10), detail (2008-08-01, 2008-08-03, 2008-08-05, 2008-08-10), lat, long) concatenates with a subset of coverage A (time (2008-08-01:2008-08-05)).

So, I'm not sure the result of time axis should be what? I think it can use the offset vector by default is 1 (day for ansidate, second for unixtime), then it will be: 2008-08-01:2008-08-10,2008-08-11,2008-08-13,2008-08-15.

the range fields of the first coverage (not sure what to to with null values though)

It can select the null values which are shared by all participating coverages.

the extra metadata of the first coverage (or we could concatenate all, or even make something custom)

It can simply concatenate the extra metadata from participating coverages.

The need for this operation in Petascope can be recalled from Simone's email to dev mail list long time ago.

I'm interested in to edit / run one single WCPS query to put "RGB query" 
+ "False Color" + "False color - cloud masked" maps one close to the 
other (as per the attached png example).
If I'm not wrong the concatenate function exists at rasql level: does 
the concatenation function exist also at petascope level?

[Peter Baumann]

I agree, there are several "spicy" problems coming specifically with coverages. Including: there might be a change in the coverage/axis grid type through concatenation, typically I guess a generalization from regular to irregular. As always, let's have a clear spec before we start.

[Bang Pham Huu]

Replying to pbaumann:

I agree, there are several "spicy" problems coming specifically with coverages. Including: there might be a change in the coverage/axis grid type through concatenation, typically I guess a generalization from regular to irregular. As always, let's have a clear spec before we start.

It is clear for me to do this ticket, except the case with irregular axis. I'm waiting for Vlad's comment in ​http://rasdaman.org/ticket/1597#comment:4, if he sees another significant problem, then, this ticket will be moved to future.

[Vlad Merticariu]

I would say that you can't concatenate geographically overlapping axes. This happens in the regular case as well: the result of concatenating Lat(0:10) with Lat(12:15) would be Lat(0:15). But what would the result of concatenating Lat(0:10) with Lat(0:5)?

So I believe this is an important distinction that we must make at petascope level, as opposed to array level.

For irregular axes it would be the same, if the axes don't overlap the result axis has the union between the direct positions of the axes that were concatenated.

I'm not sure how this affects our use cases though.

[Bang Pham Huu]

I would say that you can't concatenate geographically overlapping axes. This happens in the regular case as well: the result of concatenating Lat(0:10) with Lat(12:15) would be Lat(0:15). But what would the result of concatenating Lat(0:10) with Lat(0:5)? 

It will be of your formula

the domain set resulting from concatenation: usually for axis i that is (min(lowerBound_axis_i_coverage1, lowerBound_axis_i_coverage2, ..., lowerBound_axis_i_coverageN) : max(upperBound_axis_i_coverage1, ..., upperBound_axis_i_coverageN)). 

Lat(0:10) concatenates with Lat(0:5) is Lat(0:10), the number of grid pixels and off setvector for Lat is changed accordingly. For another example, Lat(0:10) concatenates Lat(20:30), result is Lat(0:30).

I don't see it is possible in this case, too if not follow a convention rule.

[Dimitar Misev]

Guys I'd suggest that we put this on hold until some use case pops up. I had some need for concat while formulating queries in testbed 13, but unfortunately I didn't write it in the ticket and can't remember now.
So let's just postpone it for now.

[Dimitar Misev]

We have a good use case for concat in WCPS now: non-contiguous subsetting; e.g. select only indices 1,5,10:12

[Dimitar Misev]

I think it could also be supported in WCS, which would internally translate to WCPS concat:

E.g.

&subset=ansi(date1,date2,...)

[Dimitar Misev]

To summarize

WCPS

concat

Syntax

concat covExp with covExp ... with covExp along Axis

Semantics/constraints is analogous to ​rasql. Petascope should check that the CRS of all covExp are the same, and leave the rest of constraint/type checking to rasdaman. The Axis (e.g. Lat) should be converted to the ordinal rasql number.

non-contiguous subset

Subsetting would allow specifying multiple slices/trims per axis.

Syntax

covExp[ Axis(slice1; trim1; trim2; slice2; ...) ]

Internally this is translated to a concat (preserving the order of slice/trims):

concat cov[Axis(slice1)] with cov[Axis(trim1)] with cov[Axis(trim2)] ... along Axis

WCS

non-contiguous subset

Subsets would allow to specify multiple slices or trims separated by a ;:

&subset=Axis(slice1; trim1; trim2; slice2;...)

Internally this will be translated to a WCPS non-contiguous subset:

covExp[ Axis(slice1; trim1; trim2; slice2; ...) ]

[Dimitar Misev]

Tentatively scheduled for v10

[Peter Baumann]

looks all god, except that the concat axis should keep its axis information - just change axis type from regular to irregular.

[Peter Baumann]

let me try a rule:

• IF concat intervals overlap THEN error
• IF all input axes lined up in concat are regular AND the concatenated coordinate list is regular THEN the output axis is regular, OTHERWISE it is irregular

Examples:

• concat Lat(0:10) with Lat(0:5) → error
• concat Lat(0:10) resolution 1 with Lat(11:20) resolution 1 → Lat(0:20) resolution 1 regular
• concat Lat(0:10) resolution 1 with Lat(10.5:19.5) resolution 1 → Lat(0:19.5) irregular
• concat Lat(0:10) resolution 1 with Lat(11:20) irregular → Lat(0:20) irregular