Inserting slices in middle of existing irregular timeseries

[Dimitar Misev]

I believe this is not supported at the moment with WCS-T? We have an existing irregular timeseries in petascope and the time slices are contiguous in rasdaman, inserting a time slice in the middle of the series would not be possible.

Related ticket #617

[Peter Baumann]

indeed, that should be supported. But irregular grids have been thoroughly revised in OGC Testbed-11, so we need to get our thinking caps on anyway. Suggest to devise a roadmap of changes necessary.

[damiano]

Some news Dimitar?
I noticed that you changed the milestone. When is M9.2 planned? From our side this ticket can be considered "Critical". Is very difficult to manage long time series without the possibility to add new slice inside the collection.

Damiano

[damiano]

also prepending slices at the beginning of a collection is not supported and rise the following exception:

<ows:Exception exceptionCode="OperationNotSupported">
        <ows:ExceptionText>Can not add new slice in between existing slices on irregular axis. Only adding slices on top is currently supported.</ows:ExceptionText>
    </ows:Exception>

Damiano

[Peter Baumann]

insertion in the middle is quite difficult, prepending and appending somewhat easier.
Damiano, as this is relevant for EarthServer I suggest to prioritize it there (see next WP2 telecon). Also, maybe we can learn more about the use case: why do you need to insert in the middle?
thanks,
Peter

[Dimitar Misev]

Yes this won't be a trivial fix I believe, that's why I changed the milestone.

Peter let's discuss the planning later today.

[damiano]

There are different cases where insertion in the middle is required:
1) The data provider give us missing data
2) The data at a certain date is reprocessed
3) The data provider choose to extend the spatial/temporal domain (from Europe to world wide)
4) …

As larger the collection is, then a larger reingestion time is necessary and is difficult for old and long satellite missions to acquire all the data before start ingestion

[Vlad Merticariu]

Dimitar,

I'm thinking that this may be doable with some limitations. We could simply add the new data as a new tile, and shift the domains of all other tiles (without touching the data). Now I'm not sure what happens if the shape of the tile does not correspond to the current tiling scheme of the object, would that work?

[Dimitar Misev]

I'd suggest we do it with the brute-force way first (suggested in #617), and then tweak rasdaman, what do you think?

[Vlad Merticariu]

Well, in case that can be avoided, I think we should try. Conceptually I think there are at least a few cases where this can be done without much pain, but that might turn out to be false in implementation. As each tile has a definition domain (which is metadata), if we don't consider the shape, that is all that needs to be changed. I will investigate what happens when the shape doesn't match though.

[Peter Baumann]

So what is the task, after all? We have an n-D array A, a slice position p within the extent of A along axis a, and an interval [lo,hi]. Then, we want an array A' which gets sliced at p in a, and all cells with "a" coordinate > hi get shifted by hi-lo+1. The array part "below" gets unchanged. The new part gets filled with null, defaults, or some values provided. A', hence has an increased extent.

My take is: first implement it "brute force" following Dimitar, and after that let's look for optimizaiton opportunities.

However, the given tiling structure IMO must not be compromised. (BTW, we need to complete the tiling interface by allowing to retrieve the tiling strategy used, and possibly modify it selectively.)

[Dimitar Misev]

I'm not sure what is the status here, can we close this ticket?

[Peter Baumann]

I do not see how this can be implemented in a satisfying way in the near future. If we take tiling policies serious (which we should) then there is no way around copying half of the array in general. Opting for "wontfix".

[Dimitar Misev]

Yes indeed, but I'd think copying half of the array instead of throwing an error is a better alternative?

[Vlad Merticariu]

This was implemented for insitu data, where no data shuffling is required (only metadata ops). For non-insitu, data shuffling would be too expensive.

[Vlad Merticariu]

Moving it to future but keeping it open to explore conditions under which this can be done efficiently.

[Dimitar Misev]

I think we should focus on supporting it efficiently given certain constraints.

1. Tiling that does not span the axis where the user intends to insert new slices should not be difficult to do efficiently. For example x/y tiling that does not span the t axis.
2. Regular/aligned tiling that spans the "insertion" axis might be possible to do relatively efficiently.