Petascope streaming results

[Dimitar Misev]

We should think about petascope streaming results from rasdaman, rather than of getting and shipping all at once to the client.

Right now the implementation is very unscalable as the largest result that can be delivered depends on the RAM amount given to Tomcat.

It would involve some re-engineering of the petascope code (passing around the HTTP response stream to RasUtil), but potentially rasdaman as well (I'm not sure if rasdaman is prepared for this).

[abeccati]

I would provide a streaming interface to Rasdaman instead of passing down a request stream. Something like an array stream reader API
Set chunk size and get next would be basic, we have to figure out how to pass around sdom of the chunks if needed or preferred direction for serialization into the rangeSet

[j.oosthoek@…]

Question: Could you please provide some information about problems with data too big for memory e.g. HiRISE data.

Answer: For the data too big for memory I was helped by Dimitar and Piero. I was able to rasimport the HiRISE data. The problem was in the WMS routines Piero migrated from rasdaman enterprise into community. Making pyramids resulted in errors. As far as I know this hasn’t been resolved so far, I didn’t get the HiRISE data example in fully, but full enough to show in PlanetServer. This indeed still needs to be resolved.

I’m however not sure if adding this data to rasdaman would be the best option. It is huge grayscale and RGB visual imagery. So it’s just for visualization, no WCPS queries are needed here. Currently NASA has the data made available as JPEG2000 and has a JPIP-protocol server. If there would be a JPIP client module for OpenLayers I would use that and we would not have to add this dataset to rasdaman. Maybe the ​WCS JPEG2000 extension might be an option?

[Dimitar Misev]

This ticket has nothing to do with the import of large data, but export of large amounts of data, e.g. there will be a problem if someone requests to download 10GB at once via WCS but the server has only 2GB of RAM.

[Dimitar Misev]

Should be taken into account for the next c/s protocol.

[Dimitar Misev]

Vlad, you can update the description of this ticket accordingly.