WCPS1.5_Multiple coverages returns from FOR clause

[Bang Pham Huu]

Now, WCPS 1 with this kind of query

for c in (mr, rgb) return encode(c, "png")

will have error due to coverages have different domains/dimensions. If in case of same domains, dimensions like (mean_summer_airtemp, mean_summer_airtemp_repeat), it can also just run the latter coverage in Rasql query.

In WCPS1.5, it will create multiple WCPS query based on the numbers of coverages and return in results from multipart class in Petascope. Example:

This needs the expression in "encode" should be consistent with axis label and size.

for c in (mr, rgb, mr2) return encode(c[i(10:100), j(20:300)], "png")

In case of 1 coverage has different axis label or dimensions, it can return error with this coverage, but does not affect other coverages results.

[Vlad Merticariu]

Creating, internally, several WCPS queries in this case seems correct to me. Since we have a single iterator, there is no join between coverages, so:

for c in (coverage1, coverage2) return encode(some_expression(c), "png")

is equivalent to executing

for c in (coverage1) return encode(some_expression(c), "png")
and
for c in (coverage2) return encode(some_expression(c), "png")
and merging the responses into a single multipart one.

In case there is an error executing one of the queries, I think the overall response should be an error. What do you think?

[Bang Pham Huu]

In case there is an error executing one of the queries, I think the overall response should be an error. What do you think?

so we have list of bytes array and in the end will flush each array to client, no problems if before streaming bytes, it can throw exception if something is wrong.

[Dimitar Misev]

Replying to vmerticariu:

In case there is an error executing one of the queries, I think the overall response should be an error. What do you think?

The WCPS standard doesn't say anything about how to handle this; intuitively I'd also think it should be an overall exception.

[Peter Baumann]

The WCS Processing extension expects a (=1) query as input, so we implement some behavior beyond the standard. Hence, we have degrees of freedom.

+1 for what is stated above: the first query failing aborts processing, and the result is an exception. If clients want something different they can send individual queries and, hence isolate exceptions.

[Peter Baumann]

IMHO heterogeneous coverages should be possible as long as each individual one would succeed with the query body provided. Of course, slicing in t on an x/y coverage should lead to an exception.
Actually, it surprises me that heterogeneous coverage lists do not work.

[Bang Pham Huu]

@Vlad: As I expected, there is a problem with class Multipart of Petascope, HTTP does not allow to download multiple files in 1 Request. Here is a test with real tiff files not from output of Rasql. And what browsers returns is a empty page without asking form for download files.

 OutputStream os = response.getOutputStream();
            response.setStatus(res.getExitCode());
            if (res.getData() != null) {
                
                String[] array = {"/home/rasdaman/Downloads/test/000.tif", 
                                  "/home/rasdaman/Downloads/test/001.tif", 
                                  "/home/rasdaman/Downloads/test/002.tif"};
                MultipartResponse multi = new MultipartResponse(response);
                for (String s: array) {
                    File initialFile = new File(s);
                    InputStream input = new FileInputStream(initialFile);

                    byte[] test = IOUtils.toByteArray(input);
                    //IOUtils.write(res.getData().get(i), os);
                    
                    multi.startPart("image/tiff");
                    os.write(test);
                    multi.endPart();
                }
                multi.finish();
                IOUtils.closeQuietly(os);

So the only solution now is write the list of these byte array to files and compress it to 1 zip file and stream this zip file to client. What do you think?

[Dimitar Misev]

Have a look here: ​https://msdn.microsoft.com/en-us/library/aa493937%28v=exchg.80%29.aspx

I'd expect if the Content-Type is multipart/byteranges or multipart/related it should work fine in modern browsers? There are some browser plugins so you can see the http headers of petascope's response in order to check whether the Content-Type is properly set.

[Peter Baumann]

I am puzzled, multipart works in mail readers, and I can see how multiple files go into one mail.
Note that we cannot provide a form because the client might be a program that has no idea about a form.
My understanding - and that is encoded in CIS 1.1 - is that multipart MIME is one container format, and zip is another. CIS 1.1 will any suitable container.
But back to the current state: following Bang's analysis it seems to me that petascope needs a fix?

[Bang Pham Huu]

Replying to dmisev:

Have a look here: ​https://msdn.microsoft.com/en-us/library/aa493937%28v=exchg.80%29.aspx

I'd expect if the Content-Type is multipart/byteranges or multipart/related it should work fine in modern browsers? There are some browser plugins so you can see the http headers of petascope's response in order to check whether the Content-Type is properly set.

Dimitar, it is MIME type for SMTP (mail protocol) and in HTTP it does not support as Prof. Peter has mentioned. You can see here ​https://www.dropbox.com/en/help/49. So if client does not open many requests then server only can stream all bytes to 1 response. I also cannot see the Headers from Petascope just with Firefox Network view http-headers browser-tools.

[Vlad Merticariu]

Replying to bphamhuu:

Replying to dmisev:

Have a look here: ​https://msdn.microsoft.com/en-us/library/aa493937%28v=exchg.80%29.aspx

I'd expect if the Content-Type is multipart/byteranges or multipart/related it should work fine in modern browsers? There are some browser plugins so you can see the http headers of petascope's response in order to check whether the Content-Type is properly set.

Dimitar, it is MIME type for SMTP (mail protocol) and in HTTP it does not support as Prof. Peter has mentioned. You can see here ​https://www.dropbox.com/en/help/49. So if client does not open many requests then server only can stream all bytes to 1 response. I also cannot see the Headers from Petascope just with Firefox Network view http-headers browser-tools.

That is what a multipart response is: a single file containing the appended bytes of all the parts.

But indeed, browser support for downloading multipart responses is limited. So you could try with wget. For firefox, the content-type should be "x-mixed-replace", see here: ​https://javadigest.wordpress.com/2012/02/13/downloading-multiple-files-using-multipart-response/

[Bang Pham Huu]

But indeed, browser support for downloading multipart responses is limited. So you could try with wget. For firefox, the content-type should be "x-mixed-replace", see here: ​https://javadigest.wordpress.com/2012/02/13/downloading-multiple-files-using-multipart-response/

I've tried with wget and it can return 1 file with all bytes from 3 example files, also I've tried your example in the link above. It also can just return 1 file with the big size (Firefox ask to download 3 files but it can return 0 byte for each file).

So even it can download 3 files with wget, there is 2 problems here:
+ Normally, I think no one use WCPS query with WGET (I think most time is use in WCSClient).
+ The big file it can display only the first example file content, other files does not show. Then it is the same as download 1 file.

I still have no other solution except compress to zip file. Do you have other suggestion?

[Vlad Merticariu]

So we have 2 possible solutions:

1. return a single file representing a multipart response, containing all coverages
2. return a zip archive containing all coverages

The 2nd one is more end-user-friendly, a zip archive can be handled by any user. The first one is more friendly towards web-clients (I imagine that in nasa worlwind, it is easier to read a multipart reponse, and display each part as an image, than unzipping and doing the same, but I might be wrong).

@pbaumann: what do you think, which one should we implement?

[Bang Pham Huu]

Replying to vmerticariu:

So we have 2 possible solutions:

1. return a single file representing a multipart response, containing all coverages
2. return a zip archive containing all coverages

The 2nd one is more end-user-friendly, a zip archive can be handled by any user. The first one is more friendly towards web-clients (I imagine that in nasa worlwind, it is easier to read a multipart reponse, and display each part as an image, than unzipping and doing the same, but I might be wrong).

@pbaumann: what do you think, which one should we implement?

The second case as I've pointed out (I maybe wrong also) is no any client software can understand the combined file from multiple coverages (it can only represent for the first coverage and other data from other coverages does not help for anything). I doubt about this solution.

[Dimitar Misev]

Bang, the HTTP Content-Type response header field specifies the MIME type of the content in the response. multipart/xxx is a MIME type and hence valid in HTTP response, not just SMTP. We should just find the right multipart variation that is supported in wget/curl and browsers.

[Peter Baumann]

first we need to get straight the status quo, that is: multipart. That is good enough for most clients which generate queries, in particular once we support JSON as a substitute for GML. Humans loading from browsers I'd consider a rare exception.

With the implementation of CIS 1.1 we can add more options anyway (zip, GMLJP2, SAFE, …).

[Dimitar Misev]

How come I get two separate files in Firefox with this request?

​http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=GetCoverage&coverageId=mean_summer_airtemp&format=image/tiff&mediatype=multipart/related

[Bang Pham Huu]

Replying to dmisev:

How come I get two separate files in Firefox with this request?

​http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=GetCoverage&coverageId=mean_summer_airtemp&format=image/tiff&mediatype=multipart/related

this link does not work in Chrome but in Firefox it can ask for download 2 files (GML and Tiff). I'm not sure also, but let's just ignore GML as it is ASCII text file, what I want here is download multiple binary files. And I think multipart in HTTP can support with ASCII text (txt, csv, gml, xml…) no problem.

[Bang Pham Huu]

This ticket was fixed as the multipart support of WCPS 1.5, close here, thanks.