Exporting large amounts of data truncates the first ~100 bytes

[Dimitar Misev]

More specifically it produces invalid files, file/gdalinfo don't recognize them.

Tested with exporting 500MB collection, and Konstantin reported fail with tiff only with 16MB collection.

[Dimitar Misev]

It's also reproducible with the new encode function, so it doesn't look to be specific to the conversion code itself.

[Dimitar Misev]

Tested some more

png(), encode(c, "PNG" starts failing at exporting 4MB with "Error: unable to save PNG stack"

tiff(), encode(c, "GTiff") starts failing at 8MB

[Peter Baumann]

maybe we should isolate this in a small GDAL program and then file a GDAL ticket and/or ask the GDAL list.

[Dimitar Misev]

We suspect it may be some memory corruption problem, so it might be helpful to run valgrind on directql (can be found in source:applications/directql).

I attached the script I used to reproduce this problem.

[Dimitar Misev]

Conversion sources are in source:conversion, e.g. source:conversion/png.cc

Source for the encode function is in source:qlparser/qtencode.cc

It should be tested with Java in addition (e.g. WCPS query), because as it seems it's
no problem to encode a lot of data via the Java API, so it may be helpful to get a clue why it fails with C++ (rasql client).

[Dimitar Misev]

Have you looked at this Kinga? I suspect it is a bug in the C++ client API, as it doesn't happen via rasj.

[fxavier]

By executing reproduce.sh, I get the following outputs:

---

Testing with 1 MB (786432 B)

---

output.txt rasql_1.png rasql_1.tif reproduce.sh function tiff©…failed, file not found.
output.txt rasql_1.png rasql_1.tif reproduce.sh function png©…failed, file not found.
output.txt rasql_1.png rasql_1.tif reproduce.sh function jpeg©…failed, file not found.
output.txt rasql_1.png rasql_1.tif reproduce.sh function encode(c, "GTiff")…failed, file not found.
output.txt rasql_1.png rasql_1.tif reproduce.sh function encode(c, "PNG")…failed, file not found.

---

………….
(this repeats throughout all the iterations, until my pc crashes).

rasdaman error 0: Exception: Transfer Failed
rasdaman error 801: RasManager Error: System overloaded, please try again later.
rasdaman error 801: RasManager Error: System overloaded, please try again later.
rasdaman error 801: RasManager Error: System overloaded, please try again later.
rasdaman error 801: RasManager Error: System overloaded, please try again later.
…….
(this repeats throughout all the iterations).

I don't know it this is the error or if I didn't reproduced it yet.

[fxavier]

Update in this ticket:

TIFF encode is simply not supported:
rasql -q 'select encode(c, "tiff") from greytest as c' —out file
…
Executing retrieval query…rasdaman error 0: Exception: Feature is not supported

JPEG encode is OK for small datasets (186KB?), but not supported for big ones:
(With 64KB)
Executing retrieval query…ok
Query result collection has 1 element(s):

Result object 1: going into file rasql_1.jpg…ok.

(With over 186KB +-)
Executing retrieval query…rasdaman error 0: Exception: ODMG General

PNG also has its limitations, but it works better than JPEG. This error only appears with sizes of images from ~3MB onwards:
Executing retrieval query…rasdaman error 0: Exception: ODMG General

[EDIT]:
GTIFF works. I could encode 100MB, but at 150MB it crashed, so the limit is somewhere in between.
Encoding 100MB to GTIFF puts my rasdaman using 1.6GB of RAM, and valgrind on directql doesn't behave like that.

[Dimitar Misev]

For tiff it should be

rasql -q 'select encode(c, "gtiff") from greytest as c' —out file

Can you check in the logs if there's more specific error than ODMG General?

[Dimitar Misev]

I attached a simpler script for testing attachment:test.sh​

[Dimitar Misev]

Ok further clue: the bug is irrelevant of the encoding part, even with csv it fails, and as it seems the first 100 bytes or so are cut from the result somewhere on the client side. This does not happen in rasj, so it must be in the C++ client code.

Exporting a [0:9999,0:99] mdd to csv produces proper first 300 bytes:

$ head -c 300 rasql_1.csv 
{2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2},{2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2

Exporting a [0:99999,0:99] mdd to csv however truncates the first 106 bytes (I assume, it may be even more):

$ head -c 300 rasql_1.csv 
,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2},{2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2},{2

[Dimitar Misev]

So hopefully the new protocol will fix this
Right now there is some leak going on in the RNP protocol that causes this.

[Dimitar Misev]

The new protocol fixes this, wontfix for RNP.

[Dimitar Misev]

I submitted a patch that fixes it in RNP, however rasnet is still not fixed.