Querying big result datasets

[Dimitar Misev]

Reported by Alexander Herzig

I get error messages while working on a large data set and I have no idea what’s going on…

spatial domain: [0:100499,0:137799]
data type: unsigned char

$ rasql -q "select max_cells(m[0:50000,0:50000]) from woody as m" —out string
—> works fine without error

$ rasql -q "select max_cells(m[45000:50000,45000:50000]) from woody as m" —out string
—> produces error 356: Specified domain does not intersect with spatial domain of MDD

$ rasql -q "select max_cells(m) from woody as m" —out string
—> produces error 2016: The index found more cells than allowed

[Dimitar Misev]

'non-materialised' cells in the grid

[Dimitar Misev]

test email notification

[Dimitar Misev]

I think I've reproduced how the 356 error is caused. Testing on the following collection:

create collection masstest2 GreySet
insert into masstest2 values marray x in [0:10,0:10] values 1c
insert into masstest2 values marray x in [10:20,10:20] values 2c

a query which covers both objects works fine:

select max_cells(m[2:15,2:15]) from masstest2 as m

Query result collection has 2 element(s):
  Result element 1: 1
  Result element 2: 2

while this fails

select max_cells(m[2:5,2:5]) from masstest2 as m

Executing retrieval query...rasdaman error 356: Execution error 356 in line 1, column 18, near token m: 
Specified domain does not intersect with spatial domain of MDD.

I'm not sure whether this is the right behavior or is it a bug?

[Dimitar Misev]

We consider now that the a bug, and it's also reported in #122

The

$ rasql -q "select max_cells(m) from woody as m" --out string
--> produces error 2016: The index found more cells than allowed

seems like an overflow bug.

[Dimitar Misev]

Same error when calling fill_pyramid.sh on an RGB array of size 180260 x 115332

working on domain [0:180259,0:115331]:
   fetching map information from database...fillpyramid: rasdaman error 2016: Internal error: The index found more cells than allowed.

[Dimitar Misev]

It's definitely an overflow problem.

This passes: select avg_cells(c[0:53685,0:80000]) from haiti_vnir as c
This fails: select avg_cells(c[0:53686,0:80000]) from haiti_vnir as c

53687 * 80001 = 4295013687 > 4294967296 (unsigned long, 2^{32 - 1)}

Obviously we need higher precision at some place in the code.

[Dimitar Misev]

Submitted patch that fixes the bug by changing the type of r_Area from unsigned int to uint64_t

[Dimitar Misev]

Fixed in

commit 2823756342e55ec143464d5734facec325e2a058
Author: Dimitar Misev <misev@rasdaman.com>
Date:   Fri Dec 14 23:51:50 2012 +0100

    Fix overflow bug in the index (ticket 56)