Opened 7 weeks ago

Closed 6 weeks ago

#1547 closed defect (fixed)

data corruption on ingestion

Reported by: bbell Owned by: bbell
Priority: critical Milestone:
Component: undecided Version: development
Keywords: Cc: dmisev
Complexity: Medium

Description

Vitor Gomes found a value corruption error in rasdaman, and was able to reproduce the problem with the following steps:

1) Creating the array:
struct MOD13Q1_pixel {short ndvi,evi,quality,red,nir,blue,mir,view_zenith,sun_zenith,relative_azimuth,day_of_year;octet reliability;};
typedef marray <MOD13Q1_pixel, [0:172799, 0:86599, 0:*] > MOD13Q1_cube;
typedef set <MOD13Q1_cube> MOD13Q1_cube_set;
2) Creating binary files with known values:
tr '\00' $'\01' < /dev/zero | tr -d "\n" | dd count=23000 bs=1 of=file23k
tr '\00' $'\01' < /dev/zero | tr -d "\n" | dd count=23000000 bs=1 of=file23M
3) Inserting data:
rasql -q "insert into mod13q1 values marray it in [0:0,0:0,0:0] values {1s,1s,1s,1s,1s,1s,1s,1s,1s,1s,1s,1o}"  --user rasadmin --passwd rasadmin

rasql -q 'UPDATE mod13q1 as c set c[0:99,0:9,0:0] assign $1' -f file23k --mdddomain "[0:99,0:9,0:0]" --mddtype MOD13Q1_cube --user rasadmin --passwd rasadmin
4) Selecting array content:
rasql -q "select c.nir from mod13q1 as c" --out hex

which yields values ​​other than 1.

According to dmisev, the corruption is likely happening either in the insertion of data or during the network transport of data.

Change History (2)

comment:1 Changed 6 weeks ago by bbell

While working on ticket1555, I ran some tests...

$xxd -b file23k
0000000: 00000001 00000001 00000001 00000001 00000001 00000001 ......
0000006: 00000001 00000001 00000001 00000001 00000001 00000001 ......
000000c: 00000001 00000001 00000001 00000001 00000001 00000001 ......
0000012: 00000001 00000001 00000001 00000001 00000001 00000001 ......
0000018: 00000001 00000001 00000001 00000001 00000001 00000001 ......
and so on...


so the values should not simply be 1 across the board.
If you give me a short bit-wise as
00000001 00000001

I will read it as
256 + 1 = 257

as of the current build, and with the solution-in-progress for ticket1555, the data corruption error seems to be resolved. I will update this ticket again if the patch succeeds!

comment:2 Changed 6 weeks ago by bbell

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.