Opened 4 years ago

Closed 4 years ago

#333 closed defect (fixed)

2Gb file limit

Reported by: hstamerjohanns Owned by: hstamerjohanns
Priority: minor Milestone: 9.0
Component: rasserver Version: 8.4
Keywords: Cc:
Complexity: Medium

Description (last modified by hstamerjohanns)

There seems to be a file limit of 2 Gb for files to import.

bash-4.1$ rasql --user rasadmin --passwd rasadmin --mdddomain '[0:819,0:819,0:819]' --mddtype FloatCube? -f float_d3s820.dat -q 'insert into fs3s1000 values $1'
rasql: rasdaman query tool v1.0, rasdaman v8.4.0rc.1 -- generated on 28.02.2013 15:10:32.
opening database RASBASE at localhost:7001...ok
fetching type information for FloatCube? from database, using readonly transaction...ok
reading file float_d3s820.dat...terminate called after throwing an instance of 'RasqlError?'
Aborted

Data below 2Gb (0:799, 0:799, 0:799] is imported ok.
This limit is not dependent on memory size, as the tested machine has 16 Gb of Ram.

Change History (11)

comment:1 in reply to: ↑ description Changed 4 years ago by hstamerjohanns

  • Description modified (diff)
  • Priority changed from major to minor

Replying to hstamerjohanns:

There seems to be a file limit of 2 Gb for files to import.

bash-4.1$ rasql --user rasadmin --passwd rasadmin --mdddomain '[0:819,0:819,0:819]' --mddtype FloatCube? -f float_d3s820.dat -q 'insert into fs3s1000 values $1'
rasql: rasdaman query tool v1.0, rasdaman v8.4.0rc.1 -- generated on 28.02.2013 15:10:32.
opening database RASBASE at localhost:7001...ok
fetching type information for FloatCube? from database, using readonly transaction...ok
reading file float_d3s820.dat...terminate called after throwing an instance of 'RasqlError?'
Aborted

Data below 2Gb (0:799, 0:799, 0:799] is imported ok.
This limit is dependent on memory size, as the tested machine has
16 Gb of Ram.

comment:2 Changed 4 years ago by dmisev

Most likely some int overflow, we just need to find the variable and change it to long.

comment:3 Changed 4 years ago by dmisev

Maybe you could compile with --debug-enabled, it will give a better idea of where in the code the error is thrown.

comment:4 Changed 4 years ago by hstamerjohanns

Yes there is an 'int size' in applications/rasql/rasql.cc, investigating...

comment:5 Changed 4 years ago by hstamerjohanns

the cleanest way would be to change r_Range from int to long or better int64_t. This has
storage consequences, so this needs to be discussed and aimed for 9.0.

comment:6 Changed 4 years ago by dmisev

  • Milestone set to 9.0
  • Owner changed from dmisev to hstamerjohanns
  • Status changed from new to assigned

comment:7 Changed 4 years ago by hstamerjohanns

I have just imported a 4Gb file. The issue has been fixed by the 'Expand r_Range to long long' fix.

comment:8 Changed 4 years ago by hstamerjohanns

  • Status changed from assigned to accepted

comment:9 Changed 4 years ago by hstamerjohanns

By the way, this fix only affects 64-bit systems. Due to Unix limitations, 32-bit systems are still limited
to 2 Gb. (but who is still using 32-bit Linux anyway?)

comment:10 Changed 4 years ago by hstamerjohanns

The fix needs further testing, do not use it yet.

comment:11 Changed 4 years ago by dmisev

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.