Context Navigation

Rasdaman FAQ

Here you can find answers to some of the most frequently asked questions about rasdaman. If you have a question not answered on this page, you can ask it on the mailing lists. You may want to consult the rasdaman documentation in addition.

Download
1. Cannot download from git repository
Installation
PostgreSQL
Administration
1. stop_rasdaman.sh fails to stop rasdaman
Rasql queries
rasdaview
1. rasdaview returns an error "no suitable servers started", although I have servers available
Rasj and Java
1. Overflow when unsigned bytes are retrieved with rasj
2. I get class version incompatibility error messages
WCPS / petascope
Data formats
Helping

Download

Cannot download from git repository

Probably your provider does not allow port 9418 which git uses. See the download page for alternatives.

Installation

Using customized gdal-java JNI with petascope

By default, gdal-java JNI folder exists at /usr/lib/jni (Ubuntu) and /usr/lib/java/gdal (Centos). If you use a customized gdal-java JNI folder (e.g: /opt/lib/jni), then you need to configure it manually.

For system Tomcat (e.g: tomcat 8), edit: /etc/default/tomcat8 and append JNI library path as below:
```
   JAVA_OPTS="... -Djava.library.path=/opt/lib/jni"
```

For embedded Tomcat, you can set an environment variable for JNI library path before running petascope, e.g:

    export LD_LIBRARY_PATH="/opt/lib/jni"
    java -jar rasdaman.war

    Or add the JNI library path directly
    java -jar -Djava.library.path=/opt/lib/jni rasdaman.war

Cannot import data with rasdaman 9.8.x on Ubuntu 18.04

On Ubuntu 18.04 when rasdaman 9.8.x is deployed in Tomcat 9 with the default system configuration, wcst_import cannot import data anymore with the error as below from petascope:

The URL provided in the coverageRef parameter is malformed.

Fixing this requires allowing Tomcat to use the normal system /tmp directory:

edit /lib/systemd/system/tomcat9.service and set PrivateTmp=yes to PrivateTmp=no
sudo systemctl daemon-reload
sudo service tomcat9 restart

Petascope cannot connect to rasdaman in Centos 7

Likely the problem is that the tomcat user running petascope (/var/lib/tomcat/webapps/rasdaman.war) doesn't have permission to connect to ports 7001-7010 for rasmgr and rasservers (specified in rasmgr.conf).

ERROR [08:58:34] ApplicationMain@105: Error when initializing petascope's configurations
petascope.exceptions.PetascopeException: RasManager Error: Could not connect to RasServer .
        at org.rasdaman.ApplicationMain.placeholderConfigurer(ApplicationMain.java:104)

There are two ways to fix this issue:

Disable SELinux temporarily with setenforce 0 or permanently by editing /etc/sysconfig/selinux and rebooting (more details here).

Allow ports 7001-7010 to be accessible for http with these commands (more details):

# to install the semanage tool
sudo yum install policycoreutils-python

# allow ports to be accessed
sudo semanage port -a -t http_port_t -p tcp 7001-7010

I get C++ compilation errors.

Make sure you use the proper compiler version (see RequiredPackages). GNU folks have a relaxed view on backwards compatibility even with minor release changes, and we have experienced problems related to this in the past.

I get protobuf compilation errors.

Make sure to remove system protobuf/grpc packages, as well as any manually installed libraries/binaries in /usr (look for them with find /usr -name '*proto*'). See ticket #1249 for related information.

On a 64bit architecture I get linkage errors (e.g., when linking GDAL with the rasdaman driver) like "relocation R_X86_64_32 against `.bss' can not be used"

Make sure you use the proper compiler version. GNU folks have a relaxed view on backwards compatibility even with minor release changes, and we have experienced problems related to this in the past.
Retry cmake with -DENABLE_PIC=ON

rview doesn't work in 64 bit systems

If you try to run rview and you get one of the errors:

./rview: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory

./rview: error while loading shared libraries: libXp.so.6: wrong ELF class: ELFCLASS64

then you are trying to run rview with libXp.so.6 64 bit version, and rview doesn't work with it. rview is a deprecated application and, as such, one should not expect any future workaround for this problem. You can, however, run rview in 64 bits system, if you install the libXp.so.6 32 bit version. To do this:

add repository for 32-bit systems (optional, only necessary in more recent OS where ia32-libs is not available by default, e.g. Ubuntu 64 bits 13.10 or higher):
```
$ sudo gedit /etc/apt/sources.list
```
add following line in the end of file:
```
deb  http://cz.archive.ubuntu.com/ubuntu raring main universe
```
install ia32-libs:
```
$ sudo apt-get update
$ sudo apt-get install ia32-libs
```
(this will install libXp.so.6, among other 32-bit libraries)

rview works, now, but if not, you can link the 32 bit library (/usr/lib32) to /usr/lib.

Note: Installing lib32z1, lib32ncurses5 or lib32bz2-1.0 libs DOES NOT install libXp.so.6

I get Java compilation errors.

If, during make, you get errors like this:

javac -d bin src/org/odmg/*.java src/rasj/*.java src/rasj/global/*.java src/rasj/clientcommhttp/*.java src/rasj/odmg/*.java src/rasj/rnp/*.java
Exception in thread "main" java.lang.NoSuchFieldError: java/util/zip/Inflater.strm
...many more lines...

then make sure you use the SUN Java 6 SDK and not the IBM one (which seems to come preinstalled on some distributions, and is buggy).

When installing rasdaman on CentOS 7 I get processing dependency errors on grib

Phenomenon:

--> Processing Dependency: grib_api-devel for package: rasdaman-9.8.0-14.x86_64
Package grib_api-devel is obsoleted by eccodes-devel, but obsoleting package does not provide for requirements
--> Processing Dependency: grib_api for package: rasdaman-9.8.0-14.x86_64
Package grib_api-1.12.3-5.el7.x86_64 is obsoleted by eccodes-2.9.2-1.el7.x86_64 which is already installed
---> Package xml-commons-apis.noarch 0:1.4.01-16.el7 will be installed
---> Package xml-commons-resolver.noarch 0:1.2-15.el7 will be installed
--> Finished Dependency Resolution
Error: Package: rasdaman-9.8.0-14.x86_64 (rasdaman)
           Requires: grib_api
           Available: grib_api-1.12.3-5.el7.x86_64 (epel)
               grib_api = 1.12.3-5.el7
Error: Package: rasdaman-9.8.0-14.x86_64 (rasdaman)
           Requires: grib_api-devel
           Available: grib_api-devel-1.12.3-5.el7.x86_64 (epel)
               grib_api-devel = 1.12.3-5.el7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles –nodigest

Remedy: Ignore the obsoleting packages and use the old version of grib:

yum install --exclude=eccodes* rasdaman

If this fails, alternatively try

yum install --setopt=obsoletes=0 rasdaman

After installing I cannot find the documentation in the install directory

To include PDF and HTML documentation in the build process, which takes some time to build, you have to specify explicitly -DGENERATE_DOCS=ON when running cmake.

How to reinitialize RASBASE

If you want to drop all data stored in rasdaman quickly for some reason:

SQLite/Filestorage (default):

stop_rasdaman.sh
Let RASDATA be the directory specified in the host -connect option in rasmgr.conf, likely /opt/rasdaman/data; make sure it is indeed this directory (ls $RASDATA/ should show RASBASE, TILES/, TRANSACTIONS/).
Drop the data: rm -rf $RASDATA/*
Initialize a brand new database: create_db.sh
start_rasdaman.sh

PostgreSQL:

stop_rasdaman.sh
Drop the data: dropdb RASBASE
Initialize a brand new database: create_db.sh
start_rasdaman.sh

Error executing pip3 install —user GDAL==2.2.3, fatal error: cpl_port.h: No such file or directory

Look for this header in your system using the find command from root and then include it as a global option with pip3. Generally, it should be at /usr/include/gdal.

   $ find . -name cpl_port.h
   $ sudo pip3 install --user --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==2.2.3

PostgreSQL

I get a link error, function ECPGget_connection cannot be found

What appears to have happened is that in PostgreSQL 8.3.0 they removed ECPGget_connection and they have reintroduced it by 8.3.7. So PostgreSQL versions 8.3.0 up to (but excluding) 8.3.7 cannot be used.

I want to use a different version of PostgreSQL. Is that possible?

You need to include a different JDBC postgresql driver (as a JAR) in the libraries folder and update the Makefile as needed. Then you can try out another PostgreSQL version, it may or may not work. Let us know your experiences to update this information!

When running the script "create_db.sh", I get ''AdminIf related error

…more specifically, Creating base RASBASE…Warning/error in AdminIf Connect: SQLSTATE: 08001 SQLCODE: -402

There are several possible reasons:

Execution permissions. Run the script as the user owning the postgresql processes. Use only that user for further work with rasdaman (see Installation Guide). If you use Ubuntu and installed postgres with apt-get, then logging in as user postgres is an option; I personally like to have a separate user rasdaman for this purpose where I also can set paths etc. to the executables once and for all. (The postgres db does not allow full access to rasdaman tools, because they do not run under the administrative user that launched the db server process.)

Alternatively, make sure that the operating system user under which you run create_db.sh can be authenticated via the ident method, which can be set in pg_hba.conf. E.g. if user is "rasdaman", then in pg_hba.conf you should have at least:
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
local   all         rasdaman                          ident

Blocked incoming connections. Another cause of the problem might be that postgres does not allow incoming connections. If you have Postgres v8 or more recent, modify file "postgresl.conf" and set (uncomment) the option
```
listen_addresses = 'localhost'
```
TCP-IP connection. If you have Postgres v7, set (uncomment) the following option:
```
tcpip_socket = true
```

I get "wrong connect string" error when first running rasdl

When I first try to run rasdl (or the rasdaman server, for that matter) against PostgreSQL after installation, the log says "server ok" but "wrong connect string"

There are several possible reasons:
1. Port: Running a database cluster on a separate port requires proper setting of environment variable PGPORT to allow for communication between rasdaman and PostgreSQL. (thanks to Mario Uhlig!)
2. Socket directory: using '/var/lock/postgresql' as socket directory makes rasdl fail for a reason still to be found. Switching back to the default socket dir '/var/run/postgresql' resolves the issue. (thanks to Alexander Herzig!)

Deleting big collection imported rasimport results in 'Serializable database error…'

The rasgeo toolbox which contains rasimport is phased out and de-supported. Use the standards-based WCS-T suite for coverage management, in particular: the OGC WCS-T DeleteCoverage request for a clean deletion of coverages.

Administration

stop_rasdaman.sh fails to stop rasdaman

If stop_rasdaman.sh, sudo service rasdaman stop, or sudo systemctl stop rasdaman.service fail to stop rasdaman, then it's necessary to do it manually. You have to manually kill first the rasmgr process and then the rasserver processes:

sudo pkill rasmgr
- check: if pgrep rasmgr still prints a pid, then it has to be killed with sudo pkill -9 rasmgr
repeat the same with rasserver: sudo pkill rasserver
- check: if pgrep rasserver still prints some pids, then it has to be killed with sudo pkill -9 rasserver

Afterwards you can start rasdaman again.

Rasql queries

I get syntax error unexpected name when using a floating point number

Floating point numbers must have a leading zero, for example; "0.9" instead of ".9".

My query which uses log doesn't work

for c in (rgb) return encode(log(c.0), "jpeg")

The log function needs positive numbers as arguments, and will return -Infinity if the argument is 0. You need to make sure that the input does not contain any zeros. You can check that using the query
```
for c in (rgb) return count(c.0 = 0)
```
Solution (depending on your overall formula): Add 1 to the input:
```
for c in (rgb) return encode(log(c.0+1), "jpeg")
```

My insert query produces error 821 (cell type mismatch)

… although I provide a TIFF file with correct pixel type:

insert into FloatCollection values inv_tiff( $1 )

You need to provide an explicit cast (see section 9.8 in the Rasql Guide); a correct phrasing of this query is:
```
insert into FloatCollection values (float) inv_tiff( $1 )
```

How do I position a replacement array when updating a collection?

In an update statement, the slice coordinates are specified with the set operator, however the 2D position of the replacement array is defined by the spatial domain of the array, which by default is [0:width,0:height], and can be modified with the shift operator or the —mdddomain option of rasql. For example a query to update the 3D collection C with a 2D image of size 255x210 that should replace the region at interval [10, 100:355, 100:310]:
```
update C set [10, *:*, *:*] assign shift(inv_tiff($1), [100, 100])
```
See docs for more details.

Note that the shift vector applies to the second and third dimension (slice defined in the first dimension).

How does the Storage Layout Language work?

Check the Storage Layout Language paper for more details, while the manual is not yet updated.

"Error: 100 Exception: Internal error: DL parse error."

What does this error returned by rasql mean?

You have likely inserted types to the database which names start with a rasdl reserved word. This is a known issue, see #121 for a workaround.

How can I import large images that don't fit into main memory?

Check this short guide on partial updates in addition to the documentation of the update statement in the Query Language Guide for rasql.

How to use the encode() function to export data

Check this short unofficial documentation until the Query Language Guide is updated.

rasdaview

rasdaview returns an error "no suitable servers started", although I have servers available

For historical reasons, rasdaview needs an RPC type rasserver running. See the Installation Guide for what this means and how such a server can be started.

Rasj and Java

Overflow when unsigned bytes are retrieved with rasj

If I send the query select max_cells(c.0) from rgb as c via command line (rasql utility), then the result is 255 as expected; if I send this query via RasJ, then the result is -1.

Java does not have an 8-bit unsigned data type. Byte is a signed 8 bit and hence 255 is represented as -1. What you can do is to transform bytes to char by adding 2⁸ if the 2⁷ bit is set. Tested, works. Alternatively, cast to a larger data type on server side:
```
select (long) max_cells(c.0) from rgb as c
```

I get class version incompatibility error messages

I get weird error messages from the Java runtime system which seems to hint at class version incompatibilities, such as "Class version error" or "Class not found".

There might be a conflict with the locally provided libraries, to be found at directory $PETASCOPE/lib. Check them, and if you find a mismatch then remove them and use the resp. libraries of your local installation instead. Alternatively, consider upgrading your Java SDK to version 1.6.

Check that the versions javac -version and java -version match, otherwise you may get an error Exception in thread "main" java.lang.UnsupportedClassVersionError: ..: Unsupported major.minor version 51.0

WCPS / petascope

Mixing trimming and slicing in WCPS fails

A workaround until this ticket is fixed is to separate the two operations, e.g. instead of
```
c[t(5),x(0:20),y(0:20)]
```
use
```
(c[x(0:20),y(0:20)])[t(5)]
```

"Requested more data than the server can handle at once"

If a WCS/WCPS query fails with a "Requested more data than the server can handle at once", then it's likely that you need to increase the maximum amount of RAM that Tomcat is allowed to allocate.

On most systems this can be done by editing a /etc/default/tomcat* file. Find the JAVA_OPTS option and set -Xmx to the desired value (e.g. to allow 4GB RAM, use -Xmx4G.

"OS provides only python3 GDAL package"

Currently wcst_import.sh works with python2 only and has various dependencies which must be python2-compatible as well.

If the OS doesn't provide a suitable package, it can be installed with pip, e.g.

pip install --user GDAL==2.4.3

Petascope fails to start with UnsatisfiedLinkError

Sometimes you may see an exception like the following in the tomcat log:

 ERROR [14:03:40] SpringApplication@815: Application startup failed
org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'applicationMain': Invocation of init method failed; 
nested exception is java.lang.UnsatisfiedLinkError: 'void org.gdal.gdal.gdalJNI.AllRegister()'
...

This happens when you redeploy petascope, i.e. copy an updated version of rasdaman.war to Tomcat's webapps directory, but do not restart Tomcat subsequently.

Solution: when redeploying rasdaman, make sure to restart Tomcat. This is required because petascope uses a native library from GDAL; further information in the Tomcat docs.

Modulor operation in WCPS

In WCPS, % character is used for decoding UTF-8 encoded characters, for example, character [ is encoded as %5B.

In order to perform a modulor operation, one needs to use binary function mod(a, b). For example:

    for $c in (test_mr)
    return encode( mod(5, 3) , "csv" )

Data formats

HDF4 is said to be supported, but rasdaman reports an error

…when trying to use hdf() or inv_hdf() - what can I do?

HDF4 needs to be compiled in explicitly. Re-generate with cmake option -DUSE_HDF4=ON.

NetCDF is said to be supported, but rasdaman reports an error

…when trying to use netcdf() or inv_netcdf() - what can I do?

NetCDF needs to be compiled in explicitly. Re-generate rasdaman with cmake option -DUSE_NETCDF=ON.

Supported NetCDF library version is 4.0+. It provides netcdf4 (including HDF, classic, and 64) and netcdf3 (as the API is backwards compatible). Limitations: only atomic cell types, but no struct types are supported at this moment; and metadata are not preserved (they have to be stored separately in PostgreSQL tables)

The NetCDF generated by rasdaman is not recognized by NetCDF tools

THe issue probably has to do with the pixel type. The 'char' type in rasdaman corresponds with the 'byte' type in netcdf. In netcdf, the 'char' type is used for the 8-bit characters intended for representing text. Therefore, the netcdf(char) result is in binary instead of ASCII. However, the netcdf(char) result is validated against the compliance checker. Note: netcdf(float) works fine.

Helping

I want to submit a patch, how do I do this?

Set your credentials (ie, your name, YOUR_FULL_NAME, and email address, EMAIL):

$ git config --global user.name "YOUR_FULL_NAME"
$ git config --global user.email EMAIL

rasdaman must be configured and compiled with --enable-strict to make sure that your patch doesn't introduce new warnings.
When you commit your changes to your local copy put a meaningful message:
```
$ git commit file1 file2 ... -m "ticket:NUMBER - commit message"
```
The subject of the patch must start with "ticket:NUMBER - ", where NUMBER is a valid ticket number on this trac. For example this is a valid subject: "ticket:231 - subject message…". To manually fix the subject after you have created the patch, edit the patch manually and adapt the Subject: line, e.g. from
```
Subject: [PATCH] invalid subject
```
to the below (make sure to not remove the [PATCH] part!)
```
Subject: [PATCH] ticket:NUMBER - commit message
```
It's best to always specify exactly the files that you want to commit, instead of committing everything with the -a option, so that you avoid submitting unnecessary changes in your patch.
Then you can make the patch, e.g:
```
$ git format-patch HEAD~1 
```
Finally you can submit the generated patch to the rasdaman patchmanager

Important: avoid submitting generated files like configure, Makefile.in, etc. in your patch, unless the corresponding source (configure.ac, Makefile.am, …) has been really modified. Make sure that all submitted sources have an appropriate licence header.

How to report a bug/problem

It's best if you report bugs by filling a new ticket. It's always good to include as much as possible information of the setup in which the bug occurred, so we can reproduce it. In general you should include:

gcc —version
OS and architecture
how was rasdaman configured/compiled
versions of PostgreSQL's or any other libraries that may be relevant
relevant parts of the server log files

To open a new ticket, you need an account in the trac system: please request one via one of our mailing lists.

Why do I have to accept the Rasdaman Contribution Agreement when providing a patch?

Because for legal reasons the project must have a trackable code provenance.

Last modified 14 months ago Last modified on Sep 7, 2023, 9:11:10 AM

Attachments (1)

sstdm2010.pdf (733.9 KB ) - added by Peter Baumann 13 years ago. Storage Layout Language description

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text