wiki:InstallFromSource

Version 14 (modified by pbaumann, 21 months ago) (diff)

--

Compile rasdaman from Source

This is a compilation guide for rasdaman; it has been successfully tested on various Linux systems; for Mac OS X please consider this page? in addition.

Definitions

A couple of variables are used in this guide:

  • $USER refers to the Linux user id that will be used to install and manage the rasdaman server. It is recommended that this user is a separate account dedicated for this purpose, such as rasdaman.
    • To create a new user rasdaman for example: adduser rasdaman; switch to it from your regular user: sudo -u rasdaman -i
  • $RMANHOME is the target directory where rasdaman will be installed, by default: /usr/local/rasdaman.
    • $RMANHOME/bin contains the rasdaman programs and scripts
    • $RMANHOME/etc contains the configuration
    • $RMANHOME/log the server logs.
  • $RASDATA is the directory used in rasdaman for storing array data, when not stored in a PostgreSQL database.

Packages required

The following packages are required for rasdaman and need to be installed:

  • tools:
    • git -- needed to clone the rasdaman git repository
    • autoconf, automake, m4 -- for generating the configure script and makefiles needed to compile rasdaman
    • make, libtool, pkg-config -- general tools needed to run the configure script and compile rasdaman
    • flex, bison, g++, libstdc++ -- required for compilation of the C++ codebase
    • OpenJDK 6+ -- required for compilation of the Java code (Java client API, petascope OGC frontend)
  • general libraries:
    • libssl-dev, libncurses5-dev, libedit-dev, libboost-dev (v1.48+), libffi-dev -- required for various system tasks
    • libgdal-dev, libtiff-dev, libjpeg8-dev, libpng12-dev, libnetpbm10-dev -- required fro data format support
  • database stuff: Pick one option below for rasdaman storage:
    • libecpg-dev -- required for PostgreSQL (version 9.x is recommended; 8.3.0 to 8.3.6 known to not work) to hold rasdaman arrays and/or petascope geo metadata
    • libsqlite, libsqlite-dev, sqlite3 packages -- required for storing arrays in a file system directory and the rasdaman technical metadata in SQLite; see details; note that petascope currently requires PostgreSQL independently from the PostgreSQL / file system array decision - in other words: even if for the array engine you chose to not use PostgreSQL you currently still need to install it for storing the geo metadata making an array an OGC coverage)
  • optional packages (use --with-XXX in the configure step to activate use by rasdaman):
    • libhdf4g-dev -- required for HDF4 support, configure with --enable-hdf
    • netcdf4 -- required for NetCDF support, configure with --enable-netcdf
    • doxygen -- required for HTML system documentationm configure with --with-docs
    • grpc, protobuf -- required for the rasnet client/server protocol, configure with --enable-rasnet
    • Tomcat (or another suitable servlet container) -- required for running the petascope and SECORE servlets; default, changed to embedded jetty support with --enable-jetty
    • performance boosters and additional service components offered by rasdaman GmbH
  • geo data support (optional):
    • libecpg-dev -- required for PostgreSQL (version 9.x is recommended; 8.3.0 to 8.3.6 known to not work) to hold petascope geo metadata (this dependency to be removed in future)
    • python-dateutil python-lxml python-pip python-gdal python-glob2 python-magic (required by wcst_import, a tool for importing geo-referenced data into rasdaman/petascope)

Installation commands for the packages is depending on the platform used, here a guidance for some of the most frequently asked for:

  • CentOS:
    sudo yum install \
      git make automake autoconf libtool pkgconfig m4 \
      bison gcc gcc-c++ libedit-devel zlib-devel openssl-devel flex flex-devel boost-devel \
      gdal-devel libtiff-devel hdf-devel libjpeg-devel libpng12-devel netcdf-devel netcdf-cxx-devel libpng-devel netpbm-devel \
      postgresql-devel libsqlite-devel \
      gdal-python python-setuptools \
      java-1.7.0-openjdk-devel tomcat7
    sudo easy_install glob2
    
  • Debian 7 & 8:
    sudo apt-get install \
      git make automake autotools-dev libtool pkg-config m4 \
      bison flex g++ libncurses5-dev libedit-dev libboost-all-dev \
      libtiff-dev libgdal1-dev gdal-bin libnetpbm10-dev libjpeg-dev libpng-dev \
      postgresql postgresql-contrib libecpg-dev libsqlite-dev sqlite3 \
      python-dateutil python-lxml python-gdal python-setuptools \
      openjdk-7-jdk tomcat7
    sudo easy_install install glob2
    

Prerequisites

  1. Make sure that java and javac are installed and the same version:
    java -version
    javac -version
    
  2. Allow your user to deploy to the tomcat webapps directory (only if you (i) want to run the petascope or SECORE servlets and (ii) want to use tomcat, not the embedded jetty servlet container):
    sudo adduser $USER tomcat7
    
  3. If PostgreSQL support is needed (see options): Add user in postgres:
    sudo -u postgres createuser -s $USER
    
  4. If Tomcat is used: Increase the maximum heap space for Tomcat (>=1GB). Usually this can be done by setting -Xmx1024m in JAVA_OPTS in /etc/default/tomcat7 and then restarting tomcat with sudo service tomcat7 restart.

Getting the Source

The rasdaman source tree can be fetched from the repository. This includes a version control system, effectively making different revisions of the source code available. The following alternatives describe how to clone from the repository and use it:

  • Option 1: Get the most recent development version:
    git clone git://rasdaman.org/rasdaman.git # this creates subdirectory rasdaman/
    cd rasdaman/
    
  • Option 2: Select a tagged stable release. To activate a particular tagged version use its name prefixed with a "v" (lower case V !), see the example in line 3:
    git clone git://rasdaman.org/rasdaman.git # this creates subdirectory rasdaman/
    cd rasdaman/
    git checkout v9.1.0-beta2   # to obtain a particular tagged version
    

Compile and Install rasdaman

  1. Make sure to be logged in as the rasdaman user.
  2. Choose an installation directory (here: /home/rasdaman/install) and further settings. It is best to put these in your ~/.bashrc for automatic loading upon login; don't forget to reload it first time through source ~/.bashrc)
    • Debian 8:
      export RMANHOME=/home/rasdaman/install
      export RMANSRC=/home/rasdaman/rasdaman # rasdaman sources
      export RASDATA="$RMANHOME/data"
      export CATALINA_HOME=/var/lib/tomcat7
      export PATH=$PATH:$RMANHOME/bin:/usr/lib/postgresql/9.4/bin
      
    • CentOS 7:
      export RMANHOME=/home/rasdaman/install
      export RMANSRC=/home/rasdaman/rasdaman # rasdaman sources
      export RASDATA="$RMANHOME/data"
      export CATALINA_HOME=/var/lib/tomcat
      export PATH=$PATH:$RMANHOME/bin
      
  3. Make sure to load the above variables: source ~/.bashrc
  4. Configure rasdaman:
    # generate the configure script and Makefile templates
    cd "$RMANSRC"
    autoreconf -fi
    
    # run the configure script to generate the Makefiles and check if the required 
    # packages have been installed;
    # run "./configure --help" to see a full list of the available options
    ./configure --prefix="$RMANHOME" --with-wardir=$CATALINA_HOME/webapps/ --with-netcdf \
                --with-default-basedb=sqlite --with-filedatadir="$RASDATA"
    
    • --prefix specifies the absolute path where rasdaman will be installed;
    • --with-wardir specifies that petascope (rasdaman.war) and SECORE (def.war) will be installed in $CATALINA_HOME/webapps/;
    • --with-netcdf says that we want the netcdf support in rasdaman to be enabled;
    • For more details on the rasdaman storage backends (--with-default-basedb=sqlite --with-filedatadir="$RASDATA") see this page.
  5. To compile the rasdaman system (for speedup, you may want to add a parallelization option such as -j2):
    make
    
    • The following make targets are available:
      • all: compile and link all (default)
      • install: install system to target location (and compile before, if not up-to date)
      • clean: remove all object and library files generated
  6. Install
    mkdir -p $RMANHOME
    make install
    

Rasdaman and petascope are now installed; in the following sections we will initialize and configure them.

Initialize rasdaman

  1. Initialize the rasdaman schema:
    create_db.sh
    update_db.sh
    
  2. Start the rasdaman server:
    start_rasdaman.sh
    
  3. Insert demo data:
    rasdaman_insertdemo.sh localhost 7001 $RMANHOME/share/rasdaman/examples/images/ rasadmin rasadmin
    
  4. Check if demo data have been inserted properly (you should see a list of collection names, such as mr, mr2, anthur):
    rasql -q 'select c from RAS_COLLECTIONNAMES as c' --out string
    
  5. Optionally, run the systemtests to perform a deep checking of the installation:
    cd systemtests
    make check
    

Congratulations! With this, rasdaman is running and ready for use. In $RMANHOME/etc there are a couple of configuration files that can be modified to adjust the rasdaman server, like logging, host name, etc. Each change requires restarting the rasdaman server:

stop_rasdaman.sh
start_rasdaman.sh

Initialize petascope

Petascope is the geo Web service frontend of rasdaman. It adds geo semantics on top of arrays, thereby enabling regular and irregular grids based on the OGC coverage standards. Following successful deployment, petascope accepts OGC W*S requests at URL http://localhost:8080/rasdaman/ows.

For more details, see the user guide and developer guide.

PostgreSQL

To implement the geo semantics, petascope uses a relational database for the geo-related metadata. Currently, only PostgreSQL is supported; for the future, support of SQLite is planned (in parallel to the rasdaman array engine). This is how to set up PostgreSQL for use by petascope:

  1. Prepare PostgreSQL for use with rasdaman
  1. Add a user to PostgreSQL for petascope:
    sudo -u postgres createuser -s petauser -P
    
  2. Set the metadata_user and metadata_pass variables in $RMANHOME/etc/petascope.properties to the above user and password.
  3. make sure PostgreSQL needs to be running
  4. Initialize the petascope database schema:
    update_petascopedb.sh
    

Running the Servlet

As petascope is implemented as a Java servlet it needs a servlet container for its operations. Two alternatives are available:

  • separately install some servlet container, like Tomcat, and deploy petascope there; In case of problems, consult the servlet container log files (such as /var/log/tomcat7/catalina.out on Debian 8 and /usr/share/tomcat/petascope.log on CentOS 7).
  • run the petascope service directly, by way of the embedded servlet container, jetty; More information on how to use this option can be found on this page.

Initialize SECORE

SECORE (Semantic Coordinate Reference System Resolver) is a service that maps CRS URLs to CRS definitions. This component, which is part of the standard rasdaman distribution, is used by the Open Geospatial Consortium (OGC) for operating their official CRS resolver.

SECORE, being a servlet like petascope, gets installed automatically unless --disable-java is specified at configure. The deployment directory of all war files can be set during the configure step with the --with-wardir=DIR option.

For further details on SECORE management, security and troubleshooting see the administration and developer guide pages.

Updating an existing installation

For staying up-to-date with the latest fixes, you can update the repository and recompile at any time. Assuming you have cloned the rasdaman repository from git in directory ~/rasdaman, you need to pull the latest commits, recompile and reinstall rasdaman:

cd ~/rasdaman

git pull           # get latest commits
make clean         # clean old compilation artefacts
make               # compile the updated code

# it is recommended to stop rasdaman/Tomcat before installing the new executables
sudo service tomcat7 stop
stop_rasdaman.sh

# install rasdaman/petascope, and update the database schemas
make install
update_db.sh
update_petascopedb.sh

# start services again
start_rasdaman.sh
sudo service tomcat7 start

Next steps

After successful compilation you may want to proceed to check the documentation, FAQ, and known issues list.

For more in depth details on rasdaman's installation and configuration, see rasdaman installation guide and PostgreSQL configuration hints.