On Ubuntu 18.04 gdal jar should be version 2.x

[Dimitar Misev]

Currently in the rasdaman package it's 1.10

[Dimitar Misev]

I ran sudo pkg_test.sh stable myself in Ubuntu 18.04, and now it shows the gdal error here:

Caused by: java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.AllRegister()V
    at org.gdal.gdal.gdalJNI.AllRegister(Native Method)
    at org.gdal.gdal.gdal.AllRegister(gdal.java:551)
    at org.rasdaman.ApplicationMain.loadGdalLibrary(ApplicationMain.java:226)
    at org.rasdaman.ApplicationMain.postInit(ApplicationMain.java:376)

The library in petascope is now indeed gdal-2.3.0.jar. I think the problem is that libgdal-java in Ubuntu 18.04 is version 2.2.3, so gdal-2.3.0.jar is maybe too new for it?

I replaced gdal-2.3.0.jar with gdal-2.2.1.jar, and this fixed the link error.

[Dimitar Misev]

Replying to dmisev:

I replaced gdal-2.3.0.jar with gdal-2.2.1.jar, and this fixed the link error.

From ​https://mvnrepository.com/artifact/org.gdal/gdal/2.2.1

[Dimitar Misev]

It's still an issue in the nightly package tests:

 ERROR [2019-07-27 06:46:19] SpringApplication@815: Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'applicationMain': Invocation of init method failed; nested exception is java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.AllRegister()V
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
	at org.springframework.boot.web.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:151)
	at org.springframework.boot.web.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:131)
	at org.springframework.boot.web.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:86)
	at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:169)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5225)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:980)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1852)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.UnsatisfiedLinkError: org.gdal.gdal.gdalJNI.AllRegister()V
	at org.gdal.gdal.gdalJNI.AllRegister(Native Method)
	at org.gdal.gdal.gdal.AllRegister(gdal.java:551)
	at org.rasdaman.ApplicationMain.loadGdalLibrary(ApplicationMain.java:226)
	at org.rasdaman.ApplicationMain.postInit(ApplicationMain.java:376)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
	... 31 more

[Dimitar Misev]

I attached the tomcat logs​

[Bang Pham Huu]

Replying to dmisev:

I attached the tomcat logs​

I think the correct one is not the version 2.3.0 but the version 2.2.1 (version 2.3.0 didn't work in the VM I tested) ​http://rasdaman.org/browser/cmake/modules/FindGdalJava.cmake#L30

version 2.2.1 should work for both libgdalalljni.so.20 and libgdaljni.so.20

[Dimitar Misev]

In my tests 2.2.1 did not work for both, only for libgdaljni.so.20

[Bang Pham Huu]

I tested in the Vagrant rasdaman 18.04 with nightly test from rasdaman installer

sudo ./pkg_test.sh nightly

After installation, it has these GDAL libary files

vagrant@ubuntu-bionic:~/rasdaman_installer$ ls -ltr /usr/lib/jni/
total 1072
-rw-r--r-- 1 root root  71600 Feb  6  2018 libosrjni.so.20
-rw-r--r-- 1 root root  71600 Feb  6  2018 libosrjni.so
-rw-r--r-- 1 root root 149424 Feb  6  2018 libogrjni.so.20
-rw-r--r-- 1 root root 149424 Feb  6  2018 libogrjni.so
-rw-r--r-- 1 root root  26544 Feb  6  2018 libgnmjni.so.20
-rw-r--r-- 1 root root  26544 Feb  6  2018 libgnmjni.so
-rw-r--r-- 1 root root 264112 Feb  6  2018 libgdaljni.so.20
-rw-r--r-- 1 root root 264112 Feb  6  2018 libgdaljni.so
-rw-r--r-- 1 root root  26240 Feb  6  2018 libgdalconstjni.so.20
-rw-r--r-- 1 root root  26240 Feb  6  2018 libgdalconstjni.so

gdalinfo --version
GDAL 2.2.3, released 2017/11/20

But rasdaman.war contains gdal-2.3.0.jar instead of gdal-2.2.1.jar. So, I will need to check cmake in the VM.

[Bang Pham Huu]

In the VM, I cloned rasdaman and built manually, it showed correct gdal-2.2.1.jar in rasdaman.war.

cmake ../rasdaman -DCMAKE_INSTALL_PREFIX=/home/vagrant/rasdaman_community/install -DWAR_DIR=/var/lib/tomcat8/webapps/ -DUSE_NETCDF=ON -DUSE_GRIB=ON  -DCMAKE_BUILD_TYPE="Debug" -DENABLE_DEBUG_SYMBOLS=ON -DGENERATE_DOCS=OFF

cd /home/vagrant/build/applications/petascope
make

# Maven command from make
/usr/bin/mvn -q -B package -f pom.xml 
-DbuildDir=/home/vagrant/build/applications/petascope/ 
-DgdalJavaVersion=2.2.1 -Dversion=9.8.0

So, the problem must be due to rasdaman.war in nightly build package (somehow it was built with gdal-2.3.0.jar, but I don't know how it was built.

[Dimitar Misev]

It's built with sudo ./pkg_build.sh nightly noupload

[Bang Pham Huu]

vagrant@rasdaman:~/rasdaman_installer$ sudo ./pkg_build.sh nightly noupload
pkg_build.sh: Checking OS...
pkg_build.sh: Done, detected OS 'bionic'.

Review the installation settings:

Install path: /opt/rasdaman/
User: rasdaman
Database: sqlite, /opt/rasdaman/data/
Repository: git://rasdaman.org/rasdaman.git
Version: master
Debug: False
Protocol: rasnet
Install webapps: True
  Petascopedb url: jdbc:postgresql://localhost:5432/petascopedb
  Petascopedb user: petauser
  Deployment: external
  Webapps path: /var/lib/tomcat8/webapps
  Webapps logs: /var/log/tomcat8
Insert demos: True
Run systemtest: False
Generate package: True
  Profile path: profiles/package/install.toml
  Name: rasdaman
  Description: Rasdaman is the leading Array Database for flexible, scalable analytics of massive multi-dimensional array (raster) data, such as spatio-temporal datacubes.
  Version: 9.8.0
  Iteration: 114
  Vendor: rasdaman
  Licence: GPLv3
  Category: devel
  Maintainer: Dimitar Misev <misev@rasdaman.com>
  URL: http://rasdaman.org
Validating system...
System was validated.
Installing third party dependencies on Ubuntu 18.04...
Installing packages: make libtool gawk autoconf automake bison flex git g++ unzip libpng-dev libjpeg-dev libboost-filesystem-dev libboost-thread-dev libboost-system-dev libtiff-dev libgdal-dev zlib1g-dev libffi-dev libboost-dev libnetcdf-dev libedit-dev libreadline-dev libsqlite3-dev libgrib-api-dev libgrib2c-dev curl libdw-dev libssl1.0-dev openjdk-8-jdk maven ant libgdal-java cmake ccache ruby-dev ruby rsync python-pip python-setuptools latexmk texlive-latex-base texlive-fonts-recommended texlive-latex-extra doxygen

Packages installed successfully.
Installing pip packages ['sphinx', 'sphinx_rtd_theme']... 
Pip packages installed successfully.
Checking user rasdaman....
rasdaman user does not exist. Creating...
User 'rasdaman' created successfully.
User rasdaman checked successfully.
Creating directory structure...
Directory structure created successfully.
Downloading a supported version of cmake...
Cmake installed in '/opt/rasdaman/third_party'...
Setting up the PATH environment variable in '/etc/profile.d/rasdaman.sh'...
The PATH environment variable was set up successfully.
Third party dependencies installed.
Retrieving rasdaman...
Cloning rasdaman repository at /opt/rasdaman/source/.
Rasdaman retrieved.
Building rasdaman...
Configuring rasdaman with cmake...
Rasdaman was configured with params: ['', '-DUSE_JPEG=ON', '-DUSE_TIFF=ON', '-DCMAKE_BUILD_TYPE=Release', '-DUSE_GRIB=ON', u'-DFILE_DATA_DIR=/opt/rasdaman/data/', '-DUSE_NETCDF=ON', u'-DCMAKE_INSTALL_PREFIX=/opt/rasdaman/', '-DUSE_PNG=ON', '-DUSE_HDF4=ON'].
Compiling rasdaman with make -j1 (this will take some time)...
Rasdaman was compiled successfully.
Installing rasdaman...
Rasdaman installed successfully.
Rasdaman built.
Preparing rasdaman...
Rasdaman prepared.
Finalizing installation...
Installation finalized.
Testing rasdaman...
TEST STATUS: SUCCESS
Testing done.
Packaging...
Packaging rasdaman as deb package...
  Generated preinst script at '/tmp/rasdaman-packager/opt/rasdaman/share/rasdaman/installer/preinst.sh'.
  Generated postinst script at '/tmp/rasdaman-packager/opt/rasdaman/share/rasdaman/installer/postinst.sh'.
Rasdaman packaged successfully at: /tmp/rasdaman-packager/package
Packaging done.
Rasdaman installation finished in 52.7m

In rasdaman.war, it has gdal-2.2.1 ​https://imgur.com/a/8MpDp1C

[Dimitar Misev]

Still fails, the latest nightly package from today still has gdal-2.3.0.jar.

This is the Vagrantfile used on jenkins:

Vagrant.configure(2) do |config|

  #config.vm.box = "ubuntu/bionic64"
  config.vm.box = "rasdaman/ubuntu1804"

  config.vm.box_check_update = true

  config.vm.synced_folder ".", "/vagrant", type: "rsync"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "5000"
    vb.cpus = 1
    vb.customize [ "modifyvm", :id, "--uartmode1", "disconnected" ]                                                                                   
  end

  config.vm.provision "fix-no-tty", type: "shell" do |s|
    s.privileged = true
    s.inline = "sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n \\|\\| true/' /root/.profile"
  end
  config.vm.provision "disable-apt-periodic-updates", type: "shell" do |s|
    s.privileged = true
    s.inline = "echo 'APT::Periodic::Enable \"0\";' > /etc/apt/apt.conf.d/02periodic"
  end

  config.vm.provision "shell", inline: <<-SHELL
    # set the default locale for testing
    echo 'LANGUAGE="en_US.UTF-8"' >> /etc/default/locale
    echo 'LC_ALL="en_US.UTF-8"' >> /etc/default/locale
   SHELL
end

and box version:

==> default: Box 'rasdaman/ubuntu1804' (v0.0.2) is running the latest version.

[Bang Pham Huu]

This config.vm.box = "rasdaman/ubuntu1804" already has libgdal-java installed by default with these files.

vagrant@ubuntu-bionic:~$ ls -ltr /usr/lib/jni
total 1072
-rw-r--r-- 1 root root  71600 Feb  6  2018 libosrjni.so.20
-rw-r--r-- 1 root root  71600 Feb  6  2018 libosrjni.so
-rw-r--r-- 1 root root 149424 Feb  6  2018 libogrjni.so.20
-rw-r--r-- 1 root root 149424 Feb  6  2018 libogrjni.so
-rw-r--r-- 1 root root  26544 Feb  6  2018 libgnmjni.so.20
-rw-r--r-- 1 root root  26544 Feb  6  2018 libgnmjni.so
-rw-r--r-- 1 root root 264112 Feb  6  2018 libgdaljni.so.20
-rw-r--r-- 1 root root 264112 Feb  6  2018 libgdaljni.so
-rw-r--r-- 1 root root  26240 Feb  6  2018 libgdalconstjni.so.20
-rw-r--r-- 1 root root  26240 Feb  6  2018 libgdalconstjni.so

How is rasdaman_installer copied to the VM in Jenkins?

[Bang Pham Huu]

I tested again on the Vagrant file as same as Jenkin's, check ​http://rasdaman.org/attachment/ticket/2144/ubuntu1804-jenkins.zip for logs

The built package contains gdal-2.2.1.jar as the maven command to build was

...... 301s to build custom target in petascope: /usr/bin/mvn -q -B package -f pom.xml -DbuildDir=/opt/rasdaman/source/build/applications/petascope/ -DgdalJavaVersion=2.2.1 -Dversion=9.8.0

[Dimitar Misev]

Seems to have finally worked fine today, closing ticket.