petascope and secore configuration files

[Dimitar Misev]

The logging configuration files for these components are:

• petascope: $RMANHOME/etc/log4j.properties
• secore: $CATALINA_HOME/webapps/def/etc/log4j.properties

The general configuration files are:

• petascope: $RMANHOME/etc/petascope.properties
• secore: $CATALINA_HOME/webapps/def/etc/secore.conf

It would be good to harmonize this state of affairs.. I suggest that everything goes in $RMANHOME/etc like this, such that the logging configuration is merged into the main configuration file.

component	config file
petascope	petascope.properties
secore	secore.properties

Any thoughts?

[Peter Baumann]

+1. With defaults where all log goes into $RMANHOME/log

PS: we are in the root FS again, so the hint of doing a symlink to another partition for the logs is as appropriate as ever.

[Bang Pham Huu]

would be better if you create subdirectory for log-configuration (/etc/log) and general-configuration (/etc/conf). Now, it is quite difficult to see the list of configuration file in $RMANHOME/etc.

-rwxr-xr-x. 1 rasdaman rasdaman 1669 Oct 15 11:36 log4j.properties
-rwxr-xr-x. 1 rasdaman rasdaman 2520 Oct 15 11:35 log-client.conf
-rwxr-xr-x. 1 rasdaman rasdaman 2648 Oct 15 11:35 log-rasmgr.conf
-rwxr-xr-x. 1 rasdaman rasdaman 2648 Oct 15 11:35 log-server.conf
-rwxr-xr-x. 1 root     root     9459 Oct 16 08:40 petascope.properties
-rwxr-xr-x. 1 rasdaman rasdaman 9459 Oct 16 08:32 petascope.properties.10-16-2016_08-32-04.bak
-rwxr-xr-x. 1 rasdaman rasdaman 5048 Oct 15 11:35 rasmgr.conf
-rwxr-xr-x. 1 rasdaman rasdaman  552 Oct 15 11:36 wms_service.properties

Should be not problem when merging log4j with petascope.properties, btw, don't we have secore.log (I know we have petascope.log and rasdaman) ?

[Peter Baumann]

I wouldn't do that - if we need subdirs for just config files then we are getting way too complicated. Rather, let us think about simplifying the system. This seems to be necessary periodically (we were in similar situations before).

But always have in mind that conf files are externally visible behavior of the system, so changes face a high barrier and easily require a major release change.

[Dimitar Misev]

I'm not in favor of the subdir idea. In any case there can be a single petascope.properties and secore.properties that include the logging configuration as well.

We can automatically migrate to the new conf file structure, and still support the old case so that it's backwards compatible. We don't need a major release in that case I'd say.

[Peter Baumann]

not sure that there is no impact - for example, some people parse log files, so those must have a frozen structure as well. Might be someone similarly does autoconfig scripts on log files, we never know.

[Dimitar Misev]

These are the configuration files, I don't think anyone is parsing them.. we can ask on the mailing lists just in case. The secore ones surely not, currently they are quite hidden in the WAR file.

[Dimitar Misev]

First step is to move the secore configuration to $RMANHOME/etc:

1. move $CATALINA_HOME/webapps/def/etc/secore.conf to $RMANHOME/etc/secore.properties
2. merge the contents of $CATALINA_HOME/webapps/def/etc/log4j.properties inside the new $RMANHOME/etc/secore.properties
3. change the logging config to output to $RMANHOME/log/secore.log like it's done for petascope

[Dimitar Misev]

There is a problem when configuring logging to $RMANHOME/log/petascope.log or $RMANHOME/log/secore.log.

The log is written by the tomcat system user running Tomcat. This user will usually not have write access in $RMANHOME/log/, so it will fail to write the log files. To account for this, I can think of these things:

• allow stdout logging output in addition, which is aggregated in /var/log/tomcat/catalina.out
• on startup of petascope/secore, check if it can write to $RMANHOME/log/; if not output a warning that the tomcat user needs write access to this path

[Peter Baumann]

/var/log/tomcat/catalina.out used to contain all Tomcat serves, so it will be a big mush of different service logs

[Bang Pham Huu]

​http://superuser.com/questions/280994/give-write-permissions-to-multiple-users-on-a-folder-in-ubuntu
How about add a group named: rasdaman_log which contains rasdaman user and tomcat user and it has writable permission on RMANHOME/etc/log directory?

I don't see any configuration on properties file to change the tomcat username for logging, so it is ok when make install petascope/secore will add this logging group.

[Dimitar Misev]

And how do we know the tomcat username? It's not possible to determine this always correctly.

[Bang Pham Huu]

Replying to dmisev:

And how do we know the tomcat username? It's not possible to determine this always correctly.

Rasdaman users will install from packages (so installer know which is tomcat user), in case they want to build from source code, then in the guide, tell users to set the permission to Tomcat'username which they want to use differnt in this log folder (normally the default username is: "tomcat" or "tomcat7").

[Peter Baumann]

this is too complicated and error prone. rasdaman needs to install itself without user decisions necessary by default.
BTW, a chgrp wouldn't help either - you'd need to have write access for "all" which is a security risk.

[Peter Baumann]

Actually, while I like the idea of having all rasdaman logs together (and I have strongly advocated this in the past as well) I believe we need to leave tomcat logs in place:

• it is expected behavior
• all changes needs too much fiddling with the target system

[Dimitar Misev]

Write access for group is enough, if the tomcat user is member of the rasdaman group. But yes, it's really complicated all in all and needs too much guessing.

If we use the embedded jetty server then it would make sense for the logs to be in $RMANHOME/log, otherwise we don't have much choice but to leave them bundled in catalina.out, or as the user configures them.

[Bang Pham Huu]

Replying to pbaumann:

this is too complicated and error prone. rasdaman needs to install itself without user decisions necessary by default.
BTW, a chgrp wouldn't help either - you'd need to have write access for "all" which is a security risk.

I think it is not really complicated, so when installer build rasdaman, it has 2 users:
+ rasdaman username
+ tomcat username
It will just make a new group (rasdaman_log) for this folder to write (775). I still could not know why to have to guess tomcat username here ?

[Peter Baumann]

putting tomcat in rasdaman group = security risk.

We must be able to run without modifications requiring superuser.

[Peter Baumann]

Replying to dmisev:

Write access for group is enough, if the tomcat user is member of the rasdaman group. But yes, it's really complicated all in all and needs too much guessing.

If we use the embedded jetty server then it would make sense for the logs to be in $RMANHOME/log, otherwise we don't have much choice but to leave them bundled in catalina.out, or as the user configures them.

…and tomcat configurations should not be overridden by some config file far far away - not "expected behavior".

[Dimitar Misev]

Moving to the embeded jetty server would solve a lot of our headaches

[Bang Pham Huu]

Replying to pbaumann:

putting tomcat in rasdaman group = security risk.

We must be able to run without modifications requiring superuser.

I mean, we put tomcat username and rasdaman username in another group (rasdaman_log) and they only have same permission on this log folder.

Or make it much simplifier, create rasdaman_log folder only for rasdaman username and petascope_log folder only for tomcat username (no risk any more).

[Dimitar Misev]

Changing top-level rasdaman installation directories should not be done.. pretty much ever.

Having $RMANHOME/log/webapps_logs/ for example could be an option, Peter what do you think?

[Dimitar Misev]

We should search for log4j.properties on the wiki and update the documentation.