Opened 6 months ago

Closed 5 months ago

#1415 closed defect (fixed)

petascope and secore configuration files

Reported by: dmisev Owned by: bphamhuu
Priority: major Milestone: 9.3
Component: applications Version: development
Keywords: Cc: pbaumann, vmerticafriu, bphamhuu
Complexity: Medium

Description (last modified by dmisev)

The logging configuration files for these components are:

  • petascope: $RMANHOME/etc/log4j.properties
  • secore: $CATALINA_HOME/webapps/def/etc/log4j.properties

The general configuration files are:

  • petascope: $RMANHOME/etc/petascope.properties
  • secore: $CATALINA_HOME/webapps/def/etc/secore.conf

It would be good to harmonize this state of affairs.. I suggest that everything goes in $RMANHOME/etc like this, such that the logging configuration is merged into the main configuration file.

component config file
petascope petascope.properties
secore secore.properties

Any thoughts?

Change History (24)

comment:1 Changed 6 months ago by pbaumann

+1. With defaults where all log goes into $RMANHOME/log

PS: we are in the root FS again, so the hint of doing a symlink to another partition for the logs is as appropriate as ever.

comment:2 Changed 6 months ago by bphamhuu

would be better if you create subdirectory for log-configuration (/etc/log) and general-configuration (/etc/conf). Now, it is quite difficult to see the list of configuration file in $RMANHOME/etc.

-rwxr-xr-x. 1 rasdaman rasdaman 1669 Oct 15 11:36 log4j.properties
-rwxr-xr-x. 1 rasdaman rasdaman 2520 Oct 15 11:35 log-client.conf
-rwxr-xr-x. 1 rasdaman rasdaman 2648 Oct 15 11:35 log-rasmgr.conf
-rwxr-xr-x. 1 rasdaman rasdaman 2648 Oct 15 11:35 log-server.conf
-rwxr-xr-x. 1 root     root     9459 Oct 16 08:40 petascope.properties
-rwxr-xr-x. 1 rasdaman rasdaman 9459 Oct 16 08:32 petascope.properties.10-16-2016_08-32-04.bak
-rwxr-xr-x. 1 rasdaman rasdaman 5048 Oct 15 11:35 rasmgr.conf
-rwxr-xr-x. 1 rasdaman rasdaman  552 Oct 15 11:36 wms_service.properties

Should be not problem when merging log4j with petascope.properties, btw, don't we have secore.log (I know we have petascope.log and rasdaman) ?

comment:3 Changed 6 months ago by dmisev

  • Description modified (diff)
  • Owner set to dmisev
  • Status changed from new to assigned

comment:4 Changed 6 months ago by pbaumann

I wouldn't do that - if we need subdirs for just config files then we are getting way too complicated. Rather, let us think about simplifying the system. This seems to be necessary periodically (we were in similar situations before).

But always have in mind that conf files are externally visible behavior of the system, so changes face a high barrier and easily require a major release change.

comment:5 Changed 6 months ago by dmisev

I'm not in favor of the subdir idea. In any case there can be a single petascope.properties and secore.properties that include the logging configuration as well.

We can automatically migrate to the new conf file structure, and still support the old case so that it's backwards compatible. We don't need a major release in that case I'd say.

comment:6 Changed 6 months ago by pbaumann

not sure that there is no impact - for example, some people parse log files, so those must have a frozen structure as well. Might be someone similarly does autoconfig scripts on log files, we never know.

comment:7 Changed 6 months ago by dmisev

These are the configuration files, I don't think anyone is parsing them.. we can ask on the mailing lists just in case. The secore ones surely not, currently they are quite hidden in the WAR file.

comment:8 Changed 6 months ago by dmisev

  • Owner changed from dmisev to bphamhuu

First step is to move the secore configuration to $RMANHOME/etc:

  1. move $CATALINA_HOME/webapps/def/etc/secore.conf to $RMANHOME/etc/secore.properties
  2. merge the contents of $CATALINA_HOME/webapps/def/etc/log4j.properties inside the new $RMANHOME/etc/secore.properties
  3. change the logging config to output to $RMANHOME/log/secore.log like it's done for petascope

comment:9 Changed 6 months ago by dmisev

There is a problem when configuring logging to $RMANHOME/log/petascope.log or $RMANHOME/log/secore.log.

The log is written by the tomcat system user running Tomcat. This user will usually not have write access in $RMANHOME/log/, so it will fail to write the log files. To account for this, I can think of these things:

  • allow stdout logging output in addition, which is aggregated in /var/log/tomcat/catalina.out
  • on startup of petascope/secore, check if it can write to $RMANHOME/log/; if not output a warning that the tomcat user needs write access to this path

comment:10 Changed 6 months ago by pbaumann

/var/log/tomcat/catalina.out used to contain all Tomcat serves, so it will be a big mush of different service logs

comment:11 Changed 6 months ago by bphamhuu

http://superuser.com/questions/280994/give-write-permissions-to-multiple-users-on-a-folder-in-ubuntu
How about add a group named: rasdaman_log which contains rasdaman user and tomcat user and it has writable permission on RMANHOME/etc/log directory?

I don't see any configuration on properties file to change the tomcat username for logging, so it is ok when make install petascope/secore will add this logging group.

comment:12 follow-up: Changed 6 months ago by dmisev

And how do we know the tomcat username? It's not possible to determine this always correctly.

comment:13 in reply to: ↑ 12 Changed 6 months ago by bphamhuu

Replying to dmisev:

And how do we know the tomcat username? It's not possible to determine this always correctly.

Rasdaman users will install from packages (so installer know which is tomcat user), in case they want to build from source code, then in the guide, tell users to set the permission to Tomcat'username which they want to use differnt in this log folder (normally the default username is: "tomcat" or "tomcat7").

comment:14 follow-up: Changed 6 months ago by pbaumann

this is too complicated and error prone. rasdaman needs to install itself without user decisions necessary by default.
BTW, a chgrp wouldn't help either - you'd need to have write access for "all" which is a security risk.

comment:15 Changed 6 months ago by pbaumann

Actually, while I like the idea of having all rasdaman logs together (and I have strongly advocated this in the past as well) I believe we need to leave tomcat logs in place:

  • it is expected behavior
  • all changes needs too much fiddling with the target system

comment:16 follow-up: Changed 6 months ago by dmisev

Write access for group is enough, if the tomcat user is member of the rasdaman group. But yes, it's really complicated all in all and needs too much guessing.

If we use the embedded jetty server then it would make sense for the logs to be in $RMANHOME/log, otherwise we don't have much choice but to leave them bundled in catalina.out, or as the user configures them.

comment:17 in reply to: ↑ 14 Changed 6 months ago by bphamhuu

Replying to pbaumann:

this is too complicated and error prone. rasdaman needs to install itself without user decisions necessary by default.
BTW, a chgrp wouldn't help either - you'd need to have write access for "all" which is a security risk.

I think it is not really complicated, so when installer build rasdaman, it has 2 users:
+ rasdaman username
+ tomcat username
It will just make a new group (rasdaman_log) for this folder to write (775). I still could not know why to have to guess tomcat username here ?

comment:18 follow-up: Changed 6 months ago by pbaumann

putting tomcat in rasdaman group = security risk.

We must be able to run without modifications requiring superuser.

comment:19 in reply to: ↑ 16 Changed 6 months ago by pbaumann

Replying to dmisev:

Write access for group is enough, if the tomcat user is member of the rasdaman group. But yes, it's really complicated all in all and needs too much guessing.

If we use the embedded jetty server then it would make sense for the logs to be in $RMANHOME/log, otherwise we don't have much choice but to leave them bundled in catalina.out, or as the user configures them.

...and tomcat configurations should not be overridden by some config file far far away - not "expected behavior".

comment:20 Changed 6 months ago by dmisev

Moving to the embeded jetty server would solve a lot of our headaches :)

comment:21 in reply to: ↑ 18 Changed 6 months ago by bphamhuu

Replying to pbaumann:

putting tomcat in rasdaman group = security risk.

We must be able to run without modifications requiring superuser.

I mean, we put tomcat username and rasdaman username in another group (rasdaman_log) and they only have same permission on this log folder.

Or make it much simplifier, create rasdaman_log folder only for rasdaman username and petascope_log folder only for tomcat username (no risk any more).

comment:22 Changed 6 months ago by dmisev

Changing top-level rasdaman installation directories should not be done.. pretty much ever.

Having $RMANHOME/log/webapps_logs/ for example could be an option, Peter what do you think?

comment:23 Changed 6 months ago by dmisev

We should search for log4j.properties on the wiki and update the documentation.

comment:24 Changed 5 months ago by dmisev

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.