Tomcat and Logging

Tomcat is a very popular java application server. It is very light weight and easy to use. It's derivative, tcserver, is also very nice. It has all the goodness of tomcat, and on top of that it gives you the feel of a real J2EE app server (tomcat is only java servlet compliant).

Logging from your webapp in tomcat is a piece of cake. For example, if you like to use SLF4J logging API together with logback as the backend, you just drop the jars in the webapp's WEB-INF/lib and the configuration (logback.groovy) in WEB-INF/classes and you're good to go.

You will notice, however, that something else is eating away your disk space beside your webapp's logs. Tomcat has its own logging. It writes several files, has its own logging configuration (and logging technology) and even does its own rotation for (only) some of the files. A log rotation scheme that can't be easily disabled and might not fit your global rotation scheme.

By default tomcat uses juli, which is based on the standard java.util.logging with some enhancements for supporting per-webapp configuration. This only matters to those that don't have their own logging set up inside their webapp. Also, the default configuration is quite surprising. It sets up several files for different categories that may or may not interest you.
There's:

manager.log and host-manager.log for the respective webapps (which you may not want to have deployed, but the empty log files will still be created).
localhost.log will have message about servlets coming up and down.
catalina.log will contain mostly internal tomcat messages.
Some logging messages will be sent to stdout and get mixed with actual direct writes to stdout/stderr as well as with the output from scripts that start and stop tomcat. If you redirect this to a file (catalina.out) it is not rotated and can grow very big over time. Some messages are send to both catalina.log and catalina out, which is confusing if not just redundant.

All the files created (this doesn't include the console output) use the juli file handler that by default adds a date stamp to the file name (for rolling). It is not configurable! You simply can't remove it (you can just change the prefix and suffix but there will always be a date in the middle).

These are all the options I can think about (after scouring the net for solutions):
* Replace tomcat logging with log4j or logback (requires dropping the jars in the bin dir and changing the startup script to include them in the classpath)
* Replace the juli FileHandler with the standard java.util.logging FileHandler and have only a single handler for everything, since we don't need the per-webapp logging (we set that up within the webapp)
* Use SocketHandler to write to syslog instead of directly into a file
* Use MemoryHandle since most of these message can be ignored until we actually have a problem
* Use nop logging configuration, but again we lose troubleshooting information

In any case setting the sallowOutput option to true in the context configuration file will catch some messages written to stdout/stderr and redirect them to the webapp logger. It doesn't catch all messages though (I think ones that are written early).

For rotation I want to use logrotate so it is possible to:
* Use SocketHandler to write to syslog, and syslog can be told to reopen its log files
* Enhance the logging framework to listen to a signal to reopen its files
* Use cronolog (but again we lose control of the file naming since it adds a date suffix - I want to use numbers)
* Use the (nonatomic) copytruncate option in logrotate but risk losing some messages

In the end I decided to do the following:
* Comment out everything it tomcat's logging.properties except a single catch-all FileHandler, using the default java.util.logging FileHandler instead of juli's. This includes disabling the ConsoleHandler (I don't need it since I'm running tomcat in a server and never interactively)
* Rotate the created files (the single log and the stdout redirected to a file) using logrotate with the copytruncate flag (this is not safe for the stdout file, but I don't expect it to grow much now that it doesn't include any of tomcat's own logging)
* Enabled the swallowOutput flag
* Use syslog for all the logback logs and rotate with logrotate them without copytruncate

Restoring x server zap functionality

The xorg x server on gentoo has the ctrl-alt-backspace disabled by default. I really like that functionality (especially when seeing that there is a lot of memory in use when nothing is running, after a very long uptime).

Apparently you should use kernel SAK which is superior (alt-break-K if you have magic sysrq configured in the kernel). For some reason it is not recommended to use this to SAK, so a new key sequence should be defined for it. A good place to do it in gentoo is in /etc/conf.d/local.start, and since ctrl-alt-backspace is available, it can be used for this purpose (key code 14):

# Set ctrl-alt-backspace to SAK
echo "control alt keycode 14 = SAK" | /usr/bin/loadkeys

Optionally restore the key map in /etc/conf.d/local.stop:

# restore keymap to default
/usr/bin/loadkeys --default

It is also a good place to set a replacement key sequence for reboot, if ctrl-alt-del is disabled in /etc/inittab. For example ctrl-alt-break (key code 119):

echo "control alt keycode 119 = Boot" | /usr/bin/loadkeys

Gradle ebuild

Latest ebuild can be found in github: https://github.com/asssaf/portage/tree/master/dev-java/gradle-bin

Pretty ugly (uses the bundled jars instead of installed ones), but it works.

# dev-java/gradle-bin-0.9_rc1.ebuild

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI="2"

inherit java-pkg-2

MY_PN=${PN%%-bin}
#MY_PV=${PV/_pre/-preview-}
MY_PV=${PV/_rc/-rc-}
MY_P="${MY_PN}-${MY_PV}"

DESCRIPTION="Build Tool for Java"
SRC_URI="http://dist.codehaus.org/${MY_PN}/${MY_P}-bin.zip"
HOMEPAGE="http://www.gradle.org/"
LICENSE="Apache-2.0"
SLOT="0"
KEYWORDS="~x86"

RDEPEND=">=virtual/jdk-1.5"

IUSE=""

S="${WORKDIR}/${MY_P}"


src_unpack() {
unpack ${A}
}


# TODO we should use jars from packages, instead of what is bundled
src_install() {
#TODO set GRADLE_HOME
#local jar JARS="gradle-core gradle-open-api gradle-ui gradle-wrapper"

local gradle_home="${ROOT}/usr/share/${PN}"

insinto "${gradle_home}"
#doins gradle-imports || die "Failed to install files"

cd lib
for jar in *.jar; do
 java-pkg_newjar ${jar} ${jar}
done

insinto "${gradle_home}/lib/plugins"
doins plugins/*

#sed -i "s/-${PV}.jar/.jar/" bin/gradle || die "Failed to patch launcher script"
#dobin bin/gradle || die "Failed to copy launcher script"
java-pkg_dolauncher "gradle" --main org.gradle.launcher.GradleMain --java_args "-Dgradle.home=${gradle_home}"
}

Update (2-Aug-10): I'm getting a NoClassDefError when trying to run JUnit tests. I've switched to the stable 0.8, which seems to work fine (just renamed the ebuild).

Update (27-Aug-10): I found a workaround for this issue. Adding the following lines to the build.gradle file fixes it:

test.bootstrapClasspath('/opt/sun-jdk-1.6.0.20/jre/lib/rt.jar')
test.bootstrapClasspath('/usr/share/gradle-bin/lib/commons-lang-2.5.jar')

Also updated the ebuild to 0.9_rc1.

Update (3-Nov-10): I've added additional enhancements to the ebuild but decided that keeping the blog up to date is the wrong way to go about it so I'm doing all ebuild maintenance in github from now on. My portage repository is here (and this ebuild is under dev-java/gradle-bin).

My /boot kernel management (2)

In the last post I described my /boot kernel scheme and eselect-based management tool. Another piece of that is the script to actually install a new kernel to that fancy structure.

Here it is:

#!/bin/sh

#
# "make install" script for i386 architecture
#
# Arguments:
#   $1 - kernel version
#   $2 - kernel image file
#   $3 - kernel map file
#   $4 - default install path (blank if root directory)
#

KERNEL_VERSION="$1"
KERNEL_IMAGE="$2"
KERNEL_MAP="$3"
BASE_INSTALL_PATH="$4"
KERNEL_BASE="$BASE_INSTALL_PATH/installkernel"

die () {
 ERR="$1"
 shift

 if [ -z "$1" ]
 then
  echo Aborting 1>&2
 else
  echo $* 1>&2
 fi

 exit $ERR
}


verify () {
 if [ ! -f "$1" ]; then
  echo ""                                                   1>&2
  echo " *** Missing file: $1"                              1>&2
  echo ' *** You need to run "make" before "make install".' 1>&2
  echo ""                                                   1>&2
  die 1
 fi
}

# Make sure the files actually exist
verify "$KERNEL_IMAGE"
verify "$KERNEL_MAP"

# check if the version already exists in /boot and find the appropriate p-level
KERNEL_BASE_DIR="$KERNEL_BASE/$KERNEL_VERSION"
KERNEL_PLEVEL="0"

while [ -e "$KERNEL_BASE_DIR/p$KERNEL_PLEVEL" ]
do
 KERNEL_PLEVEL="$(expr $KERNEL_PLEVEL + 1)"
done

INSTALL_PATH="$KERNEL_BASE_DIR/p$KERNEL_PLEVEL"

# backup old modules if needed
#TODO warn before overwriting existing backup?
#TODO check if the previous plevel is identical?

if [ $KERNEL_PLEVEL != 0 ]
then
 OLD_PLEVEL="$(expr $KERNEL_PLEVEL - 1)"
 MODULES_BACKUP_FILE="/lib/modules/$KERNEL_VERSION-p$OLD_PLEVEL.tgz"
 echo "Creating modules backup for previous kernel: $MODULES_BACKUP_FILE"
 tar czf "$MODULES_BACKUP_FILE" -C /lib/modules $KERNEL_VERSION || die 4
fi


# make the new kernel dir
echo "Creating new kernel dir: $INSTALL_PATH"
mkdir -p $INSTALL_PATH || die 2

# copy the file to the new kernel dir
echo "Copying kernel files"
cp "$KERNEL_IMAGE" $INSTALL_PATH/ || die 3
cp "$KERNEL_MAP" $INSTALL_PATH/ || die 3
cp .config $INSTALL_PATH/ || die 3

#TODO run eselect and make modules_install?
echo ""
echo "Done. You should probably do the following now:"
echo "1. eselect the new kernel"
echo "2. run make modules_install"
echo "3. run modules-rebuild"
echo "4. update the ChangeLog"
echo ""

In order to hook this to the standard make install command run at kernel build, this needs to be placed in /sbin/installkernel.
Alas, that position is already taken by a file belonging to the debianutils package. Arrgghhh!!
I opened a bug (and provided a patch) to make this optional.

For now the ebuild for installing the script (myinstallkernel-0.1.ebuild) must block debianutils:

EAPI="2"

LICENSE="GPL"
SLOT="0"
KEYWORDS="x86"
IUSE="+symlink"

DEPEND="symlink? ( sys-apps/debianutils[-installkernel] )"

src_install() {
 newsbin ${FILESDIR}/${P} ${PN}

 if use symlink; then
  dosym /usr/sbin/${PN} /sbin/installkernel
 fi

 insinto /usr/share/eselect/modules
 newins ${FILESDIR}/mykernel.eselect-${PV} mykernel.eselect
}

Update 03-Jun-2010:
1. Updated myinstallkernel script to version 0.3.
2. The bug wasn't accepted by gentoo, but they suggested using CONFIG_PROTECT="/sbin/installkernel" in /etc/make.conf. This kinda works, so I've removed the block from the ebuild.

My /boot kernel management

Some linux distribution simply put the kernel image at /boot/vmlinuz and maybe a backup version as /boot/vmlinuz.old. Same goes for a System.map file.

I prefer to store multiple kernel versions in a hierarchical structure with symlink pointing to the current kernel (which are in turn referenced by grub's menu).

For example, my current kernel is minigen32 (mini since it's my mac mini kernel, gen for gentoo and 32... you can guess :) ).

Under /boot/kernels I have a subdirectory for minigen32 and under that a subdir for each kernel version. I also put another subdir level for local build number, since I might have several builds of the same kernel version.

So I have:
/boot/kernels/minigen32/2.6.32-gentoo-r7/p0
/boot/kernels/minigen32/2.6.32-gentoo-r7/p1

etc., etc.

Each such directory contains the same files. The good 'ol bzImage, System.map and .config.

Then I have symlinks for stable, testing, current, previous etc. pointing to these subdirs, and a simple grub file can be written (kernel /testing/bzImage). When I want to switch kernels I just need to update the symlinks. No need to touch the grub configuration (unless of course I need to change the kernel command line).

Using gentoo's handy eselect utility i've built a module to handle this scheme for me.

$ eselect mykernel list
Available kernel symlink targets:
  [1]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p0  stable
  [2]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p1  testing
  [3]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p2  

$ eselect mykernel set testing 3
$ eselect mykernel set testing 2

$ eselect mykernel list
Available kernel symlink targets:
  [1]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p0  
  [2]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p1  stable
  [3]   /boot/kernels/minigen32/2.6.32-gentoo-r7/p2  testing

Code for the module (just drop it in ~/.eselect/modules/mykernel.eselect):

DESCRIPTION="Manage the /boot kernel symlinks"
VERSION="0.3"

local BASE_DIR="/boot"
local KERNELS_BASE="$BASE_DIR/kernels"
local TAGS=( stable testing )

# find a list of kernel symlink targets
find_targets() {
 local f
 for f in $KERNELS_BASE/*/*/p* ; do
  #[[ -d ${f} ]] && basename "${f}"
  #[[ -d ${f} ]] && echo "${f#${BASE_DIR}/}"
  [[ -d ${f} ]] && echo "${f}"
 done 
}


### show action ###

describe_show() {
 echo "Show the current tagged kernels"
}

do_show() {
 local TAGS_TO_SHOW=( "${TAGS[@]}" )
 has $1 ${TAGS[@]} && TAGS_TO_SHOW=( $1 )
 
 for TAG in ${TAGS_TO_SHOW[@]}
 do
  my_show $TAG
 done
}

my_show() {
 local TAG=$1
      
 write_list_start "Current ${TAG} kernel"
       if [[ -L $BASE_DIR/${TAG} ]] 
 then 
  write_kv_list_entry $BASE_DIR/$(readlink "${BASE_DIR}/${TAG}") ""
 else
  write_kv_list_entry "(unset)" ""
 fi
}

### list action ###

describe_list() {
 echo "List Available Kernels in $BASE_DIR"
}

do_list() {
 local i j SYMLINKS TAG targets=( $(find_targets) )
 write_list_start "Available kernel symlink targets:"

 for (( j = 0; j < ${#TAGS[@]}; j++ ))
 do
  [[ -L $BASE_DIR/${TAGS[$j]} ]] && SYMLINKS[$j]=$BASE_DIR/$(readlink "${BASE_DIR}/${TAGS[$j]}")
 done

 for (( i = 0; i < ${#targets[@]}; i++ ))
 do
  local mark=""

  for (( j = 0; j < ${#TAGS[@]}; j++ ))
  do
   if [[ ${targets[${i}]} == ${SYMLINKS[$j]} ]]
   then
    mark="${mark} $(highlight ${TAGS[$j]})"
   fi 
  done  

  targets[${i}]="${targets[${i}]} ${mark}"
 done

 write_numbered_list -m "(none found)" "${targets[@]}"
}

### set action ###

describe_set() {
 echo "Tag a kernel"
}

do_set() {
 local usage="Usage [${TAGS[@]}] [kernel]" tag=$1 target=$2 symlink
 [[ ${#} != 2 ]] && die -q ${usage}
 has $tag ${TAGS[@]} || die -q ${usage}
 
 symlink=$BASE_DIR/$tag
 if [[ -L "${symlink}" ]] ; then
  set_symlink "${target}" "${symlink}"  || die -q "Couldn't set a new symlink"

 elif [[ -e "${symlink}" ]] ; then
  die -q "Target file already exists and is not a symlink: ${symlink}"

 else
  set_symlink "${target}" "${symlink}" || die -q "Couldn't set a new symlink"
 fi 
}


set_symlink() {
 local target=${1} symlink=${2}
 if is_number "${target}" ; then
  targets=( $(find_targets) )
  target=${targets[$(( ${target} - 1 ))]}
 fi
 if [[ -z ${target} ]] ; then
  die -q "Target \"${1}\" doesn't appear to be valid!"
 elif [[ -d "${target}" ]] ; then
  local sym_dir=$(dirname ${symlink})
  if [[ ! -d ${sym_dir} ]]; then
   mkdir -p ${sym_dir} || die -q "Could not create ${my_dir}"
  fi
  ln -snf "${target#${BASE_DIR}/}" "${symlink}"
 else
  die -q "Target \"${1}\" doesn't appear to be valid!"
 fi
}

Update 3-Jun-2010: Updated eselect module to version 0.3

Remote connection to oracle as sysdba

Been really busy lately at work, so it's my first post in a while.

Having a different OS and oracle flavor than my peers have caused me quite a bit of fuss in the past. Particularly, logging in as normal user vs. sysdba, running sqlplus as the oracle user vs. a normal user (in the dba group) and local in process sqlplus vs. network sql*net. (BTW, the solution for running in process as normal user involved fixing the permissions of the oracle executable messed up by the package manage, conary. It needs to be chmod'd 6751)

A code change in our application forced me to revisit the issue with connecting as sysdba over sql*net. We had some JDBC code that was updating two schemas. Originally it was running twice as two different users and someone changed it to run once as sysdba. I will not go into why, and this change has been reverted eventually for different reasons, but I had to get it working and I couldn't figure out why sys on other Oracle's is able to connect and on mine it can't.

I found the following kludgy workaround:
1. Make sure oracle is using an EXCLUSIVE password file:

> show parameter password;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_login_passwordfile            string      EXCLUSIVE

2. Grant some user SYSDBA:

> GRANT SYSDBA TO someuser;

3. Check it:

> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP
------------------------------ ----- -----
SYS                            TRUE  TRUE
SOMEUSER                       TRUE  FALSE

4. Now export TWO_TASK in the environment and see that you can connect over sql*net

$ export TWO_TASK=LOCAL_XE
$ sqlplus someuser/password AS SYSDBA

Hurray! If you can connect with TWO_TASK set, jdbc using the thin client should work as well.

I also needed to set the permissions on the log dir because I got some:

ORA-29283: invalid file operation                               
ORA-06512: at "SYS.UTL_FILE", line 33                                                                                                 
ORA-06512: at "SYS.UTL_FILE", line 460

Sources:
http://it.toolbox.com/wiki/index.php/Remote_login_as_Sysdba_to_Oracle_database_server#Operating-System-Based_Authentication

http://www.oracleutilities.com/OSUtil/orapwd.html

Mysterious VMware error in gentoo solved

A while ago I've encountered some errors on vmware startup after an upgrade. One of them was:

/opt/vmware/workstation/lib/vmware/bin/vmware: /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/libstdc++.so.6: version `CXXABI_1.3.1' not found (required by /opt/vmware/workstation/lib/vmware/lib/libvmwareui.so.0/libvmwareui.so.0)

This is probably harmless, and the solution was easy:
$ VMWARE_USE_SHIPPED_GTK=yes vmware

The other one was more mystifying. After starting the UI and clicking the play button I get the familiar "find and run vmware-config.pl" message, which usually means the kernel modules haven't been compiled.

By the vmware-config.pl file has been removed, and besides the modules seemed to load fine.

Today I've upgraded again (after a kernel upgrade as well that solved some vmware-modules compilation problems) and got this again.
Luckily I found the solution here. Just remove a pesky /etc/vmware/not_configured file and you're good to go.

Since I knew I would probably forget about it by the time I encountered this problem again, I renamed it to something very descriptive: not_configured.this_file_makes_vmware_puke.
In second thought, I realized I should probably put it in the blog if I'd ever want to find it again.

Arrrgghh...