Tuesday, January 25, 2011

Tomcat and Logging

Tomcat is a very popular java application server. It is very light weight and easy to use. It's derivative, tcserver, is also very nice. It has all the goodness of tomcat, and on top of that it gives you the feel of a real J2EE app server (tomcat is only java servlet compliant).

Logging from your webapp in tomcat is a piece of cake. For example, if you like to use SLF4J logging API together with logback as the backend, you just drop the jars in the webapp's WEB-INF/lib and the configuration (logback.groovy) in WEB-INF/classes and you're good to go.

You will notice, however, that something else is eating away your disk space beside your webapp's logs. Tomcat has its own logging. It writes several files, has its own logging configuration (and logging technology) and even does its own rotation for (only) some of the files. A log rotation scheme that can't be easily disabled and might not fit your global rotation scheme.

By default tomcat uses juli, which is based on the standard java.util.logging with some enhancements for supporting per-webapp configuration. This only matters to those that don't have their own logging set up inside their webapp. Also, the default configuration is quite surprising. It sets up several files for different categories that may or may not interest you.
There's:

manager.log and host-manager.log for the respective webapps (which you may not want to have deployed, but the empty log files will still be created).
localhost.log will have message about servlets coming up and down.
catalina.log will contain mostly internal tomcat messages.
Some logging messages will be sent to stdout and get mixed with actual direct writes to stdout/stderr as well as with the output from scripts that start and stop tomcat. If you redirect this to a file (catalina.out) it is not rotated and can grow very big over time. Some messages are send to both catalina.log and catalina out, which is confusing if not just redundant.

All the files created (this doesn't include the console output) use the juli file handler that by default adds a date stamp to the file name (for rolling). It is not configurable! You simply can't remove it (you can just change the prefix and suffix but there will always be a date in the middle).

These are all the options I can think about (after scouring the net for solutions):
* Replace tomcat logging with log4j or logback (requires dropping the jars in the bin dir and changing the startup script to include them in the classpath)
* Replace the juli FileHandler with the standard java.util.logging FileHandler and have only a single handler for everything, since we don't need the per-webapp logging (we set that up within the webapp)
* Use SocketHandler to write to syslog instead of directly into a file
* Use MemoryHandle since most of these message can be ignored until we actually have a problem
* Use nop logging configuration, but again we lose troubleshooting information

In any case setting the sallowOutput option to true in the context configuration file will catch some messages written to stdout/stderr and redirect them to the webapp logger. It doesn't catch all messages though (I think ones that are written early).

For rotation I want to use logrotate so it is possible to:
* Use SocketHandler to write to syslog, and syslog can be told to reopen its log files
* Enhance the logging framework to listen to a signal to reopen its files
* Use cronolog (but again we lose control of the file naming since it adds a date suffix - I want to use numbers)
* Use the (nonatomic) copytruncate option in logrotate but risk losing some messages

In the end I decided to do the following:
* Comment out everything it tomcat's logging.properties except a single catch-all FileHandler, using the default java.util.logging FileHandler instead of juli's. This includes disabling the ConsoleHandler (I don't need it since I'm running tomcat in a server and never interactively)
* Rotate the created files (the single log and the stdout redirected to a file) using logrotate with the copytruncate flag (this is not safe for the stdout file, but I don't expect it to grow much now that it doesn't include any of tomcat's own logging)
* Enabled the swallowOutput flag
* Use syslog for all the logback logs and rotate with logrotate them without copytruncate