Mysterious VMware error in gentoo solved

A while ago I've encountered some errors on vmware startup after an upgrade. One of them was:

/opt/vmware/workstation/lib/vmware/bin/vmware: /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/libstdc++.so.6: version `CXXABI_1.3.1' not found (required by /opt/vmware/workstation/lib/vmware/lib/libvmwareui.so.0/libvmwareui.so.0)

This is probably harmless, and the solution was easy:
$ VMWARE_USE_SHIPPED_GTK=yes vmware

The other one was more mystifying. After starting the UI and clicking the play button I get the familiar "find and run vmware-config.pl" message, which usually means the kernel modules haven't been compiled.

By the vmware-config.pl file has been removed, and besides the modules seemed to load fine.

Today I've upgraded again (after a kernel upgrade as well that solved some vmware-modules compilation problems) and got this again.
Luckily I found the solution here. Just remove a pesky /etc/vmware/not_configured file and you're good to go.

Since I knew I would probably forget about it by the time I encountered this problem again, I renamed it to something very descriptive: not_configured.this_file_makes_vmware_puke.
In second thought, I realized I should probably put it in the blog if I'd ever want to find it again.

Arrrgghh...

Balsamiq is the New Visio

Balsamiq Mockups rocks!
After a brainstorming meeting, instead of taking the "traditional" cellphone snapshot of the whiteboard, I simply went back to my room and made a mockup of the new screen for the meeting summary email. It took literally under five minutes.

I think I will include mockups in all specification documents from now on. It's just so easy and fun to do!

P.S.: PS3 slim edition is out...

Adobe AIR on Gentoo

Balsamiq Mockups for the desktop (www.balsamiq.com) is a nice tool for generating nice looking mockups real fast (much faster than visio, for example).
I've been using the web based trial version until today, when I got my license.

Running Balsamiq locally requires installing Adobe AIR on my machine, something I haven't got to doing until now. Installing the linux version of the AIR on gentoo didn't work at all (the installer binary just exits without any message).

Trying to resolve this, one of the first google search results was a nice guide for gentoo. And it worked flawlessly - basically you need to unpack the AIR SDK instead of using the installer.

I've also written a slightly more sophisticated shell script for running AIR apps, which allows specifying the app on the command line as well as symlinking.

/usr/local/bin/air:

#!/bin/sh

SDK_DIR="/usr/local/air_1.5_sdk"
APPS_DIR="/usr/local/air_apps"

BASE="$(basename $0)"


if [ -n "$1" ]
then
        APP_NAME="$1"
        if [ ! -d "$APPS_DIR/$APP_NAME" ]
        then
                echo "App $APP_NAME not found in $APPS_DIR" > /dev/stderr
                exit 1
        fi

else
        APP_NAME="$BASE"
        if [ ! -d "$APPS_DIR/$APP_NAME" ]
        then                echo "App $APP_NAME not found in $APPS_DIR" > /dev/stderr
                echo "Usage: $(basename $0) " > /dev/stderr
                exit 1
        fi
fi

exec "$SDK_DIR/bin/adl" -nodebug "$APPS_DIR/$APP_NAME/META-INF/AIR/application.xml" "$APPS_DIR/$APP_NAME"

Now simply unpack balsamiq in /usr/local/air_apps/balsamiq, symlink and you're ready to go:

$ mkdir /usr/local/air_apps/balsamiq && unzip MockupsForDesktop.air -d /usr/local/air_apps/balsamiq
$ ln -s air /usr/local/bin/balsamiq
$ balsamiq &

Update: Support passing command line arguments to the air application (note that for balsamiq to open a file given on the command line, a full absolute path needs to be given)

#!/bin/sh

SDK_DIR="/usr/local/air_1.5_sdk"
APPS_DIR="/usr/local/air_apps"

BASE="$(basename $0)"

if [ "$BASE" = air -a -n "$1" ]
then
 APP_NAME="$1"
 if [ ! -d "$APPS_DIR/$APP_NAME" ]
 then
  echo "App $APP_NAME not found in $APPS_DIR" > /dev/stderr
  exit 1
 fi

 shift

else
 APP_NAME="$BASE"
 if [ ! -d "$APPS_DIR/$APP_NAME" ]
 then 
  echo "App $APP_NAME not found in $APPS_DIR" > /dev/stderr
  echo "Usage: $(basename $0) " > /dev/stderr
  exit 1
 fi
fi

exec "$SDK_DIR/bin/adl" -nodebug "$APPS_DIR/$APP_NAME/META-INF/AIR/application.xml" "$APPS_DIR/$APP_NAME" -- "$@"

Doing More For Gentoo

Last week I needed to dump a remote svn repository without local access. After searching for the best way to do it I finally chose rsvndump (because it allows dumping a subdirectory of the repository).

As usual, I created an ebuild in order to install it, and posted it in a bug. As some of my bugs are pretty much ignored for a long period, I took the advice to post it to the sunrise overlay.

A week later, after multiple reviews and touch ups (those guys are strict!) the ebuild is now available in the sunrise overlay.

This took a little more effort than just posting the bug, but the ebuild is now of much higher quality (and I learned quite a bit more about ebuilds). It should be easier for the devs to include it in portage this way.

Revised highlighting script to use sed = full regex support

I had a TODO in the header of my bash highlighting script (hl) for a long time to add support for case insensitive highlighting and regex support, for a very long time.

Today I finally fixed it by changing the script to use sed.

Example usage:

$ glxinfo | hl GL_ARB_multitexture\|GL_EXT_framebuffer_object
        \|GL_ARB_shader_objects\|GL_ARB_shading_language_100
        \|GL_ARB_fragment_shader

The script:

#!/bin/sh
# Highlight specified pattern

RED=`echo -en '\e[31m'`
GREEN=`echo -en '\e[32m'`
YELLOW=`echo -en '\e[93m'`
NORMAL=`echo -en '\e[00m'`

usage () {
        echo "Usage: `basename $0` [red | green | yellow] str"
        exit 1
}


[ "$1" ] || usage

if [ -n "$2" ]
then
        case $1 in
        red) COLOR="$RED";;
        green) COLOR="$GREEN";;
        yellow) COLOR="$YELLOW";;
        *) usage;;
        esac

        shift
else
        COLOR="$RED"
fi

STR="$1"

sed -E "s/($STR)/$COLOR\1$NORMAL/g"

Faking an email sender address

It's been a while since I last blogged, been busy...

It is a pretty well known fact that the SMTP protocol is very insecure, and it is easy to spoof and fake email messages. I've encountered a legitimate use case today.

Imagine you've had an email account where you keep all your stuff for archiving purposes (what? you don't keep your inbox at 0 messages?!) and let's say one day you accidentally remove all your messages. Alternatively, your SP may have gone belly up.

Fortunately, you made a backup so you have all your emails sitting safely on your local HD. But now your email is fragmented. All email you received until today is stored on your HD and anything new will be received at you actual email account. That could suck when you'd want to run a global search to find an email you can't quite remember when you received it.

So you could resend all those emails from the backup to your mail account, but then they would all seem to originate from the same source - you!

That's a lot of information lost. In this case it would be better to keep the original sender. Here's an example on how to achieve this with ssmtp (a lightweight MTA).

Make sure you've configured it to look for from addresses in the actual message text. In /etc/ssmtp/ssmtp.conf you should have:

FromLineOverride=YES

You also need to configure a mailhub for ssmtp to work, obviously:

mailhub=mailserver.corp.foo.bar

Now execute ssmtp (you'd probably want to create a script for this for mass mailing your backup):

# sendmail me@mail.corp.foo.bar
From: someone@somewhere.com                                  
To: me@mail.corp.foo.bar                            
Subject: Fake from field                                     
I've sent this with a fake from field using ssmtp.
[EOF]

Note that the from address (someone@somewhere.com) is completely bogus for the current session (so the email will appear to originate from the original sender). Pay attention that ssmtp seems to be looking for exactly 'From:' (case sensitive).

I haven't tried it, but the same principle should also work to preserve the original date of the email.

SSL with multiple hostnames

I noticed today that my SNI setup in apache is not working reliably. Apparently it takes the localhost certificate no matter how I am connecting. This seems to related to the content of /etc/hosts on the server, which I don't want to mess around with just to get SNI working.

Fortunately both IE and FF support an extension allowing multiple hostnames to be specified in the certificate in addition to the CN (reportedly Java does not support this extension, will need to check). The extension (Subject Alternate Names) is described in RFC 5280.

I've done the following to generate such certificates with openssl:

First, we need to add the extension support in openssl.cnf:

(not sure this is required, supposedly there's a security risk) in the [CA_default] section add:

copy_extensions = copy

in the [req] section add:

req_extensions = v3_req

in the [v3_req] section:

subjectAltName = IP:1.2.3.4,IP:127.0.0.1,DNS:myhost.acme.com,DNS:myhost,DNS:localhost.localdomain,DNS:localhost

Now we can generate a request, careful to include the config file so the extensions are included:

openssl req -config /etc/ssl/openssl.cnf -new -out multi.csr -key server.key

And sign it, again careful to include the config file

openssl x509 -extfile /etc/ssl/openssl.cnf -extensions v3_req -req -days 365 -in /tmp/multi.csr -CA ca.crt -CAkey ca.key -set_serial x -out multi.crt

And that's about it!

Archive portage logs

Just dump the following script in /etc/cron.daily to compress old portage logs and delete ancient ones, without using logrotate or tmpwatch/tmpreaper. After a year of running lappy, my /var/log/portage dir got to almost 1GB.

#!/bin/sh

## remove really old logs (180 days)
find /var/log/portage -name '*.log*' -mtime +180 -print0 | xargs --null --no-run-if-empty rm

## compress big logs (over 10 KB)
find /var/log/portage -name '*.log' -mtime +3 -size +10k -print0 | xargs --null --no-run-if-empty gzip

Unpack just what you need

The other day I thought it would be interesting to emerge mtree. mtree is an application that generates a detailed file listing, but that's not the interesting part of today's post. I typed emerge mtree and I see 18 MB need to be downloaded. That sounds funny, should be a small utility. Then I saw it was downloading a big pkgsrc archive from BSD. Okay, that figures they distribute everything in one big archive.

So I let it download, now it's in the unpack phase... and still unpacking... and still unpacking. Hey, what's going on? It's already unpacked 500 MB (kudos bzip2!). I know disk space is cheap today and maybe blackbird is showing its age, but unpacking hundreds of megs for this utility seems really inefficient.

I figured I could modify the ebuild to unpack only the relevant directories of the archive. I was right - a small change to the ebuild (well, fix a typo and then try again) and it works... Hardly any disk space used and I also shaved some time off the emerge.

I don't know if this would be accepted as a bug but I decided to share my optimized ebuild anyway in this bug.

Oracle VMware Support

An interesting read about the current status of Oracle support for VMware.

http://chucksblog.emc.com/chucks_blog/2009/05/why-oracle-doesnt-like-vmware.html
http://oraclestorageguy.typepad.com/oraclestorageguy/2009/04/what-the-oracle-vmware-support-statement-really-meansand-why.html
http://virtualgeek.typepad.com/virtual_geek/2009/05/oracle-on-vmware-a-manifesto.html
http://oraclestorageguy.typepad.com/oraclestorageguy/2009/05/response-to-comments-from-alessandro-perilli-blog.html
http://www.virtualization.info/2009/05/emc-attacks-oracle-on-its-vmware.html

More information about running Oracle in VMware:
http://vmware.com/partners/virtualize_oracle_landscape.html
The community: http://communities.vmware.com/community/vmtn/general/oracle
Perfomance Analysis guide: http://communities.vmware.com/docs/DOC-3930

Magic SysRq Ehancement (3)

It would be nice to take the sysrq-visual and add it as a kernel patch. I thought it would be nice to see the leds blinking while the emergency sync is running, because many times you can't know for sure when it's done.

Looking at the code, there is a call emergency_sync(), which is performed asynchronously. It prints a debug message to when it's done, but there doesn't seem to be a nice and clean way to be notified of the completion. I need to figure out which is better, calling a different sync method (synchronous), or adding a notification mechanism to the sync operation (patching the kernel code in a pretty critical area.)

Oracle Listener Configuration Revisited

Configuring the oracle 10.2 listener correctly might not be as easy as it seems.
First off, Oracle has the ability to automagically configure everything without any configuration file. The defaults should usually suffice, but if you want to make some changes it can get a bit confusing.

When the oracle service starts (and more importantly, the listener) it says to where it is logging and where it is taking the listener parameters from (for example $ORACLE_HOME/network/admin/listener.ora). Now, if for any reason Oracle can't find or read this file it will happily continue with it's automatic configuration but will do so silently, while still printing the configuration file name!

This is where it stumped me. I made some changes to that file (changed tracing level) but it didn't make any effect. Later I found the reason was that the file was owned by root, while the listener was running as oracle and had no read permission to the file. It took a while to figure that out.

Anyway, back to the problems I was trying to solve. Namely, the occasional hangs at oracle startup/shutdown. I found out that oracle (again, actually the listener) always tries to resolve the machine hostname (probably from gethostname(), dunno) and connect to the result (in netstat I saw connections to ports 199 and 1521). If there is a bogus resolution (put something unreachable in /etc/hosts), the listener process will hang for the machine's defined connect timeout (usually 2 or 5 minutes).
It seems this happens regardless of what you put down in the (HOST = x.x.x) in the listener.ora file. I tried putting down 0.0.0.0, localhost and 127.0.0.1, but in all cases it still tried resolving the hostname.

The lesson here, boys and girls, is that you must make sure that your hostname always resolves to something reachable. E.g. in /etc/hosts you should have:

127.0.0.1 myhost

or

0.0.0.0 myhost

Fair enough. But there are a few additional things that can be done. First, the old oracle init script (very old, probably taken from RedHat 9) calls lsnrctl before calling dbstart and dbshut.
I noticed that these scripts already handle the lsnrctl, so I removed the additional calls from the init script. The hang is gone! Actually the hang is still happening, but since the oracle scripts start lsnrctl in the background, it doesn't block the oracle init script anymore.

Also a warning message in the listener.log after the timeout expires drew my suspicion to the ONS ("subscription to the node down event is still pending"). I found a couple of solutions online, one is to add SUBSCRIBE_FOR_NODE_DOWN_EVENT_{listener_name} = OFF. That took care of the warning message and some of the connections (the one to 1521 is still there) but it still hangs.

Finally I also commented out the cfgHostname call in the init script since all that crap is not really needed. It's easier to set the HOST in listener.ora to something constant such as 0.0.0.0 or localhost than changing the file and moving directories around everytime the hostname/ip changes.

Magic SysRq enhancement (2)

I've rewrote this module using workqueue instead of timers and it works perfectly now on blackbird (to be tested on lappy soon).

An additional feature is that I now flip the leds twice so the original state is restored and also the blink is not missed if all leds were originally on.

Code is here

BTrace Rocks!

At work we're using a maven based build process. On the build machine it takes a while to finish the complete build (about 2 hours).

One of the final parts of the build is to generate the mvn:site. This stage takes, in my opinion, longer than it would seem necessary. Looking at #top, the machine is almost idle and there are 14 processes listed in #ps, where the last one keeps changing (I guess it finishes a directory and a new is spawned for the next one).

Due to bad design, there's no logging whatsoever in this stage and the machine seems stuck. I wanted to debug this on the build machine but didn't want to distrupt the running build (so can't start in debug mode).

To the rescue comes BTrace. A DTrace clone for java, allowing you to write small java scripts that can monitor and profile running java code. BTrace allows you to attach to an existing JVM process, without requiring any special switches to be added a priori to its command line (although it does support an -agent scenario).

That was so cool! Just downloaded one of the examples on the BTrace site, found the process pid using #jps so I can attach to it, and voila. I get printouts of all file accesses done by maven.
The BTrace examples page is full of actuall useful scripts. For example there's a script to print stacktraces for all threads (similar to #jstack).

Magic SysRq enhancement

I really love the Magic SysRq key. It's always nice to know that you can do some emergency actions even when the system is 99% locked up. What's my favorite sequence? Probably 'subo'. I'm still figuring it out.

One of my beefs with MSRK is that sometimes I can't be sure if it is responding (am I pressing the right sequence?). This usually happens when I get a crash while in X, not seeing the console.
I wrote a small kernel module to add a new action to the Magic SysRq key. The new action's, "Visualize", only purpose is to give back a cue that it's listening.

I copied from some kernel hacking example code for blinking the keyboard leds (should be possible if the kernel still reads commands from the keyboard) and for beeping (although I haven't got that working yet).

First try is using timers ('add_timer'). This seems to be working nicely on lappy but inconsistently on blackbird (sometimes the leds don't light up and sometime they light up but don't go off). I'm pretty sure that the timer function gets called because I can see the printk's. I guess this means that the IOCTL is failing. I'll try replacing the timer with work queue, see if that helps.

P.S.: Another difference between lappy and blackbird: On blackbird the 'v' from the Alt-SysRq-V sequence gets swallowed, whily on lappy it gets printed. Strange.

Googling myself again...

My name comes up a few times in the internet, in various bug reports and forum posts. I found out today it also appears in the hibernate-annotations changelog (quite undeservingly since someone else did all the heavy lifting). Nice.

Profiling Java Applications

A new beta of simon 2 is out. Also, btrace (https://btrace.dev.java.net/source/browse/btrace/docs/usersguide.html?rev=1.6) looks like an interesting alternative to JIP. It even has a visualvm plugin.

Tomcat Logging, Another Rant

For integration with another application I need the logs comming out of tomcat to have predictable names. This is not a problem for my webapp log which is created by log4j. Apparently this can be done with java logging as well. However, the tomcat main log uses JULI by default, which always creates log files with the date embedded in the file name.

There doesn't seem to be a way to configure it otherwise (except to change the logging to use a different mechanism).

ALSA, oh, ALSA

A week a two ago a hal update broke alsa on blackbird. After a few reboots I managed to get sound back, I believe it was fixed by downgrading hal-info to 20080508.

Yesterday I upgraded the kernel to 2.6.29-r1, and after the first reboot, again no sound (I also got the messages about /etc/modules.d in the boot messages as I got the last time this was b0rked).
So, I tried upgrading hal-info back to 20081219 (latest stable) and rebooted. Now it works (and no warning messages).

I'm assuming it's hal-info (maybe there is some dependency between the new hal version and new kernel version?) but maybe it has something to do with the reboot itself. We'll see next time.

GCC 4.3.2

GCC 4.3.2 just got stable on Gentoo a few days ago. For a full, clean, upgrade I need to recompile the whole system. Maybe this weekend (a long one).
I'm wondering, if I'm already recompiling everything, I should consider installing from scratch in 64bit instead of 32bit. Flash has had a 64 bit plugin for a while already, and it seems that sun's JDK is also now sporting a 64 bit plugin. That doesn't leave a lot of remaining objections to 64 bit since the last time I tried.
What about wine, does it work well in 64bit? What other binary blobs do I have that I should worry about?
I recently switched my wireless driver to net-wireless/broadcom-sta, which should work with 64bit.
Let's see... what else... I guess I would need to reinstall oracle-xe. Wait, let's just see what's in my /opt and /usr/local:

VMware, googleearth, sqldeveloper, ...

hmmm... need to think about it some more.

Update: Arrgrggg... Oracle Express doesn't come in 64 bit... Will need multilib after all.

Eclipse Java Execution Environment Description Files

At work I'm working on a java project that needs some command line arguments passed to the JRE to work correctly (maximum allowed memory, etc.).

I have these set in the Installed JREs section in Eclipse's preferences which is all good. However, it gets real frustrating because whenever I upgrade my JDK these get lost. This is because the JDK I had is no longer there an is automatically removed by eclipse, and the new one is detected and added without any customizations.

So I tried to tell my project to use an "Execution Environment" instead of a specific JRE. However eclipse (3.4) doesn't let you modify it's list of execution enviroments (or customize the command line arguments). You can, however, define a new JRE using an Execution Environment Description file.

It was hard to find documentation about this so called .ee file. I found just one page that described the possible settings that can be put in the file, with a short description, but I had to reverse engineer the values for these settings.

Finally I came up with this, which actually works:

## Execution Environment description file for MyProject
-Dee.name=MyProject_EE
-Djava.home=${ee.home}/jre
-Dee.executable=${ee.home}/jre/bin/java
-Dee.executable.console=${ee.home}/jre/bin/java
-Dee.bootclasspath=${ee.home}/jre/lib/resources.jar:${ee.home}/jre/lib/rt.jar:${ee.home}/jre/lib/jsse.jar:${ee.home}/jre/lib/jce.jar:${ee.hom$
-Dee.language.level=1.6
-Dee.library.path=${ee.home}/jre/lib/i386:${ee.home}/jre/lib/i386/server
-Dee.src=${ee.home}/src.zip
-Dee.javadoc.http://java.sun.com/javase/6/docs/api/index.html
-Dee.ext.dirs=${ee.home}/jre/lib/ext
-Dee.vm.library=${ee.home}/jre/lib/i386/server/libjvm.so
-server
-Xmx768m
-XX:MaxNewSize=384m
-XX:MaxPermSize=144m
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Notice the ${ee.home} in the paths. It seems that this is the only variable that can be used in the ee file. It get's replaced with the path to the ee file, so for this to work the ee file must be placed in the JDK's base directory.
The reason I'm using this variable instead of absolute paths is that on the next JDK upgrade I'll just need to copy the file to the new JDK's directory and it will continue working (soft linking the jdk directory doesn't work - eclipse follows the links and creates a broken JRE).
Of course you could generate a new ee file for each JDK version.

I also created another variant of this ee file with JIP command line arguments, so I have two JREs defined. This way when ever I want to profile my project, I don't need to create a special runtime configuration, but just choose a the other JRE. To make things even easier, I have two Server (tomcat) definitions, one with each JRE. Starting tomcat with JIP is just a matter of starting the second server instead of the first. Pretty nifty.
BTW, the extra setting for JIP is:

-javaagent:/usr/local/jip-1.1.1/profile/profile.jar -Dprofile.properties=/var/proj/MyProject/webapp.profile.properties

P.S.: It would be great to see this file coming from the upstream package. Maybe sun can include it in their JDK distrubtion, or maybe it could be added to gentoo's package (possibly with an eclipse/OSGi use flag?)

First Post

This blog is dedicated to any kind of interesting stuff I encounter while developing software, fiddling with linux or anything computer related.

I figure I can just blog it instead of emailing it to myself (for example when I come up with a new script).