Sunday, May 22, 2011

Agile Estimating and Planning

Estimating work is hard. It's harder to estimate accurately the farther you look ahead. It's also very hard to estimate when you're dealing with big chunks of coarse grain work, or work which is hard to compare to stuff you did in the past (using new technologies, research ideas, etc.). This is why we use rough story point estimation when estimating stories in the release planning, and leave estimation of tasks in hours only to the sprint planning.

The effort for getting an accurate estimate (as accurate as an estimate can be) is very high, but the relation between effort and accuracy is not linear. Actually you get a big part of the accuracy early on in the effort scale (and then you need exert more and more effort for diminishing gains in accuracy). I like this graph from Mike Cohn's Agile Estimating and Planning:

In order to prevent sprint planning meetings from taking ages, we timebox the discussion on each story (10 minutes). In this time the story is explained, questions are asked and clarification given and tasks are created with hour estimates. This timebox may seem constrained, but this really emphasizes that accuracy of the estimate is not so important. The important goal of the sprint planning meeting is to understand the stories a little better (to go up much higher in the accuracy axis with minimal progress on the effort axis).

It is also important to understand that the initial "plan" and estimates are just a starting point and should be constantly updated throughout the sprint. The farther into development of a story you are, the higher the accuracy of the estimates will get until they converge with the actual time when the story is finished.

So, to reiterate - the point of planning is to get a better idea of what the story is, not to predict when it will be done. The tasks are reminders (but may be updated or completely changed when better information is available) and the hour estimates are only a starting point.

When the stories are small enough (1-3 days work) and are similar to work previously done by the team, then the initial estimate and task breakdown may be very accurate. But if you have bigger or less clear stories it is less likely. Also when looking on the whole sprint it becomes harder to predict since there sometimes unexpected shifts in priorities (and external distractions), changes in story requirements when actually starting to work on them, and some stuff just isn't taken into account (dependencies between tasks, teams or team members and who actually gets to do the task and if she might need some training from the guru in that field).

You usually stop sprint planning when you've filled up the entire team's capacity according to the sum of estimates of all the tasks. But does that mean that this is what you're going to deliver in this sprint?
I think no. I ask team members to estimate in ideal hours the net work. So in the unlikely event that everything - every single thing - goes smoothly during the entire sprint, then yes, this will be the output of the sprint. But even if one small thing goes wrong, some stories will not be delivered.

This is where sprint commitment comes in. You need to decide on some "optimism-factor" and split these stories into stories the team is committed on (and there's a very good chance they will be delivered unless we have a complete catastrophe) and stretch stories.

We don't stop working when we finish all the committed stories. We planned the stretch stories and intend to deliver them too. We're just saying don't count on it. I personally believe it is better to deliver one less story that we didn't commit to, than to make a bigger commitment and not deliver on it. The point is to generate trust by constantly delivering what you promise. And if you're constantly taking chances and breaking your promises, who cares that you committed to deliver more in the beginning of the sprint (the buffer will simply move to the consumer, the PO: "ahh, they never actually deliver what they promise, so I won't tell the customers about this feature yet)"? And if everything does actually go well, you're still going to deliver those stretch stories (and give the Product Owner a happy surprise).

A factor of 70% is a good place to start (that's after the team's capacity is calculated to be about 75%-80% of the actual working hours in order to account for meetings, breaks and other "necessary evils" ), and fine tune from there. If you deliver less, increase the buffer. And if you deliver more you can carefully decrease the buffer.

And if you're feeling really good about yourself - drop hour estimate completely and take work for the sprint based only on story points and previous (average) velocity.

Sunday, April 17, 2011

The perfect linux audio setup - ALSA with late 2010 mac mini (macmini4,1)


One of the greatest additions to the unibody model mac mini is the addition of an HDMI output (through the nvidia MCP89).
Not only does this allow connecting another HD screen, but it marks the inclusion of HD multi channel audio. Not only audio over HDMI is supported but also the mini-DP output received the welcomed update. And when you thought it can't get any better - each has its own audio device (analog = 0, DP = 7, HDMI = 8). Why is this such an improvement over the optical S/PDIF in the previous models? Several reasons:
  1. The obvious - 8ch LPCM > Dolby 5.1
  2. One less cable to the receiver :)
  3. I can now use the dual analog/digital output as an analog output (because I have a multi zone receiver and the second zone can only use analog sources).

First off - getting the HDMI audio to work in linux
Kernel support - need to set SND_HDA_CODEC CIRRUS and SND_HDA_CODEC_HDMI (I'm using 2.6.38, it used to be SND_HDA_CODEC_NVHDMI).
This however is not enough on the mac mini because I get write errors when I try to open the hdmi device. Quite a bit of web research has revealed the following changes are needed:

1. Load the module with the options: snd_hda_intel.enable_msi=0 snd_hda_intel.probe_mask=0xffff,0xfff2 snd_hda_intel.model=mbp55

2. Use /etc/asound.conf and /usr/share/alsa/cards/HDA-Intel.conf from this post. This is supposed to fix the switched channels, but I'm not sure it's really required (looks a bit too complex).

This gets you basic but working setup :)

Now for the fun stuff
A few things are missing in the basic setup. First off, the HDMI output volume is fixed and the mixer doesn't affect it. I haven't figured out how to get the alsa mixer settings right. So, we'll use the alsa softvol plugin (with some degradation of quality I guess...). Also, the output needs to be dmix'ed so multiple applications can use it at the same time. And lastly, I want the audio to be distributed to all outputs simultaneously so I can hear my music on my home theater (HDMI), monitor attached headphones (DP) and in my bedroom (analog). This is done using the alsa multi plugin.

Here is my .asoundrc
# dmix the hdmi output
pcm.!hdmi {
type dmix
ipc_key 1024
ipc_key_add_uid false
ipc_perm 0660
slave {
pcm "hw:0,8"
rate 48000
channels 8
period_time 0
period_size 1024
buffer_time 0
buffer_size 4096
}
}

# dmix the dp output
pcm.dp {
type dmix
ipc_key 1025
ipc_key_add_uid false
ipc_perm 0660
slave {
pcm "hw:0,7"
rate 48000
channels 2
period_time 0
period_size 1024
buffer_time 0
buffer_size 4096
}
}

# dmix the analog output
pcm.analog {
type dmix
ipc_key 1026
ipc_key_add_uid false
ipc_perm 0660
slave {
pcm "hw:0,0"
rate 48000
channels 2
period_time 0
period_size 1024
buffer_time 0
buffer_size 4096
}
}

pcm.softvol {
type softvol
slave {
pcm "all"
}
control {
name "PCM"
card 0
}
}

pcm.!default {
type plug
slave.pcm "softvol"
}

pcm.all {
type plug
slave.pcm "multi"
ttable.0.0 1.0
ttable.1.1 1.0
ttable.2.2 1.0
ttable.3.3 1.0
ttable.4.4 1.0
ttable.5.5 1.0
ttable.6.6 1.0
ttable.7.7 1.0
ttable.0.8 1.0
ttable.1.9 1.0
ttable.0.10 1.0
ttable.1.11 1.0
}

# distribute audio to both analog and hdmi outputs
pcm.multi {
type multi
slaves.a.pcm "hdmi"
slaves.a.channels 8
slaves.b.pcm "analog"
slaves.b.channels 2
slaves.c.pcm "dp"
slaves.c.channels 2
bindings.0.slave a
bindings.0.channel 0
bindings.1.slave a
bindings.1.channel 1
bindings.2.slave a
bindings.2.channel 2
bindings.3.slave a
bindings.3.channel 3
bindings.4.slave a
bindings.4.channel 4
bindings.5.slave a
bindings.5.channel 5
bindings.6.slave a
bindings.6.channel 6
bindings.7.slave a
bindings.7.channel 7
bindings.8.slave b
bindings.8.channel 0
bindings.9.slave b
bindings.9.channel 1
bindings.10.slave c
bindings.10.channel 0
bindings.11.slave c
bindings.11.channel 1
}



Note that the DP output is set to 2ch 48KHz because it is connected to my monitor and it seems that it is limited in its capabilities (it only supports headphones anyway). This also means all other outputs need to be set to 48KHz otherwise you get errors from the multi-device pcm (it seems that all slaves used by the multi pcm need to have the same parameters). If the DP output is disabled I can use 96KHz or even 192KHz...

Next Steps...
I need to try out pulseaudio again. It might reduce the amount of alsa plugins in use. I also need to figure out how to record audio (for icecast).

Sources:
http://wiki.xbmc.org/?title=HOW-TO_set_up_HDMI_audio_on_nVidia_GeForce_G210,_GT220,_or_GT240
http://forum.xbmc.org/showthread.php?t=74778

Tuesday, March 29, 2011

Workaround for blank unlock dialog in kde with cairo-dock running

Just execute the following after cairo-dock is running:
xprop -root -remove RGB_DEFAULT_MAP


The following wrapper scripts takes care of it automatically:
#!/bin/sh
/usr/bin/cairo-dock &
sleep 5
xprop -root -remove RGB_DEFAULT_MAP


Sources:
http://bugreports.qt.nokia.com/browse/QTBUG-15092
https://bugs.kde.org/show_bug.cgi?id=262064

Tuesday, February 1, 2011

Using github and layman for my own gentoo overlay

Since I almost always have a local portage overlay in any gentoo machine I create, I've started to maintain in github.

Now, originally I just pulled the repository to /usr/local/portage.git on each machine and added the dir to PORTDIR_OVERLAY in make.conf.

But since I already have other third party overlays and maintain them using layman, I though why not use that for my own overlay as well?

So all is required is to create a custom repositories xml file and point to it from the layman config.

The repostiories xml (this one is using https since I can't access git from work - the down side is that it ask for password on every layman sync, but git://github.com/asssaf/portage.git or git@github.com:asssaf/portage.git can be used where this is not a problem):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE repositories SYSTEM "/dtd/repositories.dtd">
<repositories xmlns="" version="1.0">
<repo quality="experimental" status="unofficial">
<name><![CDATA[mygithub]]></name>
<description><![CDATA[mygithub]]></description>
<homepage>https://github.com/asssaf/portage</homepage>
<owner>
<email></email>
<name><![CDATA[Assaf]]></name>
</owner>
<source type="git">https://asssaf@github.com/asssaf/portage.git</source>
<feed></feed>
</repo>
</repositories>

And in /etc/layman/layman.cfg, add the line in bold:

overlays : http://www.gentoo.org/proj/en/overlays/repositories.xml
file:///etc/layman/localrepositories.xml


The overlay will be available for addition using 'layman -a mygithub' and pulled into (by default) /var/lib/layman/mygithub.

Bluetooth network connection to the Palm Pre from a linux machine

The Palm Pre in its current version has a very limited bluetooth stack. It only allows outgoing connections to audio devices. No serial port, PAN or even sending contacts to another phone.

It does however support incoming PAN connections.
This is how I do it.

1. First, pair the phone with your machine using your favorite gui (bluedevil, blueman) or command line utility.

2. Set up IP configuration for the bnep0 interface
In gentoo this is done in the /etc/conf.d/net file. I just needed to add the following line:
config_bnep0="10.1.1.61 netmask 255.255.255.0 broadcast 10.1.1.255"


3. Firewall - set it up to allow traffic. Also need to set up masquerading/proxyarp for returning traffic.

4. Start a connection:
pand --connect AA:BB:CC:DD:EE:FF --role PANU --nodetach --master

Where AA:BB:CC:DD:EE:FF is the mac of the palm pre bluetooth interface

This will bring up the bnep0 interface and allow connecting to the pre via ssh, etc.

Known issues:
1. Most WebOS programs will not be aware of this connection (you won't be able to use the browser). But most terminal operations will be.
1. DHCP, DNS: The DHCP configuration is geared towards a mobile wifi hotspot and is kind of screwed up (or at least I don't understand it) for this purpose.
2. Proxy ARP / IP bridging mode instead of NAT (TODO)

Tuesday, January 25, 2011

Tomcat and Logging

Tomcat is a very popular java application server. It is very light weight and easy to use. It's derivative, tcserver, is also very nice. It has all the goodness of tomcat, and on top of that it gives you the feel of a real J2EE app server (tomcat is only java servlet compliant).

Logging from your webapp in tomcat is a piece of cake. For example, if you like to use SLF4J logging API together with logback as the backend, you just drop the jars in the webapp's WEB-INF/lib and the configuration (logback.groovy) in WEB-INF/classes and you're good to go.

You will notice, however, that something else is eating away your disk space beside your webapp's logs. Tomcat has its own logging. It writes several files, has its own logging configuration (and logging technology) and even does its own rotation for (only) some of the files. A log rotation scheme that can't be easily disabled and might not fit your global rotation scheme.

By default tomcat uses juli, which is based on the standard java.util.logging with some enhancements for supporting per-webapp configuration. This only matters to those that don't have their own logging set up inside their webapp. Also, the default configuration is quite surprising. It sets up several files for different categories that may or may not interest you.
There's:

manager.log and host-manager.log for the respective webapps (which you may not want to have deployed, but the empty log files will still be created).
localhost.log will have message about servlets coming up and down.
catalina.log will contain mostly internal tomcat messages.
Some logging messages will be sent to stdout and get mixed with actual direct writes to stdout/stderr as well as with the output from scripts that start and stop tomcat. If you redirect this to a file (catalina.out) it is not rotated and can grow very big over time. Some messages are send to both catalina.log and catalina out, which is confusing if not just redundant.

All the files created (this doesn't include the console output) use the juli file handler that by default adds a date stamp to the file name (for rolling). It is not configurable! You simply can't remove it (you can just change the prefix and suffix but there will always be a date in the middle).

These are all the options I can think about (after scouring the net for solutions):
* Replace tomcat logging with log4j or logback (requires dropping the jars in the bin dir and changing the startup script to include them in the classpath)
* Replace the juli FileHandler with the standard java.util.logging FileHandler and have only a single handler for everything, since we don't need the per-webapp logging (we set that up within the webapp)
* Use SocketHandler to write to syslog instead of directly into a file
* Use MemoryHandle since most of these message can be ignored until we actually have a problem
* Use nop logging configuration, but again we lose troubleshooting information

In any case setting the sallowOutput option to true in the context configuration file will catch some messages written to stdout/stderr and redirect them to the webapp logger. It doesn't catch all messages though (I think ones that are written early).

For rotation I want to use logrotate so it is possible to:
* Use SocketHandler to write to syslog, and syslog can be told to reopen its log files
* Enhance the logging framework to listen to a signal to reopen its files
* Use cronolog (but again we lose control of the file naming since it adds a date suffix - I want to use numbers)
* Use the (nonatomic) copytruncate option in logrotate but risk losing some messages

In the end I decided to do the following:
* Comment out everything it tomcat's logging.properties except a single catch-all FileHandler, using the default java.util.logging FileHandler instead of juli's. This includes disabling the ConsoleHandler (I don't need it since I'm running tomcat in a server and never interactively)
* Rotate the created files (the single log and the stdout redirected to a file) using logrotate with the copytruncate flag (this is not safe for the stdout file, but I don't expect it to grow much now that it doesn't include any of tomcat's own logging)
* Enabled the swallowOutput flag
* Use syslog for all the logback logs and rotate with logrotate them without copytruncate