Chicago Python User Group Goes Ninja

Here is the official announcement for the Chicago Python User Group (chipy). The important thing to notice here is that I'll be presenting (or maybe the important thing is the free pizza.)

Chicago Python User Group
=========================

Calling all Chicago Python Ninja's, this will be our best meeting yet--
this Thursday. (if not, there will be some pretty darn good pizza in
here)

Insert funny comic here:

We have some interesting dragon slaying talks lined up this month.
Starting off, Garrett will mix theory with practice in another one of
his highly thought provoking talks. Professor DiPierro will go all
out with the tools that serve as his student's weapons in the real
world. Josh will brave the Python waters as n00b who comes from the
dark side and will probably tell us a interesting Ninja story while
displaying Pythonic charm and charism--all while balancing on one
leg ;) Do not try this at home. In fact, staying home will not be as
much fun or as educational, folks.

This *will* be our best meeting yet.

Topics
------

* Bazaar within Subversion, Personal Workflow - Garrett Smith
* T2 (web2py tier two) and T3 - Massimo DiPierro
* Beginning Python with AppEngine - Josh Cronemeyer

NOTE: Due to the high level of interest in various talks, I have
limited talk length to 25-30 min. Offenders will be severely punished
by my personal Ninja friend.

When
----

Thursday, December 11th, 6:30pm Pizza
Thursday, December 11th, ~ 7pm Meeting

Location
--------

ThoughtWorks, 200 E. Randolph 25th floor.

Prepare to be perfectly astounded by my one legged python slinging, pizza eating acrobatics. I'm going to announce the deployment of the best web application ever at the best python meeting ever. It will be cosmic. A total eclipse of dynamic languages.

A history of the security at the Aon Center

The AON security ninjas... Should I start from the beginning? Yes? Ok. It is a looong story about ancient Japan. Two warring feudal lords loved the same woman, Iwasaki. Both lords tried to win Iwasaki over with the traditional Japanese gift of love: wooden clogs inlaid with a jade Hello Kitty. Then, on the day Iwasaki was to choose which lord to marry she died tragically of a hangnail caused by poorly fitted clogs. Each of the lords blamed the other for her death and the bitter conflict that ensued instigated centuries of assassination and espionage. Long after both sides of the conflict were all killed the clans of ninjas that were involved, known as the Iwasaki ninjas, swore their allegiance to Iwasaki's family. The family became wealthy and powerful, but during the 19th century the head of the Iwasaki family lost an epic game of flip cup to a young John D. Rockefeller. The dishonor was so great that afterwords the allegiance of the Iwasaki ninjas transferred to Rockefeller. The ninjas were the secret of Rockefeller's success and they lived in his mansion until Rockefeller's death. Suddenly in need of a place to live, and being preferred shareholders in Standard Oil, they took over the newly constructed Standard Oil building in Chicago. There they assumed responsibility for building security.

Forty years later their only allegiance is to the building and their union, the Local Ninjitsu 393. If you visit, pay attention to the dozens of large gongs that line the walls. Each gong represents a generation of Iwasaki ninjas. DO NOT TOUCH THE GONGS! A visiting member of the Chicago Ruby User Group named Josh once rang a gong. Later that evening he swallowed his own tongue under mysterious circumstances while giving a talk about metaprogramming in ActiveRecord. Detectives later found that one of his note cards had been altered to say:

I need not your needles, they're needless to me;
For kneading of noodles, 'twere needless, you see;
But did my neat knickers but need to be kneed,
I then should have need of your ninjas indeed.

The murder remains unsolved. Frankly, for most of us in the meeting that notecard was the only part of his talk that made sense, but you know, metaprogramming is hard. The guiness book of world records lists that paragraph as the most deadly tongue twister known to man. It is also worth mentioning that ever since that day the poor fellow has been known in the Ruby community as "Because, the unlucky stiff".

Syntax Highlighting In Blogger

I have pulled various tricks in the past to get syntax highlighting in my code samples. In the past I only had highlighting for ruby code. I think I finally have a good solution and it works for any C style language.

Luka Marinko's blog has simple instructions to get this to work. Thanks Luka! Now here is some nonsense code to show what kind of highlighting you get with Java, Python, and Ruby. All the useless imports and requires are just so we can see those kinds of statements getting highlighted.
JAVA

import java.io.*;

class HelloWorld{
public static void main(String[] args) {
 hello();
}

public static void hello(){
 System.out.println("hello world");
}

}

RUBY

require 'rubygems'
class HelloWorld
def initialize
 puts "hello world"
end
end

HelloWorld.new

PYTHON

import sys
class HelloWorld:
def __init__(self):
 print "hello world"

HelloWorld()

Unit Test Your Google App Engine Models

I've been working on a project using Google App Engine (GAE) called "les Freres Jacques" That manipulates images of people's face onto the cover of this old french LP. Look for it to be out in a couple months. Anyway, My favorite way to start any project is by doing TDD. Unfortunately I'm new to python and GAE simultaneously so I had to do plenty of research to figure out how to unit test a GAE app. Most importantly for me was the ability to test my models that are based on google's datastore api. What follows is some information to get you started writing unit tests against a GAE model. First, a list of the tools you need to install.

• Nose is a tool for running your python unit tests.
• NoseGAE is a plugin for nose that bootstraps the GAE environment.

The easiest way to install these is with python's easy_install, which as far as I can tell is similar to ruby's 'gem' program and perl's 'cpan' program... though I don't know if it resolves dependencies automatically. Anyway, on OSX easy_install is installed by default so you can simply type

sudo easy_install nose

sudo easy_install nosegae

Now let's create a test to exercize a simple GAE model object. Here is a file called test_simple_model.py

import unittest
from google.appengine.api.users import User
from test_example.simple_model import SimpleModel

class TestSimpleModel(unittest.TestCase):
def test_creation(self):
 user = User(email = "test@foo.com")
 model = SimpleModel(goo_user = user)
 model.put()
 fetched_model = SimpleModel.all().filter('goo_user =', user).fetch(1)[0]
 self.assertEquals(fetched_model.goo_user, user)

The nose tool we installed earlier gives us a program called nosetests to run. When you call it it looks through your project and runs all your tests. We should call it now with the google app engine switch.

nosetests --with-gae

Wheee!! It is a lovely failing test.

======================================================================
ERROR: Failure: ImportError (No module named simple_model)
----------------------------------------------------------------------
Traceback (most recent call last):
...
from test_example.simple_model import SimpleModel
ImportError: No module named simple_model

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

Good. Now we need to write some code to get our test passing. Here is what I wrote in a file called simple_model.py.

from google.appengine.ext import db
class SimpleModel(db.Model):
goo_user = db.UserProperty()

Now when I run

nosetests --with-gae

I get the lovely

.
----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

and I am happy because I see how I can do TDD with GAE! Here is a list of references I used to figure this stuff out. Hope you find them useful. You can find the full source of this example here.

• Python Easy Install
• Python Unit Test documentation
• Python Mock Object Framework
• Python Nose
• Python NoseGAE

Update: 11/30/08
The datastore persists between tests which isn't usually what I want to happen. I submitted an issue on the nose-gae issue tracker. In the meantime here is a workaround to make sure the datastore is flushed between runs. Add this method to your test class and call it in your setUp method.

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub

def clear_datastore(self):
 # Use a fresh stub datastore.
 apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
 stub = datastore_file_stub.DatastoreFileStub('appid', '/dev/null', '/dev/null')
 apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)

Update: 1/29/09
Reading Dom's well researched and documented post on testing App Engine applications. I thought I better spruce up my own post by adding a citation. The code from the clear_datastore method above comes from this message posted on the google app engine google groups mailing list.
Update: 5/17/09
There is currently an issue with nosegae. See defect 18. There are patches that fix the issue posted there. I downloaded the source code, removed the subversion directories, patched the code and ran

easy_install .

in the root directory of the code. That fixed the issue for me.
Update: 5/17/09
As of GAE SDK 1.2.1 the appid of your datastore stub must match your appID. Make sure your call to DatastoreFileStub uses your actual app id.

Deep Clean Your Subversion Working Copy

awk and xargs. Deep cleaning action for those hard to reach places. Sometimes you just want your subversion working copy to be clean. If you have files with changes you can just revert things, but what about files that aren't under version control? Those are harder to clean up, especially if they are spread all over the place. Here is what I do when I want my working copy to be identical to the repo:

svn st | awk '{print $2}' | xargs rm -rf

That will remove all files that are out of sync with the repository. Then simply update to restore things you deleted and get up to date.

svn up

Access Google App Engine Development Server Remotely

By default Google AppEngine's (GAE) development webserver doesn't accept remote requests. If you'd like to make your devel GAE application available remotely you can use a simple proxy to forward requests. I chose to use pen proxy because it is sooooo simple. To install pen on ubuntu do

sudo apt-get install pen

to do it on OSX with macports do

sudo port install pen

Once it is installed startup your GAE application. Then start up pen like this:

pen 8079 localhost:8080

Now pen will forward requests from port 8079 to your app running at 8080.

A Software Journeyman

For the past year or two I have been hearing the words 'software' and 'craftsmanship' used together more and more frequently. Uncle Bob has been talking about it for most of a decade (click the craftsman link). So today when I heard about Corey Haines' pair programming tour I decided these two things must be connected. Corey is traveling through Ohio, Michigan, Illinois and Indiana. Every place he stops he is going to meet different software craftsmen and pair with them. He is a true Journeyman software craftsman. Wikipedia has this to say about Journeymen craftsmen.

In parts of Europe, as in later medieval Germany, spending time as a journeyman (Geselle), moving from one town to another to gain experience of different workshops, was an important part of the training of an aspirant master. Carpenters in Germany have retained the tradition of traveling journeymen until today, although only a small minority still practice it.

Very cool! If you want a chance to hear from aspirant master craftsman, Corey Haines, he will be talking about his travels as a Journeyman programmer and his chain of command gem at the December meeting of the Chicago Ruby User Group. Details are here.

ChiPy Meeting at ThoughtWorks

The December Chicago Python User Group will be hosted at ThoughtWorks! This will be the best meeting of any kind, ever. Be there! Meeting and RSVP details to follow. Oh, and if the flyer doesn't make sense to you, I usually help out with the Chirb meetings at our office and I couldn't help doing a sort of mashup with a reference to _why's foxes. See! Ruby and Python can co-exist at ThoughtWorks. Hope you like the flyer. I made it with inkscape.

Fine Print:
My loving imitation of _why's foxes appear courtesy of the CC license of the original work.

Therefore this work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.

Update Bash History in Realtime

I have long been annoyed by the behavior of Bash's history file. If you use multiple terminals all history is lost except that of the last terminal closed. The correct behavior should be to save all history from all terminals! An easy way to make that happen is just to save commands to the history file in realtime. Thanks to the Linux Commando I know how to make this happen. Here is the secret:

shopt -s histappend
PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

Put those lines in your bash_profile or bashrc. The first line tells bash to append to the history instead of completely overwriting. The second line calls history -a every time the prompt is shown, which essentially appends the last command to the history file. So simple. I wish I had known this years ago!

Lazy Quantifiers and Negative Lookaheads

The time has come,' the walrus said, 'to talk of many things: of shoes and ships - and sealing wax - of cabbages and kings. 'Lazy Quantifiers' and 'Negative Lookaheads'. The names somehow remind me of characters Lewis Carrol would think up or insults Safire would dream up for Agnew. These are the Tweedle Dee and Tweedle Dum of regexes. Today I had a regex challenge and it was a doosie! I wanted to use a regex to verify certain HTML elements all showed up in a specific HTML table. The problem boiled down to matching a < TR > to a < /TR > without getting hung up on any tags inbetween.

Let's make this concrete and offer the text we want to match (extra spaces are added between angle brackets so blogger won't swallow up my html):

< TR > < TD id = "tweedledee"> tweedledum < /TD > < TD > Error < /TD >< /TR >

Given that HTML, I want a regex that could identify a table row that has a column with tweedledee in it and an error message. The first thing I thought of didn't work. I imagined a regex something like this

/< TR[^(<\/TR)*tweedledee[^(<\/TR)*error/mi

Which to me meant match something that starts out like a table row, then has some arbitrary number of characters that are NOT the end of a table row, then match tweedledee then match some other arbitrary number of characters that are NOT the end of a table row, then match error. Ignoring newlines and case. The problem there is with my understanding of character classes. As you know [] denote a character class. One important feature of a regular expression character class is that it can only match one character at a time. Ohhhh Noooo! So how do we make this work? At first I thought the answer was the lazy quantifier (as opposed to standard greedy quantifiers), but according to my trusty book, "Mastering Regular Expressions," by Friedl the appropriate Tweedle brother is the Negative Lookahead. A Negative Lookahead, denoted (?! ...), is a positional matcher that is successful if at a given point in a regex it cannot match what is to the right of it. After confirming that Ruby does indeed support the Negative Lookahead I came up with the following solution.

/< TR>((?!<\/TR).)*?tweedledee((?!<\/TR).)*?error/mi

Which says match something that starts out like a table row, then any character which is not the end of a table row, but only up to the point that tweedledee occurs, then any character which is not the end of a table row, but only up to the point that error occurs. Ignoring newlines and case. So it turns out I needed to use both tweedle brothers to solve my delimma. Now when am I going to get a chance to use the other lookarounds, the positive and negative lookbehinds? And will I ever be able to look at a pair of TD tags again without thinking of Tweedledee and Tweedledum? Let's ask the king. 'Begin at the beginning,' the King said, very gravely, 'and go on till you come to the end: then stop.'

Navigating Larger Ruby Codebases with Vim

Lately I've been doing lots of work with Ruby, and Vim has been my editor. Working with a larger codebase makes it important that your editor helps you find things easily. Here are three of the most important code navigation functions that you simply cannot do without:

1. Jump to definition. In a large codebase you will find many method calls to methods you've never seen. Being able to quickly drill down through that unfamiliar code and to get back to where you started is massively important. This applies to unfamiliar classes, constants and variables as well.
   
2. Find method usages. Often it is nice to look at how a method is used elsewhere in the code. Or if you are refactoring a method you frequently need to modify the code that uses that method.
3. Plain old search. For sifting through a big codebase you have to be able to do a text search and to be able to quickly jump back and forth between possible matches.

All of this is standard in an IDE, but how do we do it in vim?

For 'Jump to definition' functionality Vim uses Ctags. What exactly is ctags? Ctags is a tool for indexing a source tree of language objects so that they can be found quickly with a text editor. Supported languages include: Assembler, AWK, ASP, BETA, Bourne/Korn/Zsh Shell, C, C++, COBOL, Eiffel, Fortran, Java, Lisp, Lua, Make, Pascal, Perl, PHP, Python, REXX, Ruby, S-Lang, Scheme, Tcl, Vim and Yacc. To install on OSX do "sudo port install ctags" on debian/ubuntu "sudo apt-get install ctags". Usage is simple. To index your code go to the root of your source tree and type "ctags -R ." (that period isn't a period it is the current directory). That will create a file called tags which is an index of your source code. Now when you open vim from that directory you can use ctags. Basic usage is to move your cursor over a method, class, or variable you want to see defined. Then type "ctrl ]" and it takes you to the definition. If you want to go back from where you have come type "ctrl r" and you return. Vim keeps a stack of places you have jumped so you can drill down deeply into the code with "ctrl ]" and then find your way back with "ctrl r". The manual is here. A nice vim plugin to use with this is autotags. Autotags automates the maintenance of your tags files so that you don't have to keep running the ctags command when you change the code. Try it.

Find method usages is a different problem, and unfortunately ctags won't help with it. This website seems to have a complete discussion of possible solutions to this requirement. I played with idutils a bit, but it doesn't support ruby. In the end I gave up on a vim/ruby solution to find method usages and I fall back on general searching to fulfill this requirement. Lame-ish.

A general text search however is well supported in Vim. I like to use the Grep plugin. It has good quality documentation so I won't bother with a lengthy description of how to install or use it. My typical usecase is to do a recursive grep across the codebase for something using the :Rgrep command. Then I can look over the various matches and the Grep plugin lets you jump around among the files. It is a very civilized way to search! I should probably point out that on my OSX setup I had to add the following lines to my vimrc

"fix grep
:let Grep_Find_Use_Xargs = 0
:let Grep_Default_Filelist = '*.rb'

Even on a codebase with a couple hundred ruby files grep seems blazingly fast so not being able to do indexed searches wouldn't be a problem for all but the largest code bases. One thing you will have to figure out once you start using the grep plugin is what vim buffers are and how they work. The Vim docs say a buffer is a file loaded into memory for editing and that all opened files are associated with a buffer. I recommend reading this for more info. Understanding buffers will make you more efficient and change the way you work.

The common thread among all these tools and plugins is that time spent outside of your editor is time not spent coding. If you find yourself opening and closing vim every time you edit a file during a coding session, you are doing it wrong. Would you open and close eclipse every time you wanted to edit a different file? Well maybe you would if it was as fast as vim ;) But seriously. Spend some time learning how to navigate your codebase efficiently and you will become more effective.

I guess to try and put some kind of summary on this I'll just say that while it can require some effort to learn and configure, Vim can make you a very effective programmer, and with the right plugins in your arsenal I think you can outdo an IDE, even on larger codebases.

A Difficult Road

McCain's concession speech was the highlight of his campaign. He was kind. He was gracious. It was refreshing. Thanks John, and let me highlight one point he made in that speech.

"The road was a difficult one from the outset..."

How right you are, John. The road was difficult. Check out this map of a poll taken 2 years ago.


In that poll Obama got only 28 electoral votes. Two years spent organizing and campaigning transformed that map. Obama ended up with 364. A difficult road, indeed.

Shoes Apps With Multiple Windows

From Cuberick

Creating a shoes app with multiple windows is as easy as using the window method. Here is a simple example

#!/usr/bin/env open -a Shoes.app
class Examplish < Shoes
url "/", :index
url "/new", :new_window

def index
 para link "make a new window", :click => "/new"
end

def new_window
 window do
   para "a new window"
 end
end

end
Shoes.app

However there are a few gotchas hidden in this goodness. First notice that if you execute this code the original window is now blank. At first I was surprised, but what is happening is that since you have clicked a link shoes expects you to visit a new url. We aren't supplying one. Simply add

visit "/"

after the window block to tell shoes to forward you on to the same page you were at and now we have the expected behavior.

Now, the big gotcha is that inside the window block the self object is completely different than it is outside the window block. Even more frustrating is that methods defined on the parent aren't available to the child. This can be baffling, a cause of duplication and it turns out a work around is difficult. For example, lets say we have a method called header in our main window that puts a header on the page, since self is redefined inside the new window we no longer have access to that function, therefore we have to re-implement the header function inside the window block. We need a place to put shared code! It won't be easy to find a place. We can't use a module becuase all the methods we need to add content to the page, things like 'para', 'link', 'stack', 'flow', all only exist on classes that extend Shoes, and we need a reference to the specific instance of Shoes.app that is associated with our window.

How can we have shared code when the methods we depend on only exist on a particular instance of Shoes.app? I haven't really found a satisfactory workaround to this problem. The best I have come up with is to put all shared code into a separate class. That clearly solves the problem of giving it a place that is is accesable, but what about the problem of accessing all the magic shoes functions? For that I make our new class a proxy of self (the Shoes.app object). Here is code to illustrate what I mean.

#!/usr/bin/env open -a Shoes.app
class Examplish < Shoes
url "/", :index
url "/new", :new_window

def index
  @proxy ||= ProxyHelper.new(self)
  stack do
    @proxy.header
    para link "make a new window", :click => "/new"
  end
end

def new_window
  window do
    @proxy ||= ProxyHelper.new(self)
    stack do
      @proxy.header
      para "a new window"
    end
  end
  visit "/"
end

end

class ProxyHelper
def initialize(app)
  @app = app
end

def header
  banner "Shared Code!"
end

def method_missing(meth, *args, &block)
  @app.send(meth, *args, &block)
end
end

Shoes.app

Notice how you pass the self or Shoes.app object in when you create the proxy object. Then with a bit of simple method missing magic you can now call all the magic shoes functions seamlessly... well, almost. I have noticed that the link method doesn't work unless you call it directly on the Shoes.app object. And that brings up a bigger issue with this solution. Since all the urls are defined on the other Shoes.app object you can't use any of the navigation routes. Bummer! This is a major bummer! I hope I am missing something and somebody can tell me the right way to go about this, because if not it seriously limits the usefulness of multiple windows in shoes. To my mind the proper behavior inside of a window block is to make sure the new instance of Shoes.app has access to the methods defined on the parent window as well as all the urls. I guess the Shoes::Widget is the only supported way to share code among windows, but as cool as the Widgets are, they don't seem to fit the pattern I am talking about. I'm going to take this up with the shoes mailing list, they can probably sort this out.

Continuous Integration

Here is an old diagram that was in my slide deck at OSCON 2007. Sorry I managed to cut off the .WAR file suffix for the cruise build artifacts. Anyway, I am publishing it here for easy reference.

Ruby VS Java

Recently in a conversation with a Ruby dev. I was trying to make an analogy about the differences between Ruby and Java. For some reason I used cars. It went something like this.

Going with Java is like driving a Ford Crown Victoria. The Vic. might not be much fun, but it is reliable, easy to find parts for, and boy, you sure can fit lots of stuff in the trunk. Going with Ruby is like driving a little roadster. It is fast, and you will have lots of fun with it until you try towing your boat. Oh, and by the way, finding a mechanic for that exotic Japaneese machine might be tough.

The main point of the analogy is that from a developer perspective programming with Ruby is often far more attractive than programming in Java. But it makes some other points as well: Java is ubiquitous, it has a library for everything and it is embraced by enterprise. It also highlights how powerful Ruby is, and how much smaller the developer base is. Unfortunately depicting Ruby as a sports car makes it sound impractical. It really isn't. Oh, and if you really know what you are doing you'll have no trouble towing that boat. Stupid analogies.

Ruby Autocomplete in Vim

Vim and Ruby fit together like beer and pizza, music and dancing: each an art form unto itself, but together they transcend, uh, somersaulting goose bumps. If you aren't already getting to that level with Vim and Ruby you might want to look into the following two Vim plugins: rubycomplete and supertab. Rubycomplete gives you autocomplete for variable names, method names, etc., just like you get in your IDE. Supertab hooks that functionality to the tab key (among other awesomeness), so just like at the command line you can tab complete your thoughts. Wheee!

One side note. To use rubycomplete you need to have a version of vim compiled with ruby support, otherwise you get an error message like this: "E486: Pattern not found: ErrMsg( "Error: Required vim compiled with +ruby"). To get your own version of vim with ruby support compiled in use MacPorts. The precise command you want is 'sudo port install vim +ruby'. On linux you can just apt-get the vim-ruby package and it comes with the plugin by default (but you still have to get supertab yourself). Another technical detail: you need to modify your ~/.vimrc to support all this hotness.

"ruby
autocmd FileType ruby,eruby set omnifunc=rubycomplete#Complete
autocmd FileType ruby,eruby let g:rubycomplete_buffer_loading = 1
autocmd FileType ruby,eruby let g:rubycomplete_rails = 1
autocmd FileType ruby,eruby let g:rubycomplete_classes_in_global = 1
"improve autocomplete menu color
highlight Pmenu ctermbg=238 gui=bold

That last bit with the highlight command is there to fix the default rubycomplete color scheme. Out of the box it has a horrible, horrible pink background to the menu.

As an extra bonus, if you like textmate's cmd-T action you'll want to look into fuzzyfinder_textmate. Another awesome vim plugin. Also, because color is key, look into some good vim colorschemes. I use the vividchalk vim colorscheme on my mac and desert256 on linux (see screenshot). desert256 is my favorite but I can't get 256 color support working on my mac, so i settle for vividchalk.

Running Selenium From CruiseControl with DRB

Doesn't DRb sound like a hip-hop handle from the 80's? Yo! Yo! My favorite MC is DRb! Uh. Actually it is way cooler than that, because it stands for distributed ruby. But I'm getting ahead of myself. DRb is the solution. Let's look at the problem. I'm working on a Java project that has a Selenium suite. The problem solved below was how to best kick off Selenium remotely from Cruise Control. Many options were discussed including a separate cc.rb instance and using Net::SSH. In the end we decided on DRB because we thought it would require the least effort, be cross platformy, and reliable.

Enough talk. Here is what we did: First thing was to write the ruby code that would call from a Linux server over to a Windows server. (I mention the OSes only to highlight the fact that there is nothing special that we have to do to make this work cross platform.)

The following code runs on the linux server that runs Cruise Control. The most interesting line is the one that creates the DRbObject. Basically that line goes out over the network and retrieves whatever object the drb process at druby://seleniumblx:9999 is handing out. Then it calls the run method on that object.

#!/usr/bin/env ruby
require 'drb'

class InvokeFromCruise
def run_rake
remote_runner = DRbObject.new(nil, "druby://seleniumbox:9999")
remote_runner.run
end
end

invoker = InvokeFromCruise.new
invoker.run_rake

Over here on our Windows box running Selenium, the DRb.start_service call starts a ruby process that listens on port 9999. When some other DRb process connects to that port it will simply hand over an instance of RakeRunner. RakeRunner is a dumb class that basically makes a system call to run rake with our smoke task, thus invoking the smoke suite.

require 'drb'

class StartSuite
def start
@rake_runner = RakeRunner.new
DRb.start_service("druby://seleniumbox:9999", @rake_runner)
end
def daemon
DRb.thread.join
end
end

class RakeRunner
def run
`ruby.exe C:/ruby/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb smoke`
end
end

starter = StartSuite.new
starter.start
starter.daemon

Disclaimer: The above code was slightly modified without subsequent tests.

Now to tie this together with our cruise build... First, get the listener up on the selenium box. To do that you invoke the StartSuite class from the directory that contains the selenium suite's rakefile. It will listen on port 9999 until you kill the process. Next, we tell cruise control to invoke the InvokeFromCruise class. Here is a snippet from cruise's config.xml. Notice it is simply calling a invokeSelenium task when the build is successful.

<onsuccess>
<antpublisher buildfile="/path/to/build.xml" target="invokeSelenium" anthome="/path/to/apache-ant">
</onsuccess>

Here is the invokeSelenium ant task that calls the InvokeFromCruise code. The spawn attribute of the exec tag simply tells ant not to wait around while the ruby executable runs.

<target name="invokeSelenium">
<exec executable="ruby" spawn="true">
<arg line="${parameterized-path-to-executable}/build/invoke_from_cruise.rb">
</exec>
</target>

That is pretty much it. The StartSuite class waits around on our windows box and hands out little remote control objects that can kick off a rake task. CruiseControl executes a ruby script that triggers a remote control object after every successful build. The results of the build are simply logged to a file that is in apache's web root so you can check there anytime you need to.

I was happy with how easy it was to do this with DRb. It is lightweight, cross-platform, easy to customize, and more fun than a couple of 80's rappers holding a doll house.

A Mingle in every Agile shop's cupboard?

An ever popular topic of conversation at CHI.RB is 'how I quit my old-crappy-programming-job (OCPJ) and got a shiny-new-Ruby-programming-job (SNPJ)'. In fact just the other night I was listening to such a conversation. As the storyteller went down his list of new job hotness, he rattled off highlights such as: Ruby, Rails, Agile, Mingle, XP... Hey, back up! Mingle!?!

It is nice to come across people who like your* software, and even nicer to hear them drop the name among a group of programmers without anyone saying, "What is Mingle?"

* I use the possesive your here loosely. I'm not on the Mingle team, but feel slightly involved as I too work at ThoughtWorks and occasionally I send them suggestions which they blithely ignore. Thanks guys!

Minchia, Arduino E' Grande.

Arduino is an open source platform composed of a microcontroller, and a development environment. It is a one stop shop for realizing those electronics projects that have been kicking around in the back of your mind for the past few years. You know, the theremin alarm clock that requires you to play the theme to tetris to turn it off...

Evan Farrar's Lightning talk about Arduino inspired me to buy one. And it took me awhile to get around to playing with the thing, but I finally did, and it is fantastic! For my first little project I undertook to reproduce this light mixing project. It was a little too easy, so in order to make it more interesting I decided to implement the state of the 3 lights as a Ripple Carry Adder. And to control it all with a button. It wasn't too hard, although I do have some questions about the Arduino programming language. Their documentation doesn't give any examples of how to use arrays as parameters to functions. They say the language is based on C, but when I try a C style pointer reference I get a funky error. Also when I try to assign one array the value of another array I get errors. My code works around these problems, but if anyone can show me some examples of how to do this correctly in the Arduino language I would appreciate it. Here is my first arduino code:

int pinStates[] = {HIGH,LOW,LOW};
int fullAddResult[2]; //ugh, global since i can't seem to pass arrays around
int halfAddResult[2];
int greenPin = 12;
int redPin = 11;
int bluePin = 10;
int inPin = 2;
int inPinVal = 0;
int prev;
boolean autoPilot = false;

void setup(){
pinMode(redPin, OUTPUT);
pinMode(greenPin, OUTPUT);
pinMode(bluePin, OUTPUT);
pinMode(inPin, INPUT);
prev = digitalRead(inPin);
}

//current, as in electrical current
int currentVal(boolean a){
if (a == true){
return HIGH;
}
else{
return LOW;
}
}

boolean boolVal(int a){
if (a == HIGH){
return true;
}
else{
return false;
}
}

int nand(int a, int b){
return currentVal(!(boolVal(a) && boolVal(b)));
}

int exor(int a, int b){
int ab;
int aab;
int bab;
ab = nand(a, b);
aab = nand(a, ab);
bab = nand(ab, b);
return nand(aab, bab);
}

// method implements a half adder
void halfAdd(int a, int b){
halfAddResult[0] = currentVal(boolVal(a) && boolVal(b));
halfAddResult[1] = exor(a, b);
}

// implements a full adder
void fullAdd(int a, int b, int c){
int ab[2];
int abc[2];
halfAdd(a, b);
ab[0] = halfAddResult[0];//Lame way of copying array values
ab[1] = halfAddResult[1];
halfAdd(ab[1], c);
abc[0] = halfAddResult[0];//Still Lame
abc[1] = halfAddResult[1];
fullAddResult[0] = currentVal(boolVal(ab[0]) || boolVal(abc[0]));
fullAddResult[1] = abc[1];
}

// implements a ripple carry adder
// MSB is the 0th element of the pinStates array
void increment(){
int temp[2];
halfAdd(pinStates[2], HIGH);
temp[0] = halfAddResult[0];
temp[1] = halfAddResult[1];
fullAdd(pinStates[1], temp[0], LOW);
halfAdd(pinStates[0], fullAddResult[0]);
pinStates[0] = halfAddResult[1];
pinStates[1] = fullAddResult[1];
pinStates[2] = temp[1];
}

void loop(){
inPinVal = digitalRead(inPin);
digitalWrite(redPin, pinStates[0]);
digitalWrite(bluePin, pinStates[1]);
digitalWrite(greenPin, pinStates[2]);
if (pinStates[0] == LOW && pinStates[1] == LOW && pinStates[2] == LOW){
autoPilot = true;
}

if (prev != inPinVal){
increment();
prev = inPinVal;
autoPilot = false;
}

if (autoPilot){
increment();
delay(500);
}

}

One note is that xor seems to be a reserved word (although the compiler doesn't tell you that, it just barfs up some meaningless error) so that is why my xor function is named exor. Obviously I didn't bother with Arduino's bitwise functions. My Arduino code leaves something to be desired and so do my pictures, sorry they are blurry. The code could use some cleanup, but when you flash it over to the microcontroller, it works! Toggling the button causes the light mixer to shuffle through all possible color combinations by doing binary addition on a 3 bit number. Each bit of the number represents a different color and so by running from 0 to 7 you can go from no light to white light with all the colors in between. Actually you can't get to the no light state because the state where all the lights are off takes you into "Auto Pilot" mode and it cycles through all the colors with a half second delay between changes. Wheeeeee!

Debating the Vice Debate

I made a transcript of Palin's answers from the Vice Presidential Debate tonight. Here it is:

Question: Blah, blah, blah Economy?
Answer: No Problem, Blah, blah, blah Alaska.

Question: Blah, blah, blah Foreign Policy?
Answer: Blah, blah Governor of an energy-producing state, blah, blah Alaska.

Question: Blah, blah, blah your weaknesses?
Answer: None. Blah, blah, blah executive experience, blah Alaska.

Hey there gal, you aren't running for Vice President of Alaska! What about the other 99.97 percent of Americans? Have you thought about us? I find her point of view irritating. While it is fine for her current job, it seems completely out of whack for somebody trying to become a representative for the entire nation.

Here are other assorted observations:

• Palin showed complete disregard for the moderator's questions. She clearly has a degree from the George W. Bush school of debate. The fundamental principle of that school is "Say what you like, regardless of the question."
• When asked about the causes of global warming she said she didn't want to talk about the causes, but then said the solution was cleaning up the planet and reducing pollution. I guess that means she thinks global warming is caused by humans after all. Just say it.
• She is pointedly informal, and makes self deprecating remarks about her scant experience. The only case she makes for her qualification is basically that she has common sense. If George W. Bush has taught us anything, it is that any old dumbass can be president, but that dumbass presidents suck. Please, no more unqualified applicants for this position.
  

Debating the Debate

For those of you ready to have this election decided it came as a dissapointment that nobody royally screwed themselves tonight on national television. McCain kept his temper. Obama showed some foreign policy muscle. Most people are calling this debate a draw. How boring! The only salvation for the night was the fiesta where I watched the debate. They served up enchiladas and a very posh Obama cake. Mmmm. Hope is delicious!

Here is some interesting analysis I overheard at the fiesta:

• McCain has no new foreign policy ideas. In fact he is quick to attack anyone questioning the foreign policy status quo. Yet these old ideas he is so dogmatic about have pulled us into several wars and conflicts every decade and made our country into a global bully. What is so good about that?
• McCain's campaign uses fear to appeal to voters. Fear Obama because he is too liberal. Fear Obama because he is too different. This is the same nonsense the Bush administration used to stay in power. The same nonsense they used to walk all over the constitution.
• On a positive note, at least both candidates could form a coherent sentence this year. That is more than we can say about the Bush years.
  

Software For Lazy Screencasters

Our ancestors had it rough. When they wanted to change the channel on the TV they had to use their legs to walk over, then their hand to change the channel. Progress gave us the remote control and now I can just use the tip of my finger to change the channel. When they wanted pizza they had to know what to do with flour, yeast, water, cheese, etc. then they had to make it from scratch. Progress gave us the frozen pizza and now the only question is "Is is delivery or is it DiGiorno?" Screencasting used to be something that involved buying and installing software. Fortunately somebody has come along and gotten rid of those two steps. The new lazy way is called Screencast-O-Matic and it is insanely easy. Best of all it is free. Check it out! Wanna see an example of a screencast made this way? Here is one of me writing this blog entry.

ActiveRecord Goodies

There are 3 things in the goody basket tonight:

1. Multiple belongs_to and has_many Associations: Say you have a Game object. It makes sense to have two instances of a Player object associated with it. One player is the winner, one is the loser. Do you know how to model that relationship in active record? Here is a fantastic blog post that gives a nice explanation on how to do it. If you want another example take a look at my code which does the same thing.
   
2. Logging the sql ActiveRecord generates: In rails it is obvious how to make this happen. If you are using ActiveRecord standalone you can do this:
   

   ActiveRecord::Base.logger = Logger.new(STDOUT)
   

3. Deciphering how ActiveRecord pluralizes: Sometimes I have trouble figuring out what ActiveRecord will compute the plural of a noun to be. This application shows you what ActiveRecord is thinking.
   

John McCain Hates Rails

Old man McCain hates Rails! I've got the quote right here:

"Weyrich knows that Senator McCain, throughout his career, has been very anti-rails"

Here is the source of that quotation. They seem to have misspelled Weirich's Name, but let's not dwell on details. What this means is that come November the Ruby community must rise up and protect our beloved web framework against McCain and the forces of evil! DHH and Obama are pretty tight. They both got their starts as community organizers in Chicago, which means it is a safe bet that a vote for Obama is a Vote for Rails! McCain's animosity towards Ruby probably stems from frustration; McCain's own language, 'straighttalk', a pidgin dialect of smalltalk, has turned out to be vaporware.

Feet Hurting? Try Shoes.

From Cuberick

The path to a working GUI application is often strewn with broken Chihuly sculptures and jagged discarded fish tank coral. This is where shoes, _why's tiny windowing toolkit, comes in. Last September I gave a presentation on shoes to the chicago ruby user group. Since then it has come a long way! Here are some of the awesome new features that have come along.

• Applications automagically resolve dependencies via gems.
• Shoes applications can be packaged up as .exe or .dmg for creating distributable binaries.
• The manual is coming along nicely. Three cheers for documentation!
• Custom first class widgets can be created in ruby code.
  

The most important thing about shoes is how damn easy it is. You can write a useful application in just a few lines of easy to understand code! For proof of that check this out:

#Source code borrowed from this useful article here
Shoes.app do
button( 'play' ){ @v.play }
button( 'pause' ){ @v.pause }
button( 'stop' ){ @v.stop }
@v = video "superbad.divx", :width => "100%"
end

Here is a screenshot of that in action:

See? No Chihuly sculptures in there, and no jagged discarded fish tank coral. I recently knocked together a simple shoes app for score keeping score at a local video game tournament. It tries for a funky retro look. Put on your safari helmet and head over here to see it. Real live shoes code in the wild!

Presidential Dungeons and Dragons

In internet terms the Presidential election is just a year long flamewar. It ends in November when we all go into little curtained stalls and choose the troll we like best. The troll who gets the most votes is made President and gets to be the most powerfull troll in the world for 4 years.

Following the election news closely is like reading YouTube comments. If you haven't been paying attention, here is a blow by blow I whipped up, YouTube style:


Wouldn't it be great if, instead of this vacuous circus, we had something substantial to base our decisions on? Let's get a good dungeon master and let the candidates role play the presidency on television. It would be like the debates, but useful. Seeing a candidate face problems and make decisions would give tremendous insight into how they would serve as president. C'mon, admit it, we all want to see McCain and Obama's character sheets and watching those guys stay up all night, rolling a 12d, launching cruise missles and vetoing bills? Priceless.

Movers and Shakers, Dreamers of Dreams

Here is a factoid for all you smarty pants. If you've ever used the term "movers and shakers" than you have quoted Arthur O'Shaughnessy's Ode from his book Music and Moonlight (1874). Another famous reference to the poem comes from the screenplay Charlie and the Chocolate Factory when Willy Wonka Sets Veruca Salt straight about the Snozberries. I never knew Wonka was quoting!

We are the music makers,
And we are the dreamers of dreams,
Wandering by lone sea-breakers,
And sitting by desolate streams;—
World-losers and world-forsakers, 5
On whom the pale moon gleams:
Yet we are the movers and shakers
Of the world for ever, it seems.

Here is the poem in its entirety.

Chipmunk Physics Presentation at CHIRB

Tonight I'm giving a lightning talk about the Chipmunk 2-D rigid body physics library at the Chicago Ruby User Group meeting. I plan on giving demonstrations of some simple applications I have created. One uses the accelerometer built into my mac to control a platter full of blocks on the screen. Wheee! The other is a very basic like-show-me-the-code-dude type example that people should be able to easily extend and play with.

Here are some useful research type links for tonight's participants:

• Chipmunk code
• Mac Tilt Sample Code
• Chipmunk Basics Sample Code
• Other Code
  
• Chipmunk Forums
  
• C documentation
• Gosu/Chipmunk Tutorial

Now for a quick editorial on the state of the CHIRB: Our monthly meeting is great, but far too work-centric. I enjoy programming beyond the context of web application frameworks, or resume building. That is why my 5 minute talk will be nothing but good, clean ruby fun. I encourage anyone who does clean non-work programming to step up with a lightning talk tonight!

Word of the day: Turtley

Turtley

Self referential.

Turtley derives from the phrase "Turtles all the way down", a cosmological reference to an infinite regression belief about the nature of the universe.

Fibonacci sequences are turtley.

On my previous project, which was pro bono work, for this fantastic company. The word "turtley" came out of a discussion about the "Turtles all the way down" phrase. It turned out to be both fun and useful and I highly recommend referring to things that are turtles all the way down as being turtley. There are so many turtles in software! Examples: A Scheme interpreter written in Scheme, The debian linux installer which boots a linux kernel to install linux, the smalltalk language is extremely turtley, many interesting data structures are turtley, even your filesystem.

xkcd and the dawn of Meme Fundraising

Sean Tevis' run for state representative is funded by paypal donations rewarding his clever appropriation of the xkcd webcomic. This makes the battle for Kansas' 15th congressional district the geekiest campaign for public office evar! His website gives his position on some important issues such as immigration, school funding, taxes, blah, blah, blah. Sean! What about the _important_ issues? I have so many questions! Where are you on the question of VI vs. Emacs? Can't we just write a shell script and a couple cron tasks to replace the government? In how many programming languages can you write "Hello, world!"? When I sign up as a volunteer for your campaign, do the instructions come in man page form? Inquiring minds want to know!

In the past couple days Sean has recieved over 5,112 donations! That is serious money for a state congressional race. Campaigns all over the world, pay attention! Meme fundraising is the way of the future. Not sure where to begin? A homage to xkcd is probably too advanced for most candidates to pull off. I'd recommend you start out with something easy, say...

The lolcat campaign

Some Feed Reader Gadgets Broken

Update 7/2/08 - My OEDILF Widget works again... so the problem must be fixed. Yay!
Last week I noticed that the Google gadget I had created to display limericks from the OEDILF Limerictionary had stopped working. Bummer! Yesterday I finally got around to investigating. Turns out, nothing is wrong with my gadget, rather Google broke some stuff in their API. As of recently if you use the IG_FetchFeedAsJSON() function, you always get an empty response. This is broken in the "sandbox". Supprisingly blogger is in the sandbox, because any gadgets implementing this call are broken on blogger. My gadget still works correctly when viewed from iGoogle and other such non-sandboxy places. I'm not the only one this is affecting, many of the RSS based gadgets I tried are broken including BBC news, NY Times, Wall St. Journal. According to Google this should all be fixed sometime next week. In the meantime I'll have to get my limerick fix some other way.

Do it Yourself Ubuntu Installer ISO's

This picture has a lot to say, but it says nothing whatsoever about Ubuntu installer ISO's. Who cares. This picture is so awesome I am considering using it to illustrate every blog entry I do from now on. "On the latest humanitarian crises" illustration: Ike's middle finger. "On alternative energy" illustration: Ike flying the bird and Tina. "Google's newest web framework" illustration: Ike's studded leather belt and his middle finger. Tell you what. In the spirit of the illustration i'll try to keep this post as bad ass as possible.

Listen up: A while back I blogged about debian-cd and creating a Debian net installer ISO. Imagine creating an entire operating system from scratch! Ok, not quite from scratch since you just download binaries, but hey, when you bake bread you don't mill your own flour do you? I didn't think so. So shut up. That previous debian-cd trip was just one stop along the road to creating a Xubuntu installer ISO from scratch and that is what this post will be about.

The most important thing to know about creating a Xubuntu installer is that there is a guy named Colin Watson who makes it all happen, ok so there is a whole army of people, but Colin's name is on everything. He is like the Ike Turner of the debian-installer set. Here is a link to the back room of his fun house where all the image making magic happens. He has bazaar branches of the following projects: cdimage, debian-cd, germinate, britney. You should check out branches of all those things with bzr at the link I gave you. The next thing to do is put them together. cdimage is the set of scripts that drives the whole thing, it handles synchronization of your local repository, generating the debian-cd task lists from seeds, calling debian-cd to burn the ISO's, publishing the ISO's and some other good stuff. So to "put them together" just drop debian-cd, britney and germinate into the cdimage directory.

Unless you think you are some kinda hotshot it would do you good to read the README that comes with each of these four tools, especially the cdimage one. If you don't, you better not come crying to anyone on the mailing list, because they will rip your arms off and beat you with them. It happens. I wish I could try to give exact directions for making this work, but there are too many steps and I forget things, so my recommendation is to immediately start trying to run cdimage, and resolve all the problems and complaints it has one at a time. Mostly you will just be figuring out what environment variables need to be set. (There are about 6 or 8 I set. I created a script with an export for each one and I source that before I run cdimage.) Here is the command for running cdimage: ./for_project xubuntu ./cron.daily

Great! You scrolled this far down the page so you must still have arms, or you are working your mouse with your mouth, either way, good job. Once you get through the first few errors (for not having the right variables defined) you will notice that the command is going to take forever to run. This is because it tries to rsync your local mirror with a remote mirror (Mine is about 100 GB for all the i386 stuff). Since you probably don't have a local mirror you probably can't skip this step, just rest assured that once you have it, you can re-use it and things will happen much faster, say 5 minutes or less. If you already have a local mirror you can just point the script to use that one . (I just put a symlink in the cdimage directory called ftp and it points to my local mirror at /opt/mirror.) Try not to get frustrated if you get stuck. There are lots of moving parts in here! LOTS! You are probably just missing some environment variable or you have one set to point at the wrong directory. The best way to work through things it is to read the source. Most error messages are pretty descriptive, but RESIST the temptation to hack the code. Colin's stuff really works. I got it all working with only one minor change to build_all.sh in debian-cd. For posterity, here is a list of the changes, tips and tricks that might help:

• when debian-cd kicks off via the build_all.sh script it always complains that it can't find CONF.sh even though it is clearly in the right place. I change the ". CONF.sh" line at the beginning of the file to ". `pwd`/CONF.sh" and that fixes it.
• If you already have a local mirror you can skip all the rsync business. To do this edit anonftpsynch in the cdimage bin directory and comment out the two rsync lines. I also comment out the lines that synch the indicies and overrides.
• Britney is partially C code and you have to build it. Make sure you have the python-dev-all (I think that is the name) package as well as the one listed in the README. There is a step in there with some perl that I didn't do, but it didn't seem to matter.
• Getting a local copy of the seeds really speeds up your build time a lot, so do it! You can set the LOCAL_SEEDS environment variable to the path of your seeds. If you make changes to the seeds be sure to commit them or germinate ignores your changes.
• You can disable the creation of the Jigdo (stands for jigsaw domino, which is awesome) files if you don't care about them and save a bit of time and disk. Just edit debian-cd's CONF.sh and set the JIGDOsomething-something environment variable to 0.
• Obviously you don't need 100GB mirror to create a 700 MB CD image. There are techniques for creating small local mirrors. None of them very handy. I'm using extensive --exclude parameters in my rsync command to make sure I only get i386 binaries. Check out this wiki page for info on other techniques.
• It is imperative to use debian-cd and germinate from colin's repository. You can't mix and match the packaged versions with cdimage because colin has modified things quite a bit, so stick to his sources in bazaar for everything.
• Fri. June 13th Edit: Be sure you have your gpg keys in order and cdimage is properly configured to use them or your APT repo on the CD won't be trusted, so nothing will get installed, which is kind of a bummer.
• Make sure your GPG key you are using doesn't have a password, because GPG is running in batch mode when the CD is signed and thus won't be able to prompt you for a password.
  

Whatever you do don't get discouraged. I recommend asking questions on the ubuntu-installer IRC channel and mailing lists. People don't go out of their way to help you, mostly they ignore you, but sometimes you get lucky. Also, take time to pull back and understand the big picture about how the installer works and what all the files going onto the installer are for. That sort of context can help you avoid mistakes and direct your research. Good luck. And remember what Ike Turner said: "In China, when you get to the airport everyone be talking in American slang."

C# in Depth

In the classic Steve Martin film, The Jerk, the main character, Navin, gets all worked up when his name shows up in the phone book:

Navin R. Johnson: The new phone book's here! The new phone book's here!
Harry Hartounian: Boy, I wish I could get that excited about nothing.
Navin R. Johnson: Nothing? Are you kidding? Page 73 - Johnson, Navin R.! I'm somebody now! Millions of people look at this book everyday! This is the kind of spontaneous publicity - your name in print - that makes people. I'm in print! Things are going to start happening to me now.

I'm not in the phone book, but my name is now in print!

Here is a picture of the back of C# in Depth, Jon Skeet's great new book (GO! Buy it!).

Do you see that? My name AND a quote. Things are going to start happening to me now.