Testing image generation with Cucumber and Sikuli

Image recognition can be a useful weapon in your testing arsenal. Our current project is a rails app that generates advertisements. We use sikuli's image recognition to detect that the text and images that we expected to be in the advertisement were actually in the ad. To integrate sikuli with our build we simply created a cucumber step that calls sikuli. Let's cut to the chase and see a sample test.

Scenario: Dealer places a Campaign Ad
 Given "Bill Murray"'s dealership "Ghostbusters" exists in the system
 When I log in as "Bill Murray"
 And I book a "Year End" campaign ad
 Then the ad proof will look like "ghostbusters_tag.png"

Cool. The magic sikuli step is the one that looks at the ad proof. Here is the step implementation.

Then /^the ad proof will look like "([^\"]*)"$/ do |filename|
 feature_dir = File.expand_path(File.dirname(__FILE__) + '/../../features')
 sikuli_script_dir = "#{feature_dir}/sikuli"
 verification_image = "#{feature_dir}/images/#{filename}"
 result = `/Applications/Sikuli-IDE.app/Contents/MacOS/JavaApplicationStub #{sikuli_script_dir}/verify_existance.skl '#{verification_image}'`
 result.should match(/hallelujah/)
end

Pretty straightforward. Call sikuli and pass it a sikuli script and the image. The sikuli script returns the word 'hallelujah' if it looks at your app and determines that the verification image exists anywhere on the screen. Ok. Lets drill down into what the sikuli script looks like. Get your Python goggles on...

import sys
image = sys.argv[1]
switchApp("Firefox.app")
type(Key.DOWN, KEY_CMD)
if exists(Pattern(image).similar(0.91)):
 print "hallelujah"
else:
 print "bugger"

This one is brief, but I'll explain a bit. Sikuli is a cross platform tool for scripting your computer as well as interacting via image recognition. In this script we make sure that firefox is the app in focus. The image we care about is at the bottom of the page so command down tells firefox to go to the bottom of the page. Finally it looks for the image match. The 0.91 number is basically telling sikuli to be damn sure. We print hallelujah if there is a match.

Ok. That is what I am doing, but in practice how does it work? Well, it has actually been pretty trouble free, and it even finds bugs and problems. Amazing! However it isn't perfect. Here is my long list of things that could be improved:

• Sikuli's documentation on how to run sikuli from the command line is out of date.


• You have to compile the python down to an skl file if you want to run from the command line. Annoying.


• Use sikuli IDE's screenshot mechanism. Other screenshot tools cause trouble.


• Set the similarity number high to prevent false positives. For example at 0.85 similarity, sikuli couldn't tell the difference between the word Ghostbusters and Ohostbusters.


• Creating the screenshots for tests is tedious


• My sikuli script is naive, which means if you have multiple firefox windows open it could fail erroneously


• A popup warning or some always on top window could cause the test to fail erroneously


• If the browser scales the image your test will fail erroneously

This has been indespensible for our application, because it lets us know if we have broken the integration with InDesign. It also gives us a way to test that our InDesign server is behaving. Image recognition is a bit of an edge case when it comes to testing web applications, but it is an interesting tool for testing and one that is highly adaptable. For images generation applications including charts or custom javascript controls I think image recognition would be the only way to test. It would be super cool to have an image recognition gem because then we would be able to do these kinds of tests without calling out to sikuli.

Just for fun I included this last screenshot. It is sikuli IDE's nifty little image recognition troubleshooting tool. Here I have turned down the similarity number to 0.5 and sikuli is showing 3 matches for my test image of the cucumber on the page. The bright red match is the strongest and the pink and purple are weaker.