Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Wednesday, June 17, 2009

Meta Programming with a Barf Bag


Day one of the Open Source Bridge conference was action packed. I'll probably have to write a few posts, but I'm going to start things out with notes on my favorite talk of the day "Spindle, Mutilate and Metaprogram: How far can you push it before there be dragons?" by Markus Roberts and Matt Youell. Yes the picture is them wearing colanders on their heads.

This talk was 2 hours long so it is hard to cover it in all its glory. The crux is that programmers are generally very inhibited by the orthodoxy of each particular language community to the detriment of the code they are writing. Done properly, applying the magic of meta programming simplifies problems and makes code more understandable. Markus especially seemed to wipe his boots on some of the taboos. They mentioned ActiveRecord and jQuery as examples of effective meta magic.

The talk began with a gameshow where 3 volunteers were shown code snippets and the first person who could identify what language it was got a point. The first program looked like C but was in fact Ruby. Another looked like assembly language but was Perl, they even threw some Cobol in there, but it turns out it really was Cobol. :) Each time after they revealed what the real programming language was they showed the code that allowed them to mimic the other languages. It was really a cool way to start a presentation about meta programming.

At one point they showed some C that had been made to look like another language (I don't remember which). Somebody from the audience pointed out a memory leak. Markus responded by pointing out a table in the back where they had barf bags laid out and recommended he get one because there was going to be some spectacularly intolerable displays of metaprogramming. There was lots to learn about how you can shape your own destiny in a programming language if you are willing to amp up your level of meta skulduggery. Tomorrow when I see Markus and Matt again I'll ask about their slide deck, because it would be fun to share some of their code examples.

Saturday, April 4, 2009

Pycon 2009 videos of interest


Here are a few presentations I got alot out of at Pycon. This one by Ian Bicking, Topics of Interest, was good for me because I'm new to the Python community and It really covers alot of ground and gives you one person's perspective on a bit of python history. I also like the parts where Ian dives into some non technical topics of interest like women in computing (17:00) and postmodern programming(21:50). Plus the IRC comments at the bottom are at times hilarious.
http://pycon.blip.tv/file/1932161/

The second presentation by Jack Diederich, Class Decorators Radically Simple, is one that I found useful as in my one Python project I did I never used class decorators and it was good to get some ideas about how and when to use those. Check it out.
http://pycon.blip.tv/file/1949345/

Sunday, March 29, 2009

Pycon 2009


PyCon is wrapping up and I just thought I'd mention a few things of interest that I saw.

Adam Christian gave a presentation on windmill which is a functional test tool like selenium, written in python. It looks pretty good, but I didn't really see how it differentiated itself from selenium.

Adrian Holovaty gave a talk called "Behind the scenes of EveryBlock.com" If you don't know about everyblock you should check it out. His talk was mostly about technical challenges they faced and how they addressed them. I won't go into detail, but want to say this is a very cool project and I think it offers a glimpse into the kind of useful things that would be possible if government could find a way to share their public records online.

Michael Foord gave a great lightning talk called "Metaclasses in 5 minutes" Plus he has an awesome "Knuth is my homeboy" shirt (see picture). Be like mike and get one here.

Guido Van Rossum gave a keynote. Two things about that: First, he is trying to "fade away" from being in charge of the python community. Second, python is very similar to lisp. If someone could explain to me what he meant by python and lisp being close, I would be grateful.

Joe Gregorio gave a talk on "The (lack of) design patterns in python". It was good and really applies to most any dynamic language. Read the slide deck to see that popular design pattern names are less likely to be talked about on the python mailing list than say, "the pope" or "sausage". Also some great quotes in there about why dynamic languages with metaprogramming basically have all these common patterns built in.

Jack Diederich Gave a very detailed talk on class decorators in Python, and how they make your code more readable than metaprogramming. Good info about this python language feature can be found here.

Finally, Gregor Lingl a high school teacher from vienna who wrote the turtle module from python 2.6 and 3.x spoke about "Seven ways to use Python's new turtle module" He showed off this beautiful demonstration that animated the path of the earth and moon going around the sun. Check out slides 15 and 16 of his talk. Also see slide 28 for a screenshot of his nice animation of building a penrose kite. He wrote a book on this "Python für Kids" Too bad it is in German. Logo rules!

Friday, February 6, 2009

Bash History In The Cloud

Since I began programming I have worked on at least a hundred different computers: university workstations, my own computers, dozens of employer and client workstations. Each one had a history file loaded with commands that I begged, borrowed, stole, sweated and cried for. Countless hours of work now long gone or rotting somewhere in a stack of backups.

Everyone, please. A moment of silence for all the shell commands that have gone into the bit bucket never to be heard from again. Even the mean ones that hosed my system. Especially those.

Ladies and Gentlemen I come before you now to swear that never again will I lose this information. Now with shell-sink I keep all that lovely data handy no matter where I am. On top of that I can organize and group useful sets of commands and have them at my fingertips when I need them. I dedicate the Shell-sink tool to the memory of all those late night sed and awk one liners that have saved millions of lives, but faded from memory all too quickly.

Features include:

  • Permanent and unlimited storage of bash history
  • Pull commands back out of the sink and into your shell based on a tag or keyword
  • Searchable bash history
  • Tagable bash history
  • Annotateable bash history
  • Aggregate bash history from multiple computers into one grand unified history
  • RSS feed of your bash history that you can filter
Shell-sink stores your bash history in Google's big table. Development is ongoing. And I have an ever growing feature list including command statistics, and powerful ways to share and document groups of commands.

A guide to get started can be found here. Debian/Ubuntu packages exist to make installation and updates a snap. But installation is pretty easy on other unix based systems. So far nobody has used the sink but me, so if you try it out please drop me a line with some feedback. There is a google group that would be a perfect place to leave feedback.

Saturday, January 10, 2009

Programming for Kids

As a kid I can remember going to the library to check out books about programming in BASIC. The books were geared towards young people and had plenty of games that you could tpye into your computer. Now days things aren't as easy for burgeoning young programmers. Even though computers are more prevelant than ever, the tools and knowledge for connecting kids with programming are still scarce. To get started programming you need that initial leg up. In the old days a book about BASIC was a leg up.

These days a search for "programming for kids" on Amazon wouldn't return any interesting results, until now. A new book called "Hello World! Computer Programming for Kids and Other Beginners" has just come out. It is a book in the spirit of those old BASIC programming books. It covers the fundamentals of programming using the Python language and has game programs that you can get started with including a lunar lander game and a ski-free clone. If you want to learn programming or teach it to a kid, this is your book. Enjoy!

Other resources:
  • http://scratch.mit.edu/
  • http://hacketyhack.net/
  • http://el.media.mit.edu/Logo-foundation/logo/index.html

Thursday, January 1, 2009

Creating a Launchpad PPA For a Simple Python Project


For some background information on Launchpad and PPA's see my previous post. For more information about information given here check out the list of links at the bottom of this post. Now on to the bullet points:

Getting Launchpad Ready
  1. Create a new Launchpad project.
  2. Have imported your PGP key to your Launchpad account (apt uses these to verify the authenticity and integrity of packages it installs).
  3. become an Ubuntero (i.e. you must sign the Ubuntu Community Code of Conduct)
  4. activate your PPA on your profile page or your team's overview page.
  5. Build a source package for your project.
Building A Source Package For Your Python Project
  1. I followed these instructions but since we are trying to build a source package, not a binary package I made the following changes.
  2. Added a buildsrc target to the Makefile. See my makefile for specifics.
  3. Changed rules file to have a full path to the destination directory. See my rules file. I did this because Launchpad's process will be calling the rules file directly (probably via debuild) as opposed to using ghantoos's makefile which sets up the DESTDIR environment variable.
Upload Your Source Package
  1. install dput - sudo apt-get install dput
  2. edit ~/.dput.cf so it looks like
    [my-ppa]
    fqdn = ppa.launchpad.net
    method = ftp
    incoming = ~your-launchpad-id/ubuntu/
    login = anonymous
    allow_unsigned_uploads = 0
  3. Run dput my-ppa P_V_source.changes Replace P with the package name and V with the version number.
  4. Now you have to wait 5 or 10 minutes. Launchpad is spinning up a whole virtual server that will grab all the dependencies you listed using pbuilder or some such and will do a clean build of your package. An email will get sent out when your build kicks off.
  5. If you go look at your PPA your build will have a little spinny icon next to it. Here is my PPA site for example.
  6. If something goes wrong you will want to read the build logs to diagnose the problem. Click on one of the various binary builds to get to the link. Here are some sample ones from my build.
That is pretty much it. It is alot of stuff, but I hope my code serves as an example for you to get your own PPA setup. Questions are probably best answered on IRC. #launchpad on irc.freenode.net is loaded with knowledgeable types. Or you can drop your questions in the comments of this post if you have em. I'm no expert, but I might be able to help.

Those links I promised:
  • https://wiki.ubuntu.com/PackagingGuide/Complete
  • http://ghantoos.org/2008/10/19/creating-a-deb-package-from-a-python-setuppy/
  • https://help.launchpad.net/Packaging/PPA

Distribute Your Software Just Like Ubuntu With Launchpad


Launchpad is project hosting provided by Ubuntu. It is free for projects with an open source license. The service consists of distributed version control, bug tracking, team collaboration and other features too numerous to list. The most unique feature and the one under consideration here is the PPA, "Personal Package Archive". This feature allows you to publish your software to users using the standard ubuntu software installation and update tools (apt/synaptic). Imagine finishing a new release of your software, and at the click of a button Launchpad builds your software for multiple architectures, and notifies your users that updates are available. Brilliant! Before PPA's, maintaining a standalone package repository was out of the realm of what mere mortals could accomplish. Now launchpad has removed nearly all the barriers to entry. In my opinion this makes Launchpad the best project hosting out there.

It isn't all beer and skittles though. There is much to know about building packages for ubuntu. I had to spend the better part of a day learning about how debian packages work in order to get my PPA up and running. (It was worth the effort). And launchpad does have its warts. The user interface isn't very nice when compared to git-hub or lighthouse, and with the large number of features packed in there, it takes too long to figure out how to do something for the first time. Also, Launchpad can seem sluggish, especially the bug tracker, which can be infuriating if you are updating lots of bugs.

To sum it all up Launchpad isn't perfect, but it is the only solution for all-in-one project hosting: distributed version control, issue tracking, collaboration and software distribution. In my mind git-hub, lighthouse and google code are the only real contenders for the title. Git-hub has a great UI but doesn't offer issue tracking. Lighthouse has another great UI, but doesn't have version control. Google code is wonderfully simple, but doesn't offer distributed version control. If only I could combine the best parts of Git-hub, Launchpad, Lighthouse and Google code...

Coming up... Based on my own experiences of creating a PPA on launchpad for a project of my own I am going to write a walkthough of setting up your own PPA. However, I am going to make it a separate entry. If you are interested just read my next post.

Monday, December 8, 2008

Chicago Python User Group Goes Ninja


Here is the official announcement for the Chicago Python User Group (chipy). The important thing to notice here is that I'll be presenting (or maybe the important thing is the free pizza.)

Chicago Python User Group
=========================

Calling all Chicago Python Ninja's, this will be our best meeting yet--
this Thursday. (if not, there will be some pretty darn good pizza in
here)

Insert funny comic here:

We have some interesting dragon slaying talks lined up this month.
Starting off, Garrett will mix theory with practice in another one of
his highly thought provoking talks. Professor DiPierro will go all
out with the tools that serve as his student's weapons in the real
world. Josh will brave the Python waters as n00b who comes from the
dark side and will probably tell us a interesting Ninja story while
displaying Pythonic charm and charism--all while balancing on one
leg ;) Do not try this at home. In fact, staying home will not be as
much fun or as educational, folks.

This *will* be our best meeting yet.

Topics
------

* Bazaar within Subversion, Personal Workflow - Garrett Smith
* T2 (web2py tier two) and T3 - Massimo DiPierro
* Beginning Python with AppEngine - Josh Cronemeyer

NOTE: Due to the high level of interest in various talks, I have
limited talk length to 25-30 min. Offenders will be severely punished
by my personal Ninja friend.

When
----

Thursday, December 11th, 6:30pm Pizza
Thursday, December 11th, ~ 7pm Meeting

Location
--------

ThoughtWorks, 200 E. Randolph 25th floor.

Prepare to be perfectly astounded by my one legged python slinging, pizza eating acrobatics. I'm going to announce the deployment of the best web application ever at the best python meeting ever. It will be cosmic. A total eclipse of dynamic languages.

Saturday, November 29, 2008

Syntax Highlighting In Blogger


I have pulled various tricks in the past to get syntax highlighting in my code samples. In the past I only had highlighting for ruby code. I think I finally have a good solution and it works for any C style language.

Luka Marinko's blog has simple instructions to get this to work. Thanks Luka! Now here is some nonsense code to show what kind of highlighting you get with Java, Python, and Ruby. All the useless imports and requires are just so we can see those kinds of statements getting highlighted.
JAVA

import java.io.*;

class HelloWorld{
public static void main(String[] args) {
hello();
}

public static void hello(){
System.out.println("hello world");
}

}

RUBY

require 'rubygems'
class HelloWorld
def initialize
puts "hello world"
end
end

HelloWorld.new

PYTHON

import sys
class HelloWorld:
def __init__(self):
print "hello world"

HelloWorld()

Unit Test Your Google App Engine Models


I've been working on a project using Google App Engine (GAE) called "les Freres Jacques" That manipulates images of people's face onto the cover of this old french LP. Look for it to be out in a couple months. Anyway, My favorite way to start any project is by doing TDD. Unfortunately I'm new to python and GAE simultaneously so I had to do plenty of research to figure out how to unit test a GAE app. Most importantly for me was the ability to test my models that are based on google's datastore api. What follows is some information to get you started writing unit tests against a GAE model. First, a list of the tools you need to install.
  • Nose is a tool for running your python unit tests.
  • NoseGAE is a plugin for nose that bootstraps the GAE environment.
The easiest way to install these is with python's easy_install, which as far as I can tell is similar to ruby's 'gem' program and perl's 'cpan' program... though I don't know if it resolves dependencies automatically. Anyway, on OSX easy_install is installed by default so you can simply type
sudo easy_install nose
sudo easy_install nosegae
Now let's create a test to exercize a simple GAE model object. Here is a file called test_simple_model.py
import unittest
from google.appengine.api.users import User
from test_example.simple_model import SimpleModel

class TestSimpleModel(unittest.TestCase):
def test_creation(self):
user = User(email = "test@foo.com")
model = SimpleModel(goo_user = user)
model.put()
fetched_model = SimpleModel.all().filter('goo_user =', user).fetch(1)[0]
self.assertEquals(fetched_model.goo_user, user)
The nose tool we installed earlier gives us a program called nosetests to run. When you call it it looks through your project and runs all your tests. We should call it now with the google app engine switch.
nosetests --with-gae
Wheee!! It is a lovely failing test.

======================================================================
ERROR: Failure: ImportError (No module named simple_model)
----------------------------------------------------------------------
Traceback (most recent call last):
...
from test_example.simple_model import SimpleModel
ImportError: No module named simple_model

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)
Good. Now we need to write some code to get our test passing. Here is what I wrote in a file called simple_model.py.
from google.appengine.ext import db
class SimpleModel(db.Model):
goo_user = db.UserProperty()
Now when I run
nosetests --with-gae
I get the lovely
.
----------------------------------------------------------------------
Ran 1 test in 0.008s

OK
and I am happy because I see how I can do TDD with GAE! Here is a list of references I used to figure this stuff out. Hope you find them useful. You can find the full source of this example here.
Update: 11/30/08
The datastore persists between tests which isn't usually what I want to happen. I submitted an issue on the nose-gae issue tracker. In the meantime here is a workaround to make sure the datastore is flushed between runs. Add this method to your test class and call it in your setUp method.

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub

def clear_datastore(self):
# Use a fresh stub datastore.
apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
stub = datastore_file_stub.DatastoreFileStub('appid', '/dev/null', '/dev/null')
apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)
Update: 1/29/09
Reading Dom's well researched and documented post on testing App Engine applications. I thought I better spruce up my own post by adding a citation. The code from the clear_datastore method above comes from this message posted on the google app engine google groups mailing list.
Update: 5/17/09
There is currently an issue with nosegae. See defect 18. There are patches that fix the issue posted there. I downloaded the source code, removed the subversion directories, patched the code and ran
easy_install .
in the root directory of the code. That fixed the issue for me.
Update: 5/17/09
As of GAE SDK 1.2.1 the appid of your datastore stub must match your appID. Make sure your call to DatastoreFileStub uses your actual app id.

Friday, November 21, 2008

ChiPy Meeting at ThoughtWorks

The December Chicago Python User Group will be hosted at ThoughtWorks! This will be the best meeting of any kind, ever. Be there! Meeting and RSVP details to follow. Oh, and if the flyer doesn't make sense to you, I usually help out with the Chirb meetings at our office and I couldn't help doing a sort of mashup with a reference to _why's foxes. See! Ruby and Python can co-exist at ThoughtWorks. Hope you like the flyer. I made it with inkscape.

Fine Print:
My loving imitation of _why's foxes appear courtesy of the CC license of the original work.
Creative Commons License
Therefore this work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.

Sunday, May 13, 2007

Migrating from Wordpress to Blogger


A general trend in my techno-lifestyle has been the move from do-it-myself, to FREE services. Take my email for example. I used to refuse to use Hotmail or Yahoo! because I didn't trust my communications to some company, and I wanted to use Imap. Along came gmail, and suddenly I could get tons of space, and an awesome client for free. Bye-bye exim and bye-bye Thunderbird. What about storing my photos online? No way was I going to put all my personal photos on some company's servers. Then Flickr began offering Unlimited storage at an amazingly low price, and their software made sharing a joy. Bye-bye jigl and my ever increasing storage woes. I guess what is driving my adoption of these services is the maturity they have achieved. Ajax makes the interfaces useable and with prices hovering just above free nothing else makes any sense.

So I recently realized that I was really not using my virtual server anymore. Google gives you free subversion repositories, free email, and free blogs. So the time had come to kill my server and the only thing standing in my way was migrating my blog from wordpress to blogger. It turns out this is not a straightforward thing to do.

Step 1: see if anyone else had done it. Of course others have done it, but there is no automatic solution (that works). There are dozens of how to documents on moving from blogger to wordpress, but I found only 1 useful for doing the opposite. It also came with a pretty well made Java tool. Here is a link to the tool. Too bad it errored out when I tried migrating my blog. It wasn't open source either so I couldn't try to fix it. I think it is designed to move a wordpress.com blog, not a standalone hosted instance of wordpress. In any case after fiddling with it for half an hour I thought I had better start rolling my own solution.

Step 2: I used mysqldump to grab my WP database from my server and then imported it into a local mysql database.

Step 3: I downloaded the python version of Google Blogger API and used it to create a few dummy posts and comments.

Step 4: I wrote a python script and migrated my entire blog comments and all in about 20 seconds.

Lessons learned:
  • The Blogger API wasn't intended for migrations. You can't set the author of the comments programmatically. They belong to the person you login as. You can't backdate comments, so they appear anachronistic on the site. This is too bad, because there were some interesting folks who had left me comments. I would have liked to preserve that history.
  • Python is a nice language, but I hate, hate, hate the whitespace sensitive nature of it. I also find it a bit less elegant than Ruby. Google, you need to stop pretending that Python is so great and get your API out there in Ruby.
  • I'm never going to use this code again, thus it ain't very reuseable. My unit tests depend on my specific wordpress database. Stuff is just hardcoded in. I totally ripped off some of the sample code that came with the api download. For what it is worth I threw it up on my google code site. Feel free to check it out. It would be easy to make it better, but I don't want to mess with it.