Sunday, June 15, 2008

Some Feed Reader Gadgets Broken

Update 7/2/08 - My OEDILF Widget works again... so the problem must be fixed. Yay!
Last week I noticed that the Google gadget I had created to display limericks from the OEDILF Limerictionary had stopped working. Bummer! Yesterday I finally got around to investigating. Turns out, nothing is wrong with my gadget, rather Google broke some stuff in their API. As of recently if you use the IG_FetchFeedAsJSON() function, you always get an empty response. This is broken in the "sandbox". Supprisingly blogger is in the sandbox, because any gadgets implementing this call are broken on blogger. My gadget still works correctly when viewed from iGoogle and other such non-sandboxy places. I'm not the only one this is affecting, many of the RSS based gadgets I tried are broken including BBC news, NY Times, Wall St. Journal. According to Google this should all be fixed sometime next week. In the meantime I'll have to get my limerick fix some other way.

Wednesday, June 11, 2008

Do it Yourself Ubuntu Installer ISO's

This picture has a lot to say, but it says nothing whatsoever about Ubuntu installer ISO's. Who cares. This picture is so awesome I am considering using it to illustrate every blog entry I do from now on. "On the latest humanitarian crises" illustration: Ike's middle finger. "On alternative energy" illustration: Ike flying the bird and Tina. "Google's newest web framework" illustration: Ike's studded leather belt and his middle finger. Tell you what. In the spirit of the illustration i'll try to keep this post as bad ass as possible.

Listen up: A while back I blogged about debian-cd and creating a Debian net installer ISO. Imagine creating an entire operating system from scratch! Ok, not quite from scratch since you just download binaries, but hey, when you bake bread you don't mill your own flour do you? I didn't think so. So shut up. That previous debian-cd trip was just one stop along the road to creating a Xubuntu installer ISO from scratch and that is what this post will be about.

The most important thing to know about creating a Xubuntu installer is that there is a guy named Colin Watson who makes it all happen, ok so there is a whole army of people, but Colin's name is on everything. He is like the Ike Turner of the debian-installer set. Here is a link to the back room of his fun house where all the image making magic happens. He has bazaar branches of the following projects: cdimage, debian-cd, germinate, britney. You should check out branches of all those things with bzr at the link I gave you. The next thing to do is put them together. cdimage is the set of scripts that drives the whole thing, it handles synchronization of your local repository, generating the debian-cd task lists from seeds, calling debian-cd to burn the ISO's, publishing the ISO's and some other good stuff. So to "put them together" just drop debian-cd, britney and germinate into the cdimage directory.

Unless you think you are some kinda hotshot it would do you good to read the README that comes with each of these four tools, especially the cdimage one. If you don't, you better not come crying to anyone on the mailing list, because they will rip your arms off and beat you with them. It happens. I wish I could try to give exact directions for making this work, but there are too many steps and I forget things, so my recommendation is to immediately start trying to run cdimage, and resolve all the problems and complaints it has one at a time. Mostly you will just be figuring out what environment variables need to be set. (There are about 6 or 8 I set. I created a script with an export for each one and I source that before I run cdimage.) Here is the command for running cdimage: ./for_project xubuntu ./cron.daily

Great! You scrolled this far down the page so you must still have arms, or you are working your mouse with your mouth, either way, good job. Once you get through the first few errors (for not having the right variables defined) you will notice that the command is going to take forever to run. This is because it tries to rsync your local mirror with a remote mirror (Mine is about 100 GB for all the i386 stuff). Since you probably don't have a local mirror you probably can't skip this step, just rest assured that once you have it, you can re-use it and things will happen much faster, say 5 minutes or less. If you already have a local mirror you can just point the script to use that one . (I just put a symlink in the cdimage directory called ftp and it points to my local mirror at /opt/mirror.) Try not to get frustrated if you get stuck. There are lots of moving parts in here! LOTS! You are probably just missing some environment variable or you have one set to point at the wrong directory. The best way to work through things it is to read the source. Most error messages are pretty descriptive, but RESIST the temptation to hack the code. Colin's stuff really works. I got it all working with only one minor change to build_all.sh in debian-cd. For posterity, here is a list of the changes, tips and tricks that might help:

  • when debian-cd kicks off via the build_all.sh script it always complains that it can't find CONF.sh even though it is clearly in the right place. I change the ". CONF.sh" line at the beginning of the file to ". `pwd`/CONF.sh" and that fixes it.
  • If you already have a local mirror you can skip all the rsync business. To do this edit anonftpsynch in the cdimage bin directory and comment out the two rsync lines. I also comment out the lines that synch the indicies and overrides.
  • Britney is partially C code and you have to build it. Make sure you have the python-dev-all (I think that is the name) package as well as the one listed in the README. There is a step in there with some perl that I didn't do, but it didn't seem to matter.
  • Getting a local copy of the seeds really speeds up your build time a lot, so do it! You can set the LOCAL_SEEDS environment variable to the path of your seeds. If you make changes to the seeds be sure to commit them or germinate ignores your changes.
  • You can disable the creation of the Jigdo (stands for jigsaw domino, which is awesome) files if you don't care about them and save a bit of time and disk. Just edit debian-cd's CONF.sh and set the JIGDOsomething-something environment variable to 0.
  • Obviously you don't need 100GB mirror to create a 700 MB CD image. There are techniques for creating small local mirrors. None of them very handy. I'm using extensive --exclude parameters in my rsync command to make sure I only get i386 binaries. Check out this wiki page for info on other techniques.
  • It is imperative to use debian-cd and germinate from colin's repository. You can't mix and match the packaged versions with cdimage because colin has modified things quite a bit, so stick to his sources in bazaar for everything.
  • Fri. June 13th Edit: Be sure you have your gpg keys in order and cdimage is properly configured to use them or your APT repo on the CD won't be trusted, so nothing will get installed, which is kind of a bummer.
  • Make sure your GPG key you are using doesn't have a password, because GPG is running in batch mode when the CD is signed and thus won't be able to prompt you for a password.
Whatever you do don't get discouraged. I recommend asking questions on the ubuntu-installer IRC channel and mailing lists. People don't go out of their way to help you, mostly they ignore you, but sometimes you get lucky. Also, take time to pull back and understand the big picture about how the installer works and what all the files going onto the installer are for. That sort of context can help you avoid mistakes and direct your research. Good luck. And remember what Ike Turner said: "In China, when you get to the airport everyone be talking in American slang."

Sunday, June 1, 2008

C# in Depth


In the classic Steve Martin film, The Jerk, the main character, Navin, gets all worked up when his name shows up in the phone book:

Navin R. Johnson: The new phone book's here! The new phone book's here!
Harry Hartounian: Boy, I wish I could get that excited about nothing.
Navin R. Johnson: Nothing? Are you kidding? Page 73 - Johnson, Navin R.! I'm somebody now! Millions of people look at this book everyday! This is the kind of spontaneous publicity - your name in print - that makes people. I'm in print! Things are going to start happening to me now.

I'm not in the phone book, but my name is now in print!

Here is a picture of the back of C# in Depth, Jon Skeet's great new book (GO! Buy it!).

Do you see that? My name AND a quote. Things are going to start happening to me now.

A Noob Guide to Creating a Debian Installer ISO


It is hard to find an easy to understand set of instructions for creating a custom debian installer the real way. And when I say the 'real way' I mean using the same tool that the debian folks use to create their ISOs. That is why I am going to take some time to write out a very basic set of instructions. While this set won't tell you how to do that neat-o custom thing you are trying to do that has brought you here in the first place, it will get you from 0 to custom ISO in a hurry, orienting you to the process and giving you enough knowledge to tackle some of the more interesting customizations you want to try out.

Before we get to the nuts and bolts, here is the high level of what we are going to do. We are creating a bootable debian-installer ISO. To do this we use a tool called debian-cd, which is basically a build file and lots of perl scripts that automate everything. debian-cd requires that we have a local debian repository, and that is where most of the work is. Then you configure debian-cd to point to your local repository, and you tell it what you want it to make, there are tons of options here, but I am going to give instructions for the i386 netinstaller CD. Once it is configured you just run a few make tasks and you are done. Now without further blabbing I'll cut to the chase.

Create Your Local Repository

The steps enumerated here are just taking the package set from an existing debian net installer cd and putting them into a local repository on your machine. Be aware there are many different ways to do this including using apt-move, apt-mirror, reprepro, etc. Don't be confused if you see other people doing this different ways in different places.
  1. Download a debian netinst ISO
  2. Mount it like this: mount -o loop -t iso9660 filename.iso /mnt/place_to_mount_it
  3. recursively copy the pool directory into an empty directory called debian (we will refer to this directory as our local repository from now on).
  4. To get 2 other packages (debootstrap and installation-guide) we will need in there do the following. Download this and this. Move them to where they belong in the local repository. mv /path/to/debootstrap_blah_blah.deb /path/to/local/repo/debian/pool/main/d/debootstrap/ and mv /path/to/installation-guide_blah.deb /path/to/local/repo/debian/pool/main/i/installation-guide/
  5. We also need the override files for our repository. I just grab these from the remote repository here and copy them into the local repository in a new directory called indices. Make sure you get all the ones for the dist you are trying to build (etch).
  6. Now in order to make this a repository that apt will recognize we need to create the Packages.gz file, which is basically an index of everything in our repo. We use the apt-ftparchive command to do this in the next step.
  7. create a directory called config in your local repository
  8. create a file called apt.conf like this but with your paths substituted.
  9. create a file called etch-packages.conf like this but with your paths substituted.
  10. put both those files in the local repository config directory you just created.
  11. run the command like this: apt-ftparchive -c /path/to/apt.conf generate /path/to/etch-packages.conf
  12. You should get some errors like this: E: Could not open file /path/to/debian/dists/etch/main/binary-i386/Packages.gz.new - open (2 No such file or directory). Basically apt-ftparchive needs you to make the directory structure for it. Use the mkdir -p with the path the errors give you to create all the necessary directories.
Congratulations, when you get through all that you have a valid local repository with all the packages a debian netinstall needs! On to the next bit.

Put debian-installer in The Local Repository

Now debian-cd needs us to have our debian-installer (which is basically a mini Debian image in its own right) in a certain place in our local repository. These steps accomplish that. Note that there are multiple ways to do this and you can even build your own from source for further customization. For our demonstration we will just use the stock installer from the remote repo.
  1. Download the installer-i386 directory from a remote repository (it MUST be the from the same release as the ISO you grabbed so don't go mixing different repositories in here unless you know what you are doing). use wget to grab it, like this: wget -r ftp://ftp.us.debian.org/debian/dists/etch/main/installer-i386/ (make sure you run against the ftp site. the http site gives all kinds of junk html to you).
  2. Move the debian/dists directory that you just downloaded into the local repository like this: mv ftp.us.debian.org/debian/dists /path/to/debian
  3. Now we have to create a packages.gz for the installer packages too. So create a config file like this called etch-di.conf and put it into the same directory as the other repository config files from earlier.
  4. Now run this to generate: apt-ftparchive -c /path/to/apt.conf generate /path/to/etch-di.conf
Finishing Touches on the Local Repository
  1. create a directory called tools in the root of your local repo: mkdir /path/to/repo/debian/tools
  2. create a directory called doc in the root of your local repo: mkdir -p /path/to/repo/debian/doc/FAQ/html
  3. execute: touch /path/to/repo/debian/README
  4. execute: copy the release file from the ISO: cp /media/cdrom0/debian/dists/etch/Release /path/to/repo/debian/dists/etch/Release
  5. make sure you have a directory called current under /path/to/repo/debian/dists/etch/main/installer-i386/ (I didn't have that but had a dir called 20070308etch2 and I made a simlink from current to that with this command: ln -s /path/to/repo/debian/dists/etch/main/installer-i386/20070308etch2 /path/to/repo/debian/dists/etch/main/installer-i386/current)
Great! Now our local repository is complete and we are ready to generate an installer ISO from it!

Configure debian-cd

Here is where we set about configuring debian-cd to use our local repo.

  1. Edit CONF.sh and change the export MIRROR line to 'export MIRROR=/path/to/repo/debian'
  2. Edit CONF.sh and change the export TDIR line to 'export TDIR=/path/to/a/tmp/dir'
  3. Edit CONF.sh and change the output directory line to 'export OUT=/path/to/some/debian-cd-test
  4. Edit CONF.sh and change the apt tmp dir line to 'export APTTMP=/path/to/a/tmp/dir/apt'
  5. Uncomment the line that says #export INSTALLER_CD=2
  6. Login as root: sudo su - (don't forget the minus symbol at the end)
  7. Source the CONF.sh file: source CONF.sh
  8. run the following commands in order (if you get errors from them make sure to run make distclean before you try to re-run any commands otherwise you will have a dirty build). You can also consult the README in the /usr/share/debian-cd directory for more info about what is happening.
  9. make status
  10. make packagelists TASK=tasks/debian-installer+kernel-etch
  11. make image-trees
  12. make images
That is everything!!! Check the output directory you specified in CONF.sh for the image. You will find a bootable ISO that will install Debian on your system. Please note I have tried to be as specific as possible with these instructions, but frankly it was lots of typing and remembering. If you see or better yet experience problems with these let me know and I'll be happy to fix them.