Saturday, May 29, 2010

Free Web Hosting on App Engine

I don't have much going on at www.swwomm.com; just a few static mockups and other documents (this blog is hosted by blogger.com). So I finally got around to transferring it from a paid host (lylix; no complaints other than that they have the gall to charge money for their services) to a free one: Google's App Engine (GAE).

I'm not sure why there isn't already a cookbook for transferring a static site to GAE, since it's pretty easy and painless — so here's my recipe.

Gotchas

But first, note that your lunch is not completely free — Google will 503 your ass if you exceed its (extremely generous) daily quotas for bandwidth and processing power.

Also, there are a few other GAE gotchas which apply to static content:

  1. Can't host a "naked" domain (ie example.com instead of www.example.com).
  2. Limit of 3000 files per app.
  3. No automatic directory listings.
  4. No automatic directory redirect (ie redirect from /foo to /foo/).
  5. No custom 404 page.

You can get around #3 and #4, however (as I describe below).

Setup

The first thing you need to do is get the GAE SDK. I'm using the version for python on linux.

With the SDK installed, you can start developing right away on your local dev machine. To deploy an app, of course, you need to sign up for a GAE account. They make you verify your account with an SMS message, so have your cell phone ready.

Hello World

Building a static app is really simple:

  1. Create a directory for your project (ie myapp).
  2. Create a sub-directory named static (or really, whatever you want to name it) inside the project directory. This will be the web root.
  3. Copy your static files into static.
  4. Add a boilerplate app.yml to the project directory.

This is the boilerplate app.yml:

application: myapp version: 1 runtime: python api_version: 1 default_expiration: "1d" handlers: # show index.html for directories - url: (.*/) static_files: static\1index.html upload: static(.*)index.html # all other files - url: / static_dir: static

The first line (application: myapp) specifies the name of your app. This name doesn't matter for development on your local machine; but when you use the GAE dashboard to create a new app, you'll be prompted for a name. The name must be unique globally on GAE (ie it has to be a name no other GAE user has claimed for his or her app), and GAE uses it in the default url for your application (ie http://myapp.appspot.com/). Once you've created the name and set up the app in the GAE dashboard, go back and change it here in app.yml.

The default_expiration setting is the default http cache-age for your static files; "1d" = one day, "4h" = 4 hours, etc. You can configure a separate expiration time for each url handler, but "1d" is probably good for most static content.

The first url in the handlers section captures all urls which end with a trailing-slash. These are directories; with static content you usually either want to display the index.html file of the requested directory, or, if there's no index.html, just the bare directory listing. Unfortunately GAE doesn't support listing static files, so this handler always just tries to display the index.html file in the requested directory.

The second url in the handlers section captures all other urls, and simply serves the requested static file.

Test It Out

At this point, you've already built a fully-functioning GAE app. You can test it out by running the test appserver (where $APPENGINE_HOME is the path to the GAE SDK on your local box, and $MYAPP_HOME is the path to your project directory):

$ $APPENGINE_HOME/dev_appserver.py $MYAPP_HOME

This boots up a test GAE appserver on port 8080. Enter http://localhost:8080/ into your browser address bar; it should load up the index.html from the root of your static directory.

Directory Woeage

But as I pointed out above, if you navigate to a sub-directory, and omit the trailing slash (ie http://localhost:8080/foo), you won't see the index.html for that sub-directory — you'll just get a blank page (and Google's generic 404 page when deployed to GAE).

This we can fix, however, by implementing a trivial RequestHandler in python. Create a file called directories.py (or whatever the heck you want to call it) in your project directory, and dump this into it:

import cgi from google.appengine.ext import webapp from google.appengine.ext.webapp.util import run_wsgi_app class RedirectToDirectory(webapp.RequestHandler): def get(self): self.redirect(self.request.path + "/", permanent=True) application = webapp.WSGIApplication([('/.*', RedirectToDirectory)], debug=True) def main(): run_wsgi_app(application) if __name__ == "__main__": main()

This creates a RequestHandler called RedirectToDirectory; this class simply appends a trailing slash to the current url and redirects. The other non-boilerplate line is just below, where RedirectToDirectory is registered to be used for all urls (/.*) handled by this directories.py script.

Next, drop in a url entry for the directories.py script into the handlers section of your app.yml (and yes, the order of the url entries is important):

application: myapp version: 1 runtime: python api_version: 1 default_expiration: "1d" handlers: # show index.html for directories - url: (.*/) static_files: static\1index.html upload: static(.*)index.html # redirect to directories (/foo to /foo/) - url: .*/[^.]+ script: directories.py # all other files - url: / static_dir: static

This url entry will capture all the urls which don't end in a trailing slash and don't have a file extension. It will handle requests for these urls by sending them to your directories.py script, which in turn will redirect them back to your app — but with a trailing slash this time.

Directory Listings

But if your directory doesn't have a index.html file, you're still SOL — unlike a normal webserver, you can't configure GAE to just display the directory listing. And you can't just implement a handler to do this — GAE apps don't have access to their static filesystem.

One possible workaround to this would be to store all your files as entries in the Big Table DB, and then serve them (and the directory listings) dynamically. This would require writing a bunch of code, however, when all you really want is just to serve some freaking static files already.

So I compromised and wrote a simple perl script which automatically creates static index.html files for a configurable list of static directories. To make it work, create a directories sub-directory in your project, and add to it four files:

make.pl
#!/usr/bin/perl -w open LIST, "directories/list.txt" or die $!; while (<LIST>) { s/\n//; # strip newline $title = $dir = $_; $title =~ s!.*/!!; # strip path from directory name # open directory index.html for writing open INDEX, ">$dir/index.html" or die $!; # dump header template into index.html # replacing %title% with directory name open HEAD, "directories/head.html" or die $!; while (<HEAD>) { s/%title%/$title/g; print INDEX; } close HEAD; # dump directory listing into index.html open DIR, "ls -lh $dir |" or die $!; while (<DIR>) { s/\n//; # strip newline # parse fields listed for each file @fields = split / +/, $_, 8; # skip lines that aren't file listings # and also skip this index.html next if ($#fields < 7 || $fields[7] eq 'index.html'); # print a table row for this file print INDEX '<tr><td class="name"><a href="' . $fields[7] . '">' . $fields[7] . '</a></td><td class="size">' . format_size($fields[4], $fields[0]) . '</td><td class="modified">' . $fields[5] . ' ' . $fields[6] . '</td></tr>' . "\n"; } close DIR; close INDEX; # dump footer into index.html `cat directories/foot.html >> $dir/index.html`; } close LIST; # format file size a little nicer than ls sub format_size { my($size, $perm) = @_; # skip for subdirectories return '-' if $perm =~ /^d/; # add bytes abbr $size .= 'B' if $size =~ /\d$/; return $size; }

This is the perl script. When you create it, make sure you make it executable:

$ chmod +x directories/make.pl

When you run it (after you've created the other three files), make sure you run it from your project directory:

$ cd myapp $ directories/make.pl
list.txt
static/foo static/foo/bar static/baz

This is the list of directories for which to auto-generate index.html files. Please note that the make.pl script will delete the existing index.html files in these directories. So make sure that you list only directories which don't have a custom index.html. (Plus this is another good reason to be using version control on your project.)

head.html
<html> <head> <title>%title% - My App</title> </head> <body> <h1>%title%</h1> <table> <thead> <tr><th>Name</th><th>Size</th><th>Modified</th></tr> </thead> <tbody>

This is the first part of template for the auto-generated index.html files. The perl script will replace the instances of %title% in this file with the directory name.

foot.html
</tbody> </table> </body> </html>

This is the second part of the template.

So add to directories/list.txt the paths of the directories you want to have listings, customize directories/head.html and directories/foot.html to your liking, and run directories/make.pl. This will create your directory-listing index.html files.

Deploy

And now you're ready to deploy. Make sure you've created the app in the GAE dashboard and updated your app.yml with its appspot name; then upload it:

$ $APPENGINE_HOME/appcfg.py update $MYAPP_HOME

You'll have to enter your GAE credentials, and wait for a minute or two while your app boots up on GAE; when your app is deployed and ready to use, the script will let you know. If you screw up your credentials, delete your ~/.appcfg_cookies file (which caches them), and try again.

If you've also signed up for Google Apps, you can use your (separate) Google Apps dashboard to configure a sub-domain of a domain you own to point to your deployed app (ie http://www.example.com/). Otherwise, you have to access it via its sub-domain of the appspot domain (ie http://myapp.appspot.com/).

Bon appétit!

Thursday, May 27, 2010

Lucid Windows with Xmonad

I just got a new "Bonobo" laptop from System 76 (which comes with Ubuntu 10.04 Lucid Lynx installed); and I set it up with xmonad, a nice clean, tiling window-manager. But instead of messing around with .xsession, I'm launching it via the "hybrid" method detailed on the xmonad haskell wiki. With this mechanism, I can still login to x with the default gnome desktop — but now I can also log in with xmonad; plus I can keep the gnome-panel etc that comes with the default ubuntu install. Also, this is pretty much the simplest way to set up xmonad on Lucid.

1. Install Xmonad

So the first thing is to install xmonad:

$ sudo apt-get xmonad

2. Create an Xmonad.start Script

Now crate a shell script to execute when you login with an xmonad session. It will simply set the WINDOW_MANAGER environment variable to xmonad, and then start up gnome in the regular fashion:

$ sudo echo '#!/bin/sh export WINDOW_MANAGER=xmonad gnome-session' > /usr/local/bin/xmonad.start $ sudo chmod +x /usr/local/bin/xmonad.start

3. Update Xmonad.desktop

Lucid has already created an xmonad.desktop entry in /usr/share/applications and /usr/share/xssessions for you; you just need to update them to run your xmonad.start script (instead of running xmonad directly):

$ sudo perl -pi -e 's/^Exec=xmonad$/Exec=xmonad.start/' /usr/share/{applications,xsessions}/xmonad.desktop

4. Set up Xmonad.hs

If you want some sort of dock displayed (like gnome-panel, dzen, xmobar, etc.) you need to set up a custom .xmonad/xmonad.hs configuration file; and make sure you include manageDocks in the manageHook, and avoidStruts in the layoutHook (and you need to import XMonad.Hooks.ManageDocks for this).

You can try out my xmonad.hs, or this simplified xmonad.hs:

import XMonad import XMonad.Hooks.ManageDocks import XMonad.Layout.Grid import XMonad.Layout.Master -- automatically manage docks (gnome-panel/dzen/xmobar/etc) myManageHook = manageDocks -- each layout is separated by ||| -- layout 1: grid w/ master (expand/contract by 3/100; master takes up 1/2 of screen) -- layout 2: standard full (master window fullscreen, hides dock) myLayoutHook = (avoidStruts $ mastered (3/100) (1/2) $ Grid) ||| Full main = xmonad $ defaultConfig { manageHook = myManageHook <+> manageHook defaultConfig, layoutHook = myLayoutHook }

After editing xmonad.hs, validate it with the following command:

$ xmonad --recompile

If you're already running xmonad, you can reload it with the new config by pressing <mod>-q.

5. Try it Out

You're all ready to try it out! Log out of your current x session (ie the Log Out... menu item in the gnome-panel), and on the login screen, select XMonad from the Sessions menu on the login screen's bottom panel:

If you're new to xmonad, follow the rest of the xmonad guided tour. If you screwed up something and xmonad won't start or appears to hang, hit <ctrl>-<shift>-<f1> to get to a terminal, login, fix xmonad.sh (with xmonad --recompile to check that it compiles), and then restart x with sudo service gdm restart.

Here's the finished product on my laptop:

Sunday, May 9, 2010

Firing Up My Jetpack

Jetpack is the new style (still in early development) of firefox extensions. Mozilla developers have made their in-progress work on this available for some time, however, and I finally took a look at the "jetpacks" available at the Jetpack Gallery. I was pretty impressed with what I found: there's already jetpack versions of some of the my favorite existing firefox extensions, like for flash blocking (ClickToFlash), capturing the colors used on a web page ("eyedropper" style: JetColorPicker), and taking screen-grabs (JetShot).

While I think I'll stick with the conventional firefox-extension versions of each of these tools for now, I was really impressed with how short and sweet the code to many of these jetpacks are. I can see why mozilla is planning to make this the future of extension building: there are tons and tons of web developers who are familiar enough with javascript and dom-manipulation to be able to whip these things out, and the jetpack-management UI has that same "view source" concept built into it that helped fuel the original web 1.0 explosion in the 90s.

But anyway, while I was looking at some of the jetpacks, I got the itch to built a few of my own. And the experience was both glorious and frustrating. Glorious because jetpacks are so wickedly easy to build and install; frustrating because there isn't yet much documentation for building them.

The "retired" jetpack documentation at the Mozilla Developer Center (and API Reference on your own installed jetpack page) are the only docs relevant to the current jetpack extension. The in-progress Jetpack SDK is the second cut at the "jetpack" methodology, and is designed to work with a forthcoming version of firefox — and not the current version of the jetpack extension (the distinction between these two different forms of "jetpack" are not called out clearly anywhere, which makes it confusing to figure out where to get started).

But enough ranting for now. If you want to learn how to build jetpacks, you pretty much just have to do it the same way you learned to build web pages — by viewing the source of other jetpacks. That's what I did, and here are the first few jetpacks I built:

Lookup in Wikipedia

I actually built the Lookup in Wikipedia jetpack second, but it's even simpler than the first, so I'll show it off first. It adds a "Lookup in Wikipedia" menu item to the right-click popup menu, so that when you select some text and then right-click and select this menu item, it'll open up a new tab with the wikipedia page for the text you selected:

// ==UserScript== // @name Lookup in Wikipedia // @description Adds a context menu item to lookup the selected text in wikipedia. // @copyright 2010 Justin Ludwig (http://jetpackgallery.mozillalabs.com/contributors/justinludwig) // @license The MIT License; http://www.opensource.org/licenses/mit-license.php // @version 1.0 // ==/UserScript== jetpack.future.import("menu"); jetpack.future.import("selection"); // add page context-menu item jetpack.menu.context.page.add({ label: "Lookup in Wikipedia", icon: "http://en.wikipedia.org/favicon.ico", command: function() { // lookup the current document's language var docLang = jetpack.tabs.focused.contentDocument.getElementsByTagName("html")[0].lang; // lookup the browser's preferred language var userLang = jetpack.tabs.focused.contentWindow.navigator.language; // ignore everything in the lang tag except for the general language var lang = (docLang || userLang).toLowerCase().replace(/([a-z]+).*/, "$1"); // when the user selects this menu-item, open a new tab with the wikipedia page for the text on the page that the user had selected jetpack.tabs.open("http://" + lang + ".wikipedia.org/wiki/" + encodeURIComponent(jetpack.selection.text)); } });

If it didn't have a little fancy code for trying to figure out what language to use, it would be a one-liner. For the language, you can see I justed used the standard browser dom of the current tab (jetpack.tabs.focused.contentDocument for what we normally refer to just as document and jetpack.tabs.focused.contentWindow for window) to get the current document language, or the browser's preferred language. Opening a new tab with the selected text is a breeze with the jetpack api (jetpack.tabs.open(url) to open the tab, and jetpack.selection.text to get the currently selected text). The jetpack.tabs object is part of the core jetpack api, but accessing jetpack.menu and jetpack.selection requires the jetpack.future.import() commands at the top of the script.

My Last Modified

I totally ripped off the idea for My Last Modified jetpack from azu_re's lastModified jetpack, but I made the display a little more customizable so it plays nice with Vimperator (Vimperator unfortunately seems to break the display of most of the jetpacks which show stuff in the status bar — I wish in fact Vimperator would just leave the status bar alone entirely, because I can see with time I'm going to want to give it over entirely to jetpacks).

Both mine and azu_re's displays the last-modified date for the current page in a little block on the status bar. The difference with mine is that the date format and css style of the block in the status bar is fully customizable:

// ==UserScript== // @name My Last Modified // @description Displays page's last-modified date. // @copyright 2010 Justin Ludwig (http://jetpackgallery.mozillalabs.com/contributors/justinludwig) // @license The MIT License; http://www.opensource.org/licenses/mit-license.php // @attribution azu_re (http://jetpackgallery.mozillalabs.com/jetpacks/367) // @version 1.0 // ==/UserScript== var manifest = { settings: [ // date/time format (using strftime) { name: "dateFormat", type: "text", label: "Date Format", "default": "%m/%d/%y %H:%M" }, // css style for status bar display { name: "statusStyle", type: "text", label: "Status Bar CSS Style", "default": "color:white; background-color:black; font-size:9px; width:75px; padding:2px;" } ] }; jetpack.future.import("storage.settings"); /** * Formats a date using the specified format. * @param format Format string. * @param date JS date object. * @return Formatted string. */ function strftime(format, date) { ... } jetpack.statusBar.append({ // add display div to status bar html: "<div id='my-last-modified' style='" + jetpack.storage.settings.statusStyle + "'></div>", onReady: function(widget) { // update display when tab focused or loaded function update() { var currentDocument = jetpack.tabs.focused.contentDocument; // get last-modified date of current tab var date = new Date(currentDocument.lastModified); // format date per user preference var status = strftime(jetpack.storage.settings.dateFormat, date); // update display $(widget).find('#my-last-modified').text(status); }; jetpack.tabs.onFocus(update); jetpack.tabs.onReady(update); } });

(I omitted the source to the strftime function from this listing, just because it's lengthy and has nothing jetpack-specific in it. You can get the full code for it from the My Last Modified jetpack's page in the jetpack gallery, though.)

The jetpack.statusBar.append() function is a super-easy way of adding some content to the browser's status bar — all you have to do is specify the initial html container, via the html option, and a function that's called when the html has been added to the status bar, via the onReady option. Since in this case the content changes whenever the active tab changes, my onReady function has an update() function in it that it registers to be called when a different tab is focused (via jetpack.tabs.onFocus()) or created (via jetpack.tabs.onReady()). The last line of the update() function uses jquery, conveniently included by default into each jetpack, to lookup the html container and update its text.

This jetpack also makes use of jetpack's super-easy way of exposing and persisting per-jetpack preferences: all you have to do is add a manifest global with a settings array property in it; each item in that array represents a separate preference. Users can access the preferences of any jetpack by typing "about:jetpack" in firefox's address bar, clicking the "Installed Features" link on the resulting page, and then clicking the "settings" link next to the listing for the jetpack. In the source code, you can access each setting as a property on the jetpack.storage.settings object (ie jetpack.storage.settings.dateFormat for a custom preference named dateFormat). You just have to remember to import the storage.settings "package" at the top of your jetpack to make this all work.

Log Open Ajax Events

The Log Open Ajax Events jetpack is the only really original jetpack I've built so far. It's a little more complicated (and took some trial and error to figure out which APIs to use), but still really simple. If the current page has included the Open Ajax Hub (which does the product I work on at work), this jetpack logs any events published to it to the Firebug console.

This is something for which I had already built a bookmarklet; but in jetpack form I don't have to turn it on every time I reload or view a different page — I can just turn it on when I want to debug something, and continue getting the events until I'm completely finished with the problem.

// ==UserScript== // @name Log OpenAjax Events // @description Logs configured openajax events to firebug. // @copyright 2010 Justin Ludwig (http://jetpackgallery.mozillalabs.com/contributors/justinludwig) // @license The MIT License; http://www.opensource.org/licenses/mit-license.php // @version 1.0 // ==/UserScript== var manifest = { settings: [ // comma-separated list of events to which to subscribe (* and ** wildcards allowed per OpenAjax spec) // clear this value completely to disable logging of any events { name: "eventName", type: "text", label: "Event Name", "default": "**" }, ] }; jetpack.future.import("storage.settings"); function subscribe() { // "window" context object of the current page var w = jetpack.tabs.focused.contentWindow.wrappedJSObject; // no firebug console or no openajax on this page if (!w.console || !w.OpenAjax) return; // lazy init our custom data object on each page var name = jetpack.storage.settings.eventName || ""; var data = w._logOpenAjaxEventsData; if (!data) data = w._logOpenAjaxEventsData = { name: "", subs: [] }; // already subscribed to this page (or no events to subscribe for) if (name == data.name) return; // remove existing subscriptions for (var a = data.subs, i = a.length - 1; i >= 0; i--) w.OpenAjax.hub.unsubscribe(a[i]); data.subs = []; // add new subscriptions if (name) for (var a = name.split(","), i = a.length - 1; i >= 0; i--) data.subs.push(w.OpenAjax.hub.subscribe(a[i], function(name, pubData, subData) { // log event (in a form that's collapsable so as not to pollute the log too badly) var o = {}; o["OpenAjax event: " + name] = pubData; w.console.dir(o); })); // remember event names we subscribed to (to check if they changed later) data.name = name; } // re-subscribe as necessary on every page focus or load jetpack.tabs.onFocus(subscribe); jetpack.tabs.onReady(subscribe);

I have it try to re-subscribe to the Open Ajax Hub every time a tab is focused or created (jetpack.tabs.onFocus() and jetpack.tabs.onReady() again). It stores its own custom _logOpenAjaxEventsData in each tab to remember if it's already subscribed, though, so it only actually re-subscribes if the user has updated the jetpack's eventName preference in the interim. The trick with this jetpack (that I had to find by combing through the source of a bunch of other jetpacks) was how to access the javascript "context" object of the current tab. Usually when scripting an html page, we access both the browser dom and the javascript context of the current page (the thing that holds the "global" objects) via the window property. In jetpack, you have to access these things through two distinct objects: the browser dom through jetpack.tabs.focused.contentWindow, and the javascript context via jetpack.tabs.focused.contentWindow.wrappedJSObject.

The only other tricky thing in this jetpack is just the way the Open Ajax events are logged; to make them display nicely in the firebug console (so that each event is displayed initially on a single line, but you can still drill into all the event properties) I created a new simple object for each event, and set the only property of that object to the event data, dynamically creating this one property's name with the event name in it as a friendly label for the event data object (ie "OpenAjax event: CAF.Update" in the screenshot above).

Log Window Events

I also created a Log Window Events jetpack that works pretty much the same as the Log Open Ajax Events jetpack. The difference is that it listens for standard window events (like onclick, onkeypress, etc.) instead of Open Ajax events. I won't bother including the source to it here, but like all jetpacks in the jetpack gallery, you can view the full code from its jetpack page (which, like I mentioned at the top, is probably the best innovation of this innovative extension style).

Let 'Er Rip!

So I love what the jetpack team has done so far, and I'm looking forward to the day when jetpack is included as part of the default firefox install. And hopefully as work on the new jetpack SDK progresses, many of the documentation holes will be filled in. But even if not, as long as this thing has "view source", I predict it will be a smashing success.