Monday, January 31, 2011

Groovy HTTPBuilder Cookies

Groovy's super-convenient HTTPBuilder makes web-scraping a breeze. Surprisingly, it doesn't seem to have a built-in DSL for adding pre-configured cookies to your client's session. That's okay, because they're pretty easy to add (once you plow through the documentation). Here's a quick example:

#!/usr/bin/groovy import groovyx.net.http.HTTPBuilder import org.apache.http.impl.cookie.BasicClientCookie @Grapes([ @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.1') ]) class MyScraper { static http = new HTTPBuilder('http://www.example.com/') static void main(String[] args) { try { run args } catch (e) { e.printStackTrace() } } static run(args) { // add some cookies addCookie domain: 'www.example.com', path: '/', name: 'mycookie', value: 'myvalue' addCookie domain: 'www.example.com', path: '/', name: 'anothercookie', value: 'anothervalue' // make some requests using the cookies def html = http.get(path: '/search', query: [q:'groovy']) // ... } // adds a cookie to the http client // with the specified name and value, and optional domain and path static addCookie(m) { // create the basic cookie object def cookie = new BasicClientCookie(m.name, m.value) // add optional cookie properties m.findAll { k,v -> !(k in ['name', 'value']) }. each { k,v -> cookie[k] = v } // add the new cookie to the client's cookie-store http.client.cookieStore.addCookie cookie } }

Thursday, January 27, 2011

Literal Tabs in Vi

My .vimrc, like any sane developer's, is configured to use spaces instead of tabs (with set expandtab). But sometimes you do actually need to insert a literal tab character (like when you're editing tab-delimited data). The web is pretty unhelpful on this topic — most of its advice is directed toward turning tabs into spaces, not vice-versa. Fortunately, vi's own help content has the answer (:help expandtab): in insert mode, just type [ctrl]-v [tab].

[Ctrl]-v is also useful when you need to insert other control characters, like a carriage return ([ctrl]-v [ctrl]-m), or a null ([ctrl]-v 0), etc. And you can also use it to insert characters via their unicode code-points (like μ [greek mu] with [ctrl]-v u03bc).

The other option for inserting literal tabs (useful if you need a bunch of them) is to temporarily turn off the expandtab setting, so that pressing [tab] in insert mode actually just inserts a literal tab. You can temporarily toggle off the expandtab setting with :set noexpandtab (or just :set noet), and then turn it back on again with :set expandtab (or :set et).

Sunday, January 16, 2011

MySQL to Groovy Gotchas

I've been running some raw sql in groovy recently (against a mysql db), and while most of the time the marshalling from sql result-sets to groovy objects is super convenient, there have been a couple things that caught me by surprise.

TINYINT Display Width

I had originally assumed that the "display widths" for integer types (like the (4) in INT(4)) were merely ornamental. But it turns out that, at least for TINYINTs, somewhere along the sql-to-groovy marshalling chain the display width is used to determine whether a TINYINT value should be marshalled as a boolean (for TINYINT(1)) or as a byte (for TINYINT(2)).

That's actually pretty clever, but not what I expected — maybe for BIT(1), where the values could only be 0 or 1— but not for TINYINT(1), where you'd expect the values might at least range from -9 to 9 (and still can actually be -128 to 127).

GROUP_CONCAT

I would have expected the result of this to be marshaled as a String (at least for TEXT fields) — but this seems to be instead marshalled as some sort of primitive array. When I dumped out its class name, it was [B. I think that might be like a primitive byte array; however, using groovy's join() method didn't work on it, so there must be something a little more complicated going on there. I didn't bother to check it out further — I just used the following to convert it to a string:

def sql = new Sql(dataSource) sql.eachRow('SELECT GROUP_CONCAT(my_field) AS concat FROM my_table GROUP BY other_field') { row -> println (row.concat as Character[]).join('') }

Sunday, January 2, 2011

Android SDK Upgrade

Just re-built an old android 1.0 application with the latest sdk (r8); was pretty smooth — all I had to do was:

  1. Install the official java 6 sdk (apt-get install sun-java6-jdk).
  2. Download the latest android sdk.
  3. Install the latest android sdk components.
  4. Download the latest eclipse ide ("classic" package).
  5. Install (and configure) the adt eclipse plugin.
  6. Delete the 1.0 project's .classpath and .project files.
  7. Create a new "android project" in eclipse in the same directory as the old one (where you deleted the old .classpath and .project.
  8. Filter out the .svn directories in eclipse's build path (if I had installed subclipse or something similar I probably could have skipped this).
  9. Add a new uses-sdk element to the AndroidManifest.xml to ignore some retro-fitted permission requirements.
  10. Increment the android:versionCode attribute in the AndroidManifest.xml.
  11. Generate some new android-market graphics (two screenshots, a high-resolution icon, a "promotional" graphic (seems to be used on the market's details page for the app), and a "feature" graphic (seems to be used on the market's top-level page if the app is featured on it).
  12. Clean rebuild, export the package, test it, publish!

Friday, December 10, 2010

Long Live XML

Reading Norman Walsh's Deprecating XML article (and others, like James Clark's XML vs the Web), on the topic of web apps removing XML from their APIs, made me reflect a little about what's happened to XML over the past decade. I began the decade working for a dot-bomb that was way into XML, envisioning a world-wide "XML backbone" of deep data-integration.

While that has not come to pass, I still see XML as having changed the world. Before XML, data exchange was almost always done either via bare-bones CSV (comma-separated values) or inscrutable binary formats. Even though CSV is at least human-readable, and binary formats can be carefully structured, there was just nothing (popular) in the 90's with more than a couple of these attributes:

  • human-readable
  • unicode built-in
  • platform/vendor-agnostic
  • structured
  • extensible

XML was never great for raw data — its real strength is structuring and annotating text — but each of the above was a big win for developers trying to share data. Having a default data-format that just worked everywhere for everything freed the developers of the 00's to think about what to do with the data, not how to send it on the wire.

In the 10's JSON will be used for data, and XML can go back to what it does best: text. So, okay, XML is dead; long live XML!

Sunday, June 20, 2010

Problem Accessing Grails Layouts

Just a quick note for posterity: If you're running a grails app (or probably any webapp) on jetty, and you configure the app to use the root (/) contextPath, and you leave jetty's default root webapp (/usr/share/jetty/webapps/root) around, you're in for a world of hurt. With grails, you'll get an obtuse 404 error page with the following message:

Problem accessing /WEB-INF/grails-app/views/layouts/my-custom-layout.gsp

(And no info in the logs.) The fix is of course to delete that vestigial root webapp.

Saturday, June 19, 2010

Cherokee Django

The Cherokee Cookbook and Yann Malet's Blog Roll post have pretty helpful directions for setting up the Cherokee webserver with Django. I found it a little tricky to set it up on a virgin Ubuntu Lucid Lynx system, though, so I thought I'd share my steps to get it going:

Install Cherokee and Django

You can get a fairly recent version of each from the Lucid repositories. Because you will compile uWSGI in a sec, you will also need to get python-dev and libxml2-dev:

$ sudo apt-get install cherokee libcherokee-mod-libssl python-django python-dev libxml2-dev

Install uWSGI

According to the cherokee docs, the best way to run django is via uWSGI. On the uwsgi home page, they do have a link to an ubuntu ppa for uwsgi. It looks pretty well maintained, and it's nice of Steve Crozz to maintain it, but I don't feel great about using someone's personal-package archive in a production environment. So I figured I would just get the code and build it myself.

Unfortunately uwsgi doesn't quite have the "forehead install" kind of make file. As I mentioned before, on ubuntu you need the python-dev and libxml2-dev debs, and you also need to specify the version of python you will use it with. These are the exact steps I took:

$ wget -nd -O /tmp/uwsgi.tar.gz http://projects.unbit.it/downloads/uwsgi-0.9.4.4.tar.gz $ tar xz -C /tmp -f /tmp/uwsgi.tar.gz $ cd /tmp/uwsgi-0.9.4.4 $ VERSION_TO_MAKE=`python --version 2>&1 | perl -ne '/(\d)\.(\d)/; print "$1$2"'` $ make -f Makefile.Py$VERSION_TO_MAKE $ sudo mv uwsgi$VERSION_TO_MAKE /usr/local/bin/uwsgi

Create the django_wsgi File

You need to have a wsgi script just like Yann described in his Blog Roll post; since I'm not all that hip to django, though, I didn't know exactly what to call it or where to put it — this is how I eventually ended up organizing my django project:

/srv sites MyProject __init__.py django_wsgi.py manage.py settings.py urls.py

That is, I called the wsgi script django_wsgi.py, and put it in the root of my django project, at the same level as the settings.py file. The django_wsgi.py script contains exactly what Yann described (where MyProject is the name of your project):

import os import django.core.handlers.wsgi # init django settings os.environ['DJANGO_SETTINGS_MODULE'] = 'MyProject.settings' # define wsgi app application = django.core.handlers.wsgi.WSGIHandler() # mount this application at the webroot applications = { '/': 'application' }

Configure Cherokee

Since I'm a complete cherokee noob, the cherokee docs were a little bit thin on this point (the screencast included in the docs might have been exactly what I needed, but I can't stand to sit around and watch screencasts, so I skipped it); this is what you do:

  1. Run the cherokee admin server: $ sudo cherokee-admin
  2. Open up your web browser to http://127.0.0.1:9090 and enter the credentials that cherokee-admin spit out.
  3. Click the Virtual Servers navigation link.
  4. Click the Wizards button at the bottom of the virtual servers list.
  5. Select Platforms from the Category list, and click the Run Wizard link next to uWSGI.
  6. On the first page of the wizard, enter the full domain name of your server in the New Host Name (once you get everything working with the new virtual server added by the wizard, you may want to delete it and just re-add the uwsgi handler manually in the default virtual server).
  7. The Document Root field is irrelevant for now; I just set it to my django project's static directory.
  8. For the Configuration File field, enter the path to your wsgi script (you'll change this setting in a sec anyway, so don't sweat it too much), like /srv/sites/MyProject/django_wsgi.py.
  9. For the Same logs as vserver field, I have no idea what this does, so I chose the default.
  10. Click the Submit button to finish the wizard, then go back and click the Information Sources navigation link, and select the new uWSGI 1 source you just created.
  11. The Interpreter field will contain something like this:
    /usr/local/bin/uwsgi -s 127.0.0.1:41058 -t 10 -M -p 1 -C -w /srv/sites/MyProject/django_wsgi.py
    — change it to this:
    /usr/local/bin/uwsgi -s 127.0.0.1:41058 -t 10 -M -p 1 -C --pythonpath /srv/sites -w MyProject.django_wsgi
    (where /srv/sites is the directory which contains your django project-directory (MyProject), and MyProject is the name of your project).
  12. Save it, and try navigating to your server in a browser.

Troubleshooting

Getting the right Interperter field setting in cherokee took me a while, and the logs were of no help. What you need to do is just run the uwsgi command (/usr/local/bin/uwsgi ...) yourself in a shell (killing the existing uwsgi instances if necessary); as long as you run it with the same socket as specified in cherokee's Connection field, cherokee will try to use the uwsgi instance you spun up, instead of spinning up its own instance. When you run uwsgi from a shell, you do get all the regular logging and error messages you would expect to see from django (from the shell stdout/stderr).

The other thing I found (which is probably pretty obvious if you know what you're doing) is that I had to give the uwsgi user (by default the same as the cherokee user, www-data) write rights on the django project. So I ended up creating a new copy of the django source from my working copy to /srv/sites/MyProject, and then making www-data the owner of that copy (sudo chown -R www-data:www-data /srv/sites/MyProject). There's probably a better way to do that, but this way worked.