Monday, January 31, 2011

Groovy HTTPBuilder Cookies

Groovy's super-convenient HTTPBuilder makes web-scraping a breeze. Surprisingly, it doesn't seem to have a built-in DSL for adding pre-configured cookies to your client's session. That's okay, because they're pretty easy to add (once you plow through the documentation). Here's a quick example:

#!/usr/bin/groovy import import org.apache.http.impl.cookie.BasicClientCookie @Grapes([ @Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.1') ]) class MyScraper { static http = new HTTPBuilder('') static void main(String[] args) { try { run args } catch (e) { e.printStackTrace() } } static run(args) { // add some cookies addCookie domain: '', path: '/', name: 'mycookie', value: 'myvalue' addCookie domain: '', path: '/', name: 'anothercookie', value: 'anothervalue' // make some requests using the cookies def html = http.get(path: '/search', query: [q:'groovy']) // ... } // adds a cookie to the http client // with the specified name and value, and optional domain and path static addCookie(m) { // create the basic cookie object def cookie = new BasicClientCookie(, m.value) // add optional cookie properties m.findAll { k,v -> !(k in ['name', 'value']) }. each { k,v -> cookie[k] = v } // add the new cookie to the client's cookie-store http.client.cookieStore.addCookie cookie } }

Thursday, January 27, 2011

Literal Tabs in Vi

My .vimrc, like any sane developer's, is configured to use spaces instead of tabs (with set expandtab). But sometimes you do actually need to insert a literal tab character (like when you're editing tab-delimited data). The web is pretty unhelpful on this topic — most of its advice is directed toward turning tabs into spaces, not vice-versa. Fortunately, vi's own help content has the answer (:help expandtab): in insert mode, just type [ctrl]-v [tab].

[Ctrl]-v is also useful when you need to insert other control characters, like a carriage return ([ctrl]-v [ctrl]-m), or a null ([ctrl]-v 0), etc. And you can also use it to insert characters via their unicode code-points (like μ [greek mu] with [ctrl]-v u03bc).

The other option for inserting literal tabs (useful if you need a bunch of them) is to temporarily turn off the expandtab setting, so that pressing [tab] in insert mode actually just inserts a literal tab. You can temporarily toggle off the expandtab setting with :set noexpandtab (or just :set noet), and then turn it back on again with :set expandtab (or :set et).

Sunday, January 16, 2011

MySQL to Groovy Gotchas

I've been running some raw sql in groovy recently (against a mysql db), and while most of the time the marshalling from sql result-sets to groovy objects is super convenient, there have been a couple things that caught me by surprise.

TINYINT Display Width

I had originally assumed that the "display widths" for integer types (like the (4) in INT(4)) were merely ornamental. But it turns out that, at least for TINYINTs, somewhere along the sql-to-groovy marshalling chain the display width is used to determine whether a TINYINT value should be marshalled as a boolean (for TINYINT(1)) or as a byte (for TINYINT(2)).

That's actually pretty clever, but not what I expected — maybe for BIT(1), where the values could only be 0 or 1— but not for TINYINT(1), where you'd expect the values might at least range from -9 to 9 (and still can actually be -128 to 127).


I would have expected the result of this to be marshaled as a String (at least for TEXT fields) — but this seems to be instead marshalled as some sort of primitive array. When I dumped out its class name, it was [B. I think that might be like a primitive byte array; however, using groovy's join() method didn't work on it, so there must be something a little more complicated going on there. I didn't bother to check it out further — I just used the following to convert it to a string:

def sql = new Sql(dataSource) sql.eachRow('SELECT GROUP_CONCAT(my_field) AS concat FROM my_table GROUP BY other_field') { row -> println (row.concat as Character[]).join('') }

Sunday, January 2, 2011

Android SDK Upgrade

Just re-built an old android 1.0 application with the latest sdk (r8); was pretty smooth — all I had to do was:

  1. Install the official java 6 sdk (apt-get install sun-java6-jdk).
  2. Download the latest android sdk.
  3. Install the latest android sdk components.
  4. Download the latest eclipse ide ("classic" package).
  5. Install (and configure) the adt eclipse plugin.
  6. Delete the 1.0 project's .classpath and .project files.
  7. Create a new "android project" in eclipse in the same directory as the old one (where you deleted the old .classpath and .project.
  8. Filter out the .svn directories in eclipse's build path (if I had installed subclipse or something similar I probably could have skipped this).
  9. Add a new uses-sdk element to the AndroidManifest.xml to ignore some retro-fitted permission requirements.
  10. Increment the android:versionCode attribute in the AndroidManifest.xml.
  11. Generate some new android-market graphics (two screenshots, a high-resolution icon, a "promotional" graphic (seems to be used on the market's details page for the app), and a "feature" graphic (seems to be used on the market's top-level page if the app is featured on it).
  12. Clean rebuild, export the package, test it, publish!