Grabbing Rotten Tomatoes movie ratings with Clojure

flikr pic by Gammelmark

Currently I'm teaching myself Clojure from Stuart Halloway's excellent book Programming Clojure. Here's my first program that does something; a simple web page scraper to get the critics and audience ratings for movies off Rotten Tomatoes. Here's how it looks at the REPL:

rottentomatoes.core> (pmap-get-movie-ratings "lord of the rings")
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_return_of_the_king/
Audience 83
Critics 94
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_fellowship_of_the_ring/
Audience 92
Critics 92
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_two_towers/
Audience 92
Critics 96
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings/
Audience 74
Critics 50
movie url: http://www.rottentomatoes.com/m/master_of_the_rings_the_unauthorized_story_behind_jrr_tolkiens_the_lord_of_the_rings/
Audience 34
Critics null
movie url: http://www.rottentomatoes.com/m/jrr-tolkien-and-the-birth-of-the-lord-of-the-rings/
Audience 93
Critics null
movie url: http://www.rottentomatoes.com/m/jrr_tolkien_and_the_birth_of_the_lord_of_the_rings/
Audience 32
Critics null
movie url: http://www.rottentomatoes.com/m/more_at_imdbpro_creating_the_lord_of_the_rings_symphony_a_composers_journey_through_middle_earth/
Audience 100
Critics null
nil

I use leiningen to develop with Clojure (it's like Maven for Java), so if you want to build the project here's my project configuration that includes the dependencies used. I'm using swank-clojure which enables the REPL to function with emacs slime. http.async.client is a clojure API that builds on Netty and I use that for the GET requests to the Rotten Tomatoes server.

(defproject rottentomatoes "1.0.0-SNAPSHOT"
:description "Clojure code to grab movie ratings from Rotten Tomatoes"
:dependencies [
     [org.clojure/clojure "1.2.0"]
             [org.clojure/clojure-contrib "1.2.0"]
     [http.async.client "0.2.1"]
     ]
:main rottentomatoes.core
:dev-dependencies [
         [swank-clojure "1.2.1"]
          ]
)

And here's the code:

(ns rottentomatoes.core
(:gen-class)
(:require
[clojure.contrib.str-utils2 :as s]
[http.async.client :as c]))

(import [java.net URLEncoder]
 [java.lang.Character])

(def *base-url* "http://www.rottentomatoes.com")
(def *search-end-point* "/search/full_search.php?search=")

(defn first-match-after [re1 re2 seq]
"Splits the sequence SEQ using RE1 then searches after the first match and before the next match for the first occurence of RE2"
(let [[_ _ after] (s/partition seq re1)]
 (re-find re2 after)))

(defn response-status-code [resp]
(:code (c/status resp)))

(defn scoop-url [url]
"Use the http client to do a GET on the url"
(let [resp (c/GET url)]
   (c/await resp)
   [(response-status-code resp) (c/string resp)]))

;; Get movie urls
;; Does a search of Rotten Tomatoes for the search text, then scrapes the results
;; for the page for each movie. Returns a collection of the movie urls

(defn get-movie-urls [search-text]
(let [encoded-search-text (URLEncoder/encode search-text)
 [code body] (scoop-url (str *base-url* *search-end-point* encoded-search-text))
 ]
 (when (= code 200)
   (let [[_ _ after] (s/partition body #"<span>Title</span>")]
 (let [[_ & results] (s/partition after #"\"(/m/.*/)\"")]
   (map #(str *base-url* (second %)) (take-nth 2 results)))))))

;; Given a movie url GET the page then scrape it for the citic and audience rating

(defn get-movie-rating [movie-url]
(let [[code body] (scoop-url movie-url)]
 (if (= code 200)
   {:critics (second
     (first-match-after #"class=\"critic_side_container" #">([0-9]+)<" body))
    :audience (second
       (first-match-after #"class=\"fan_side" #">([0-9]+)<" body))})))   

;; Finds the ratings for all Rotten Tomatoes movies that match the search string and prints them out

(defn get-movie-ratings [search-str]
(let [urls (get-movie-urls search-str)]
 (when (> (count urls) 0)
   (doseq [url urls]
 (let [ratings (get-movie-rating url)]
   (printf "movie url: %s\n\tAudience %s\n\tCritics %s\n" url (:audience ratings) (:critics ratings)))))))

;; Slight variant on above that uses pmap so that the requests are done in parallel

(defn pmap-get-movie-ratings [search-str]
(let [urls (get-movie-urls search-str)]
 (when (> (count urls) 0)
   (let [ratings (pmap #(get-movie-rating %) urls)
     url-and-ratings (map vector urls ratings)]
 (doseq [[url ratings] url-and-ratings]
   (printf "movie url: %s\n\tAudience %s\n\tCritics %s\n" url (:audience ratings) (:critics ratings)))))))

I'm using the str-utils2 module for it's regex function partition, which will split a sequence up by regex matches. This made it easy to write the function `first-match-after', which finds a regex then finds the first occurrence of some text after that regex.

It was so easy to parallelize the requests. My first attempt at get-movie-ratings retrieved each movie page synchronously. By using pmap I was able to make it do the requests via thread pools, and thus return in a few seconds for many movie matches.

The code is much shorter than it would have been in Common Lisp, at least the way I program. I love the destructuring syntax, and that maps, vectors and lists can be returned from functions and manipulated without much effort.

I'm still new to Clojure so if you feel you can improve the code or have any feedback please let me know.

View Data from the Clojure REPL

Here's a nice debugging feature in Clojure. The inspect module lets you look at variables in a popup JFrame. The two examples below show how you quickly view data in a table or tree format. This is really handy to quickly view data from the REPL.

(require 'clojure.inspector)

(clojure.inspector/inspect-tree '(1 (a b) 2 (c d) 3 (e f )))

(clojure.inspector/inspect-table '((1 2 3) (a b c) (e f g)))

F# vs C#

Nice article comparing directly some code written in C# vs one in F#

http://sharp-gamedev.blogspot.com/2010/12/on-performance-of-f-on-xbox-360.html

What's interesting is that now that VM's are starting to become the new platforms, we are starting to be less restricted by language choice. When it's native code with hand crafted memory management you want, it has to be C++.

But when you start to look at the JDK and the .Net VM's, the language choice has far less impact on performance... after all you're using the same garbage collector, same base libraries and so on.

This is great news for those of us with more peculiar tastes in language (I like Clojure and Common Lisp for example).

Although I think it will be a few years until AAA console games run on VM's, if ever, due to the nature of that business. Memory is always at a premium, and the goal is to choke every last hz of CPU performance.

Just browsing

It's handy in emacs to be able to go straight to your browser to view a page, and you can do fancy stuff with `webjump' as I mentioned in a previous post.

Another handy function is `browse-url' which will prompt for the url but default to whatever your point is at.

If you're in a html file and you'd like to open that with your browser then `browse-url-of-file' will open it up.

Finally, and I just found this today, if you have some html code in a buffer that isn't even a file, you can open that in your browser to using `browse-url-of-buffer' and emacs will write it to a temporary file and open that in the browser for you.

rgrep on windows 7 for emacs

I lost an hour configuring this, so seems worthy of a blog post.

A fresh install of emacs for windows will have functionality that does not work because it depends on unix style utilities.

One very useful example is the command rgrep, which searches files recursively through subfolders looking for a regular expression in those files.

Under the hood it uses the unix command line tools find, and egrep. Unfortunately the windows version of find takes entirely different parameters and will not function. In fact you will get an error that looks like this:

FIND: Wrong parameter format

Using the set of native ports of Unix command line tools UnxUtils you can easily fix this:

1. Download the zip file and extract it to c:\unxutils
2. Add the following path to the very front of your path by editing your system environment variables C:\unxutils\usr\local\wbin\;

That's it. You can run rgrep now and hopefully you're up and running.

If not make sure you have restarted emacs so it picks up the new setting of PATH. Open a shell in emacs and type 'find --version'. You should see something like this if your path is configured correctly:

c:\find --version

find --version

GNU find version 4.1

and if not you will see:

C:\Windows\system32>find --help

FIND: Parameter format not correct

Couple of mysql tips

Wondering what the heck is going on on your mysql DB?

show processlist ;

Shows you all the active threads, which is very handy indeed. It shows what each thread is doing, and how long it has been doing it for.

Thread safe record insertion

When doing a record insert, another thread could be trying to insert a record that would violate any DB contraints, and if it gets there before you, then your insert will fail.

There may be cases where you want don't want to have it fail and try again; you may want to take a different action if the record already exists. For example lets say you have some data like this, where the Name is a unique key...

Name, Number of products purchase

Bob, 1,

Ben, 2,

Now if two threads try to insert a user 'Roger' who has just bought an item, then the one that arrives at the server last will fail.

INSERT INTO purchase (Name, PurchaseCount) VALUES ('Roger', '1') ON DUPLICATE KEY UPDATE PurchaseCount=PurchaseCount+1;

This will handle the key collision and execute the update clause instead, in the event that somebody beat you to it. This makes it much easier to write thread safe code that adds records.

Adjusting server logging level at runtime

log4j is an awesome logging library for Java. One thing I found out today is that you can change the logging level (debug, warn, info) at runtime, with a few lines of code.

static
{
       // This monitors the log4j for changes over a specified period of milliseconds
       PropertyConfigurator.configureAndWatch("./resources/log4j.properties", 60000);
}

What this function does is to tell log4j to check the configuration file every minute (60000 ms) for changes. So you can deploy a production server with only warn and error logging, but if something goes wrong you can enable debug and info logging (for example) simply by editing the configuration file and waiting a minute.

(source code formatted by this handy website http://formatmysourcecode.blogspot.com)