New blog location

Hi! I've moved this technical blog to github. Here's the link:

justinhj.github.io

Including equations in a blogger post

This is a test of following the guide here http://www.codecogs.com/latex/integration/htmlequations.php
A sample equation using latex:

\frac{1+sin(x)}{y}

Installing RPostgreSQL for R on Max OS Yosemite

I went through a painful period of digesting wrong answers to this question on Google. In fact most of the answers are just of the type "this is somebody else's problem so we can't help"

Anyway this is the eventual sequence of steps I found to install this that works:

brew install postgresql
svn checkout http://rpostgresql.googlecode.com/svn/trunk/ rpostgresql-read-only
cd rpostgresql-read-only
R CMD INSTALL --preclean RPostgreSQL

then in R just type library('RPostgreSQL') and you're good to go

Easy find

I found myself typing find . -name "*.java" | xargs -iHn "something"  so often I decided to make it into a bash script and put it in a new ~/Dropbox/bin folder so that I never have to do it again. Worse I never use the print0 option which let's you search files that have spaces in the filename, as it's too hard to remember. So, here's the script.

Sending notifications from Emacs (Mac OS X)

Sending notifications from emacs is something I find useful. In an earlier blog post I talked about how to use Growl to do so. http://justinsboringpage.blogspot.com/2009/09/making-emacs-growl.html

Well now you don't need Growl any longer. There's a neat github project called terminal notifier https://github.com/alloy/terminal-notifier which let's you send notifications from the terminal.

You can install it simply, via Homebrew or Rubygems as follows:

$ [sudo] gem install terminal-notifier

OR

brew install terminal-notifier

Then you can send notifications using a command like this:

terminal-notifier -message "hello"

Finally in order to send the notification from emacs we need to write a little Emacs lisp.

Check out this gist for the code I use:

https://gist.github.com/justinhj/eb2d354d06631076566f#file-gistfile1-el

This lets you send a notification in the future using M-x timed-notification

You are prompted for a time, and the format of that time can be given in a human readable way such as "2 seconds" or "5 minutes" (If you're curious for the allowed options look at the info page in emacs for the function timer-duration )

Then you are prompted for the message "Go to the store", and the message will be sent.

The code is very simple, it simply uses run-at-time to run the terminal command in the future. A useful command is find-executable, which given a name will find that executable in your path and run it. This makes configuring tools like this less effort.

Troubleshooting

Hey if it doesn't work first time what can you do?

1. Check if terminal notify is installed correctly by running at terminal
2. If that succeeds and you don't get a message check if you have enabled do not disturb mode
3. Otherwise if that succeeds and yet emacs isn't sending messages you likely don't have the executable on the path. M-x customize-variable exec-path

Watch

So this is pretty cool.

The watch command (available linux and on Mac via brew) will run a program every n seconds and display the results in a terminal.

For example, the following common will show the display above with human readable disk free space on your system. But since the command can be anything you want this is a pretty powerful tool.

watch -n 3 df -h

Checkout out your DB tables

I was doing some DB work today and wanted to be able to sort all DB tables based on the date they were created. Turns out you can do some neat stuff by looking in the information_schema.tables. For example this shows all the InnoDB tables.

select `table_schema`, `table_name`, `create_time`  FROM information_schema.tables where engine = 'InnoDB' order by create_time desc  ;

Configuring emacs to send iCloud mail on Mac OS X

Pic from ajc1 on Flikr

It's handy to be able to send emails from emacs, and this guide will show how to set up SMTP via an iCloud email account.

Step 1. Install gnutls

iCloud requires you to send emails over secure channel, and emacs supports sending email with starttls or gnutls. gnutls is available through brew

To install it is easy:

brew install gnutls

Wait a few minutes while your Mac gets hot downloading and compiling!

Step 2. Create an authinfo file

emacs can look in a file ~/.authinfo to find your login credentials, so create that file and fill in the blanks.

touch ~/.authinfo

chmod 600 ~/.authinfo

The contents of the file should read:

machine smtp.mail.me.com port 587 login YOURNAME@icloud.com password YOURPASSWORD

Step 3. Configure emacs

Add the following to your .emacs file:


(setq
 send-mail-function 'smtpmail-send-it
 message-send-mail-function 'smtpmail-send-it
 user-mail-address "YOURNAME@icloud.com"
 user-full-name "YOUR FULLNAME"
 smtpmail-starttls-credentials '(("smtp.mail.me.com" 587 nil nil))
 smtpmail-auth-credentials  (expand-file-name "~/.authinfo")
 smtpmail-default-smtp-server "smtp.mail.me.com"
 smtpmail-smtp-server "smtp.mail.me.com"
 smtpmail-smtp-service 587
 smtpmail-debug-info t
 starttls-extra-arguments nil
 starttls-gnutls-program (executable-find "gnutls-cli")
 smtpmail-warn-about-unknown-extensions t
 starttls-use-gnutls t)

Note that your gnutls program may be in a different spot. Find it with:

mdfind -name gnutls-cli 

Step 4. Testing

To compose an email C-x m

Enter an email and hit C-c c to send it.

If it works, great! If not switch to the *Messages* buffer for hints on what may have gone wrong.

Step 5. Sending emails from elisp code



(message-mail recipient subject)
(message-send-and-exit)))))

find grep on Mac OS X

On linux machines I search files using find, egrep and xargs as follows:

  find . -name "*.cpp" | xargs -i egrep -iHn "some search string" {}

this outputs any matches with the filename and number and also disables case dependency.

On my Mac it doesn't work. I tried reverting to egrep -r (to search recursively) instead, but that doesn't work. It just fails silently too. I tried installing findutils with brew to see if that helped, as often gnu tools are more up to date in brew than in the Apple version, but that didn't help.

So after some fiddling I found that the syntax below works:


  find . -name "*.cpp" | xargs egrep -iHn "some search string"

Only subtly different!

Actually, hold up, this does not work for filenames that have spaces in them. :(

Try this instead:

find . -type f -print0 | xargs -0 egrep -iHn "some search string"

J.

Starting a program to run in the background from a DOS prompt

Picture by ro_buk on Flickr

From a bash shell you can run a command in the background by adding an "&" to the end of the command, but how do you do the same thing in Windows?

Using the START command lets you run a task in the background (maximised or minimised), or in the foreground.

For example the following would run memcached in the background minimized.

C:\>start /min D:\platform\memcached.exe

Microsoft's documentation is below, but it seems the options have changed. /m does not work but /min does.

Microsoft's documentation on START

Making a emacs lisp expression expand itself to XML

My response to the blog article An Emacs Programming Challenge

The goal is to make the emacs lisp record below execute itself and produce XML. My lisp has gotten rusty so it took me a couple of hours to get this working, but I think I got there...

You can see it properly formatted and coloured here

It's fairly straightforward. The only complexity is the use of lexical-let. emacs lisp is dynamically scoped, so in the function I create for each keyword, where it prints the symbol name is a free variable at runtime. So if sym-name is not defined you'll get an error, otherwise you'll get whatever it is defined as instead of the value you wanted.

By using lexical-let any variables are bound lexically, that is stored in an environment that stays with the function we create when it is executed (a closure).

(require 'cl) ;; uses lexical-let

(defun make-xml-izer(name)
 "Given a symbol NAME this makes a function that outputs an xml string for that
symbol and using fset binds it to the same symbol so it becomes executable.
This pollutes the global function namespace so be careful with which names you
pass in"
 (lexical-let ((sym-name name))
   (fset (intern name)
   (lambda(&rest input)
     "returns a string representing the xml encoding of the input sexp"
     (let ((res (format "<%s>" sym-name)))
       (dolist (item input)
  (if (listp item)
      (setf res (concat res (eval item)))
    (setf res (format "%s%s" res item))))
       (setf res (format "%s" res sym-name))
       res)))))

;; executing this makes all the symbols in the list below executable 
(mapcar (lambda(s)
   (make-xml-izer
    (symbol-name s)))
 '(record date millis sequence logger level class method thread emessage exception frame line))

;; the sample record
(record
 (date "2005-02-21T18:57:39")
 (millis 1109041059800)
 (sequence 1)
 (logger nil)
 (level 'SEVERE)
 (class "java.util.logging.LogManager$RootLogger")
 (method 'log)
 (thread 10)
 (emessage "A very very bad thing has happened!")
 (exception
   (emessage "java.lang.Exception")
   (frame
     (class "logtest")
     (method 'main)
     (line 30))))

eredis update

I've been busy on my emacs redis client eredis and it now supports the entire API. It still needs a bit more polish but it should be a workable Redis client now, and I will continue to play with the org-mode table integration which I think has a lot of potential uses: for example making a gui to edit server parameters in just a few seconds.

Also made a new video showing some of the new org table creating commands and the monitor mode that shows the Redis commands as they are run on the server:

eredis: a Redis client in emacs lisp

I set up a google code project today for eredis. A Redis client in emacs lisp.

The program consists of a single emacs lisp file eredis.el

emacs lisp includes facilities for writing network applications. In my code I use `make-network-process' to open a connection to a specified redis server. Then the Redis api is exposed.

One nice feature of emacs I have used is org-table-mode. This lets you edit and manage the data in a Redis server in an org table. For example, you can grab all keys matching a pattern and create an org table from the key value pairs, then edit that table. You can then send it back to Redis with interactive commands that send either the whole table, or just the current row back to Redis using mset or set.

This work flow is not safe when working with multiple users, if you care about overwriting each others data. For example, I could store the last values you got from Redis in addition to your edited values. When you go to set a new value I first grab it from Redis, check if it has changed since you got it, and if so warn you (showing you the new value). For many work flows this would work well. For example the use case of a group of users editing a shared DB of configuration data.

Emacs progress indication

When programming in emacs lisp, there is an easy way to show progress feedback to the user when a task will take some time. Here's a block of code from the elisp manual showing how to do it.

(let ((progress-reporter

(make-progress-reporter "Collecting mana for Emacs..."

0 500)))

(dotimes (k 500)

(sit-for 0.01)

(progress-reporter-update progress-reporter k))

(progress-reporter-done progress-reporter))

I've incorporated this into my duplicate files code, linked below...

http://code.google.com/p/justinhj-emacs-utils/source/browse/trunk/duplicates.el

More on duplicate file handling in emacs dired

I had some good feedback on my last post, that it would be useful to be leave just the superfluous duplicate files marked in the dired buffer. After doing that you can then copy them to another folder, or delete them.

I've added a function to do this `dired-mark-duplicate-files', and updated the google code site with the changes.

To copy to another folder use R and select a folder to move the dupes to. In order to delete them hit D.

Finding duplicate files in a dired buffer

picture by Donald MacLeod

This is a an example of programming emacs in emacs-lisp just to give an idea of what you can put together in an hour or two. I was looking at a dired buffer with a bunch of photos in, and some were the same photo that I'd downloaded twice. So I started thinking about writing a utility in emacs to automatically find and remove the duplicate files. In this post I'll just show the code for finding the files and display their filenames in a buffer.

I've put the source on google code.

After downloading you can load the source into emacs and call `eval-buffer', then open up a dired buffer to try it out. For this to be useful you need some duplicated files, so make some if you need to.

Mark the files you want to check for duplicates. For example to mark all jpg files you would type %m to mark files matching a regexp and type .*\.jpg

Now execute the command `dired-show-marked-duplicate-files' and after a short delay (in my test 80 jpg photos took about 5 seconds) you'll see a buffer called 'Duplicated files' which contains a list of the files which have the same contents.

Next steps for this little project will be to give you an interactive way to delete the duplicated files. I haven't decided quite how I'd like that to work, drop me an email if you have an idea. I've been thinking about perhaps resetting which files are marked so that only the duplicates are marked. At that point you can then hit R to move them to another spot, or delete them with x.

Now some comments about the code involved...

Most of the work is done in the function dired-show-marked-duplicate-files. First line " (interactive)" makes it an interactive function, meaning the user of emacs can invoke it.

"(if (eq major-mode 'dired-mode)" will check that we're in the right kind of buffer, because it makes no sense to run this in another mode.

In order to find the duplicate files I just need to walk the list of marked files, generate the md5 value of the contents of each one and add it to hash table. The keys in the hash table will be the md5, and the values will be a list of files with that md5. Once we've done that, finding duplicates is a simple matter of walking the hash table keys and displaying any where the value has multiple entries.

"(let ((md5-map (make-hash-table :test 'equal :size 40)))" Creates the hash table, making sure we use 'equal to match our filenames.

"(let ((filenames (dired-get-marked-files)))" this gets the marked files as a list of filenames

The next little bit of code is just to store the item in the hash table after getting the md5. There's no function in emacs to get the md5 of a file, but you can get the md5 of a string, so I wrote a helper function for getting the contents of a file into a temporary buffer first.

(defun md5-file(filename)

"Open FILENAME, load it into a buffer and generate the md5 of its contents"

(interactive "f")

(with-temp-buffer

(insert-file-contents filename)

(md5 (current-buffer))))

Finally I want to display the results, so I create a buffer and then use maphash (walks the keys of a hash table executing a function on each) with a helper function `show-duplicate' which simply writes the values of the hash table entry into that buffer.

Programmer tips for Mac OSX

Some tips for programmers on the Mac.

emacs: the best place to get emacs for Mac seems to be here http://emacsformacosx.com/ which is also the most no nonsense website design ever

Not really a Mac tip, but I stick my .emacs configuration file in a Dropbox folder, along with any emacs libraries and emacs lisp code I write. Then where-ever I install emacs I make a simple .emacs that points to the one in the Dropbox folder. This also forces me to make sure any platform specific emacs stuff is properly handled.

Clipboard: copy and paste between the terminal and other apps can be done with pbcopy and pbpaste. For example a long complicated command line you want to email to yourself, just do:

echo "long complicated bash command line you don't want to retype" | pbcopy

And then you can Command-V that into your email window. Going the other way is just as simple; Command-C the text you want and pop it into the terminal window with pbpaste.

Open: If you want open an application from the command line you can do it like this:

open -a SomeApp /Users/yourname/SomeFile.hai

You can open a folder in finder

open /Folder/

or

open /Folder/SomeFile.hai

to open that file with it's default application.

Check out the help 'man open', to see other stuff like how you pipe stdout into an application.

Finally check out this guys OpenTerminalHere script. This pops an icon in finder that lets you open a terminal window in the highlighted folder.

http://www.entropy.ch/software/applescript/

MovieRatings

A little side project I did when learning about Clojure was to grab movie ratings from Rotten Tomatoes, which I did a post about here:

http://justinsboringpage.blogspot.com/2011/01/grabbing-rotten-tomatoes-movie-ratings.html

This is just an update that I've posted the whole leiningen project onto github

https://github.com/justinhj/movieratings

Talking to mysql from Python on Mac OS X 10.6

image by Sam Pullara

Here's a problem I couldn't solve with Google, although it seems to be a moving target so YMMV.

I wanted to do some work driving a MySQL database with Python. On Windows and Linux I've used MySQLdb so I decided to do the same on Mac.

For reference I got mysql (client and server) through mac ports.

mysql5 --version

mysql5 Ver 14.14 Distrib 5.1.45, for apple-darwin10.4.0 (i386) using readline 6.1

and Python is the stock version:

Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)

[GCC 4.2.1 (Apple Inc. build 5646)] on darwin

First download from here http://sourceforge.net/projects/mysql-python/ and extract the file somewhere...

mkdir ~/pythondb

cd ~/pythondb

tar -vxf ~/Downloads/MySQL-python-1.2.3.tar.gz

Then you need open up the site.cfg file and make a change as below. Your mysql_config5 maybe in a different spot. You can find out with the command 'which mysql_config5'.

# The path to mysql_config.

# Only use this if mysql_config is not on your PATH, or you have some weird

# setup that requires it.

mysql_config = /opt/local/bin/mysql_config5

Now execute the following:

python setup.py build

python setup.py install

If everything works you can now import MySQLdb in your python program and start interacting with mysql.

Grabbing Rotten Tomatoes movie ratings with Clojure

flikr pic by Gammelmark

Currently I'm teaching myself Clojure from Stuart Halloway's excellent book Programming Clojure. Here's my first program that does something; a simple web page scraper to get the critics and audience ratings for movies off Rotten Tomatoes. Here's how it looks at the REPL:

rottentomatoes.core> (pmap-get-movie-ratings "lord of the rings")
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_return_of_the_king/
Audience 83
Critics 94
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_fellowship_of_the_ring/
Audience 92
Critics 92
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings_the_two_towers/
Audience 92
Critics 96
movie url: http://www.rottentomatoes.com/m/lord_of_the_rings/
Audience 74
Critics 50
movie url: http://www.rottentomatoes.com/m/master_of_the_rings_the_unauthorized_story_behind_jrr_tolkiens_the_lord_of_the_rings/
Audience 34
Critics null
movie url: http://www.rottentomatoes.com/m/jrr-tolkien-and-the-birth-of-the-lord-of-the-rings/
Audience 93
Critics null
movie url: http://www.rottentomatoes.com/m/jrr_tolkien_and_the_birth_of_the_lord_of_the_rings/
Audience 32
Critics null
movie url: http://www.rottentomatoes.com/m/more_at_imdbpro_creating_the_lord_of_the_rings_symphony_a_composers_journey_through_middle_earth/
Audience 100
Critics null
nil

I use leiningen to develop with Clojure (it's like Maven for Java), so if you want to build the project here's my project configuration that includes the dependencies used. I'm using swank-clojure which enables the REPL to function with emacs slime. http.async.client is a clojure API that builds on Netty and I use that for the GET requests to the Rotten Tomatoes server.

(defproject rottentomatoes "1.0.0-SNAPSHOT"
:description "Clojure code to grab movie ratings from Rotten Tomatoes"
:dependencies [
     [org.clojure/clojure "1.2.0"]
             [org.clojure/clojure-contrib "1.2.0"]
     [http.async.client "0.2.1"]
     ]
:main rottentomatoes.core
:dev-dependencies [
         [swank-clojure "1.2.1"]
          ]
)

And here's the code:

(ns rottentomatoes.core
(:gen-class)
(:require
[clojure.contrib.str-utils2 :as s]
[http.async.client :as c]))

(import [java.net URLEncoder]
 [java.lang.Character])

(def *base-url* "http://www.rottentomatoes.com")
(def *search-end-point* "/search/full_search.php?search=")

(defn first-match-after [re1 re2 seq]
"Splits the sequence SEQ using RE1 then searches after the first match and before the next match for the first occurence of RE2"
(let [[_ _ after] (s/partition seq re1)]
 (re-find re2 after)))

(defn response-status-code [resp]
(:code (c/status resp)))

(defn scoop-url [url]
"Use the http client to do a GET on the url"
(let [resp (c/GET url)]
   (c/await resp)
   [(response-status-code resp) (c/string resp)]))

;; Get movie urls
;; Does a search of Rotten Tomatoes for the search text, then scrapes the results
;; for the page for each movie. Returns a collection of the movie urls

(defn get-movie-urls [search-text]
(let [encoded-search-text (URLEncoder/encode search-text)
 [code body] (scoop-url (str *base-url* *search-end-point* encoded-search-text))
 ]
 (when (= code 200)
   (let [[_ _ after] (s/partition body #"<span>Title</span>")]
 (let [[_ & results] (s/partition after #"\"(/m/.*/)\"")]
   (map #(str *base-url* (second %)) (take-nth 2 results)))))))

;; Given a movie url GET the page then scrape it for the citic and audience rating

(defn get-movie-rating [movie-url]
(let [[code body] (scoop-url movie-url)]
 (if (= code 200)
   {:critics (second
     (first-match-after #"class=\"critic_side_container" #">([0-9]+)<" body))
    :audience (second
       (first-match-after #"class=\"fan_side" #">([0-9]+)<" body))})))   

;; Finds the ratings for all Rotten Tomatoes movies that match the search string and prints them out

(defn get-movie-ratings [search-str]
(let [urls (get-movie-urls search-str)]
 (when (> (count urls) 0)
   (doseq [url urls]
 (let [ratings (get-movie-rating url)]
   (printf "movie url: %s\n\tAudience %s\n\tCritics %s\n" url (:audience ratings) (:critics ratings)))))))

;; Slight variant on above that uses pmap so that the requests are done in parallel

(defn pmap-get-movie-ratings [search-str]
(let [urls (get-movie-urls search-str)]
 (when (> (count urls) 0)
   (let [ratings (pmap #(get-movie-rating %) urls)
     url-and-ratings (map vector urls ratings)]
 (doseq [[url ratings] url-and-ratings]
   (printf "movie url: %s\n\tAudience %s\n\tCritics %s\n" url (:audience ratings) (:critics ratings)))))))

I'm using the str-utils2 module for it's regex function partition, which will split a sequence up by regex matches. This made it easy to write the function `first-match-after', which finds a regex then finds the first occurrence of some text after that regex.

It was so easy to parallelize the requests. My first attempt at get-movie-ratings retrieved each movie page synchronously. By using pmap I was able to make it do the requests via thread pools, and thus return in a few seconds for many movie matches.

The code is much shorter than it would have been in Common Lisp, at least the way I program. I love the destructuring syntax, and that maps, vectors and lists can be returned from functions and manipulated without much effort.

I'm still new to Clojure so if you feel you can improve the code or have any feedback please let me know.