Fiscal Conscience: Ha! How many times have you said that?
Me: Dozens of times!
Fiscal Conscience: And did you ever actually do it?
Me: No, but this time it will be different!
Fiscal Conscience: You're going to have do to better than that!
Me: Fine, I'll show you.

GnuCash is a wonderful piece of open-source accounting software. I have used it myself, dozens of times. But as it happens I inevitably stop using it. Not because the software is flawed per se, it does everything a good accounting application should do:

• Use Double-Entry accounting
• Support multiple currencies
• OFX import from online banking sites
• etc

The only thing they get wrong is the fundamental design choice they made when developing the application: they made it a GUI. Sure, GUIs are great, but in my opinion GUIs should be interfaces to a service oriented application rather than the application itself. Unless I'm sitting at my desk at home I can't use my application. I don't consider remote desktops a reasonable response to this problem due to the huge network latency issues as well as firewall problems.

I want to be able to record my transactions from anywhere. At home at my desk, at work at my desk, on my N800 when traveling, or on my phone when running around. With an application fundamentally written as a GUI this is essentially an insurmountable problem. An application written as a service can do all of these things quite easily with minimal amounts of programming.

Enter Ledger.

Ledger does not keep track of your accounts. You keep track of your accounts in a plain text file and Ledger helps you understand them better. I can update my checkbook register from any text editor, anywhere I am. Most often this is Emacs through SSH on my home computer. Because of the triviality of the file format (being plain text) I can write a simple application that takes text messages from my phone and adds them to the register automatically.

But what about all my automatic payments?

I have most of my monthly payments automatically debited from my account. This is nice since I don't have to spend the time submitting payments to X number of companies each month, but is a nightmare when it comes time to balancing my checkbook. What I have done in the past is to just import an OFX file from my bank directly into GnuCash. This works great until you realize that something is wrong with the import and your checkbook is no longer balancing correctly. For this reason alone I prefer to record each transaction I make by hand. My money is my responsibility after all.

So, I wanted to be instantly notified of any debit on my checking account as soon as it happens. This gives me the following:

• A text message to my phone describing the time, place and amount of the transaction
• The total balance of my account
• A way to constantly keep an eye on how I'm spending my money without the hassle of (remembering to) signing into my bank account
• A reminder to balance my checkbook. (tells me when my actual checking balance has been out of sync with ledger for over 48 hours.)

The central peice to this process is getting the transaction data from my bank account. My bank offers online banking but it does not offer an easy way to download my transactions. Sure it supports Quicken and MS Money exports, but its all protected behind a password protected javascript-enabled website. I wanted to use the wonderful mechanize library for python. However, it doesn't support javascript.

Enter Selenium RC.

Selenium RC is an application that can remote control a real web browser and submit and receive data. I run selenium which launches Firefox in a headless X-server (Xvfb) and continuously refreshes my online banking site and parses out my account activity. This is probably a bit memory intensive to always keep a browser open for this specific purpose. I may want to experiment with python-spidermonkey in the future and go back to using mechanize, but I think there's a lot of glue missing in that solution whereas Selenium RC works perfectly right now!

No code to share at the moment, it's all a giant hack. If a similar setup appeals to someone out there, let me know and I'll think about releasing something.

I created a directory to hold all of my emacs specific python functions: ~/.emacs.d/ryan-pymacs-extensions

I wrote the following python function, shorten_url.py in that directory:

!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = "Ryan McGuire (ryan@enigmacurry.com)"
__date__   = "Mon Sep 15 12:27:14 2008"

import doctest
import urllib2
import re

def shorten_with_is_gd(url):
    """Shorten a URL with is.gd

    >>> shorten_with_is_gd('http://www.enigmacurry.com')
    'http://is.gd/FFP'

    """
    u = urllib2.urlopen("http://is.gd/api.php?longurl="+url)
    return u.read()

def shorten_in_text(text):
    """Shorten all the urls found inside some text

    >>> shorten_in_text('Hi from http://www.enigmacurry.com')
    'Hi from http://is.gd/FFP'

    """
    replacements = {} #URL -> is.gd URL
    #Only check for urls that start with "http://" for now
    for m in re.finditer("http://[^ \n\r]*", text):
        try:
            replacements[m.group()] = shorten_with_is_gd(m.group())
        except:
            replacements[m.group()] = m.group()
    for url,replacement in replacements.items():
        text = text.replace(url, replacement)
    return text

if __name__ == '__main__':
    doctest.testmod(verbose=True)

and the following lisp makes "M-x shorten-url" do the rest of the replacement work:

;add ~/.emacs.d/ryan-python-extensions to python path
(pymacs-exec "import sys, os")
(pymacs-exec "sys.path.append(os.path.join(os.path.expanduser('~'),'.emacs.d','ryan-pymacs-extensions'))")

;;Shorten URLs with is.gd
(pymacs-exec "import shorten_url")
(defun shorten-url (start end)
  (interactive "r")
  (let ((region (buffer-substring start end)))
    (let ((rt (pymacs-eval (format "shorten_url.shorten_in_text('''%s''')" region))))
      (kill-region start end)
      (insert rt)
      )
  ))

Half way though implementing this, wouldn't you know it, I found ErcPageMe which does exactly what I wanted. I figured I was learning quite a bit and I continued writing my own version. I expanded on their code and (at least for me) made some improvements. So kudos go to whoever wrote ErcPageMe

The following code will pop up a message on your gnome desktop alerting you whenever you receive a personal message or when someone mentions your nickname in a channel. It also avoids notification for the same user in the same channel if they triggered a message within the last 30 seconds.

Here is my lisp and embedded python/pymacs code:

(defun notify-desktop (title message &optional duration &optional icon)
  "Pop up a message on the desktop with an optional duration (forever otherwise)"
  (pymacs-exec "import pynotify")
  (pymacs-exec "pynotify.init('Emacs')")
  (if icon
      (pymacs-exec (format "msg = pynotify.Notification('%s','%s','%s')"
                           title message icon))
    (pymacs-exec (format "msg = pynotify.Notification('%s','%s')" title message))
    )
  (if duration
      (pymacs-exec (format "msg.set_timeout(%s)" duration))
    )
  (pymacs-exec "msg.show()")
  )

;; Notify me when someone wants to talk to me.
;; Heavily based off of ErcPageMe on emacswiki.org, with some improvements.
;; I wanted to learn and I used my own notification system with pymacs
;; Delay is on a per user, per channel basis now.
(defvar erc-page-nick-alist nil
  "Alist of 'nickname|target' and last time they triggered a notification"
  )
(defun erc-notify-allowed (nick target &optional delay)
  "Return true if a certain nick has waited long enough to notify"
  (unless delay (setq delay 30))
  (let ((cur-time (time-to-seconds (current-time)))
        (cur-assoc (assoc (format "%s|%s" nick target) erc-page-nick-alist))
        (last-time))
    (if cur-assoc
        (progn
          (setq last-time (cdr cur-assoc))
          (setcdr cur-assoc cur-time)
          (> (abs (- cur-time last-time)) delay))
      (push (cons (format "%s|%s" nick target) cur-time) erc-page-nick-alist)
      t)
    )
  )
(defun erc-notify-PRIVMSG (proc parsed)
  (let ((nick (car (erc-parse-user (erc-response.sender parsed))))
	(target (car (erc-response.command-args parsed)))
	(msg (erc-response.contents parsed)))
    ;;Handle true private/direct messages (non channel)
    (when (and (not (erc-is-message-ctcp-and-not-action-p msg))
               (erc-current-nick-p target)
	       (erc-notify-allowed nick target)
	       )
      ;Do actual notification
      (ding)
      (notify-desktop (format "%s - %s" nick
                              (format-time-string "%b %d %I:%M %p"))
                      msg 0 "gnome-emacs")
      )
    ;;Handle channel messages when my nick is mentioned
    (when (and (not (erc-is-message-ctcp-and-not-action-p msg))
               (string-match (erc-current-nick) msg)
               (erc-notify-allowed nick target)
	       )
      ;Do actual notification
      (ding)
      (notify-desktop (format "%s - %s" target
                              (format-time-string "%b %d %I:%M %p"))
                      (format "%s: %s" nick msg) 0 "gnome-emacs")
      )
    )

  )

(add-hook 'erc-server-PRIVMSG-functions 'erc-notify-PRIVMSG)

My Emacs Environment

Below are the Emacs features most applicable to Python development:

• Rope and Ropemacs
  Rope is a general (non-emacs specific) Python IDE library. It has awesome support for multiple refactoring methods and code introspection. Inside Emacs, this gives us:
  • Full (working!!) code-completion support of modules, classes, methods etc. (M-/ and M-?)
  • Instant documentation for element under the cursor (C-c d)
  • Jump to module/class/method definition of element under the cursor (C-c g). This works for any Python code it finds in your PYTHONPATH, including things from the stdlib.
  • Refactoring of code (like rename — C-c r r)
  • List all occurences of of a name in your entire project
  • and More.
• YASnippet
  YASnippet is a snippet tool like TextMate. You can expand user defined keywords into whole blocks of predefined code. This is especially useful for the usual boilerplate that would go into a python file like

  #!/usr/bin/env python
  and
  if __name__ == '__main__':

  Granted, Python doesn't require much boilerplate, and therefore this package is much more suited to languages like Java, but I bring it up because I think its cool and if you get into the habit of using it, then a few keystrokes saved here and there will add up over time.

• Subversion support with psvn.el
  Psvn is a comprehensive subversion client for Emacs. It integrates well with ediff mode so you can use it to check changes between versions. It does all of the other boring subversion stuff well too.
• Ido-mode for buffer switching and file opening.
  Emacs, to the uninitiated, can be confusing because by default there is only one view into a single file at a time. How does one get to another file? Instead of cluttering the interface with GUIness and making the user click somewhere (and thereby forcing the user to waste their time by moving their hand off of the keyboard), Emacs gives powerful ways to switch between files. Ido-mode is one of these useful ways — it makes a list of open files starting with the most frequently visted files and widdles this list down as you type part of the filename. You can have dozens of files open and only be a few keystrokes away from any one of them.

A lot of people, for whatever reason, don't feel that Emacs is an IDE at all. I don't personally care what you define it as — the fact remains — Emacs is a powerful Python environment and despite being over 32 years old has proven to be just as modern as any IDE today, and remains THE most configurable editor (operating system?) ever.

I've tarred up my Emacs environment for general consumption. Instructions:

• Install Pymacs
• Install Rope and Ropemacs
• BTW, those three packages should be the only packages other than Emacs you'll need. Everything else is self contained.
• Extract the tarball to your home directory. This creates a directory called ryan-emacs-env.
• Rename "ryan-emacs-env" to ".emacs.d"
• Symlink my dot-emacs file to your .emacs. Run "ln -s .emacs.d/dot-emacs .emacs".
• If you also want to do Java development run "tar xfvz jde-2.3.5.1.tar.gz". I leave it tarred because you don't need to pollute your environment if you're not going to use Java. (Also for whatever reason, jde doesn't like to be stuck in my subversion repository so I just leave it tarred up and untar on every machine I check it out on.)

Extra tips:

• Put your .emacs.d directory under version control. Never rely on your distros emacs packages, install all future elisp files yourself in your .emacs.d file and commit to your repository often. This way you've got an environment that is easily transportable and synchronizable across multiple machines. This is the major reason why my emacs environment was so fast to trasnfer from my macbook pro to sontek's laptop during the demo.
• Speaking of sontek, he brought up an excellent point in #utahpython the other day, he's not going to be using my emacs environment except for reference, instead he's starting with a clean slate. This is by far the best and most prudent thing to do. My emacs environment is a culmination of several years of plugging in and deleting various packages and writing various snippets of elisp. Your needs are always going to be different than mine and you are also going to be better off by educating yourself along the way by creating your own.

Some more fun Emacs evangelism:

• a fascinating screencast of Emacs in the role of Ruby editor (many tips here apply to Python too)
• Emacs blogs I read:
  • Emacs life
  • Life is too short for bad code
  • Minor Emacs Wizardry
  • M-x all-things-emacs
  • Platypope.org
  • Know of any other good ones? Let me know.

Last spring I wrote an article about Multi-TTY Emacs. I had promised that I would update instructions for building it on Ubuntu, but since then I have been mainly using Gentoo on my laptop and never got around to updating it. Sorry

I switched to Ubuntu Gutsy Gibbon yesterday. I also got Multi-TTY Emacs running, and here's the video to prove it (If this is too small to see, download the original ogg theora version) :

Building CVS Emacs the old fashioned way

For those of you that want the easy way out, I've prebuilt an emacs CVS package that includes the Multi-TTY patch. I've tested that the package works on both Gutsy as well as Feisty.

For those of you that would like to learn how this is done, here's my recipe:

Install all the dependencies

sudo apt-get install build-essential cvs texinfo libx11-dev libxpm-dev libjpeg-dev libpng-dev libgif-dev libtiff-dev libgtk2.0-dev checkinstall

Check out the Emacs code from CVS

cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/cvsroot/emacs co emacs

Compile the source

cd emacs
./configure --with-gtk --prefix=/usr/local
make bootstrap
make

Checkinstall will build a basic emacs package for us with a minimum of fuss. We don't want to automatically install this package though for reasons we'll see below

sudo checkinstall -D --install=no

Checkinstall will ask you to change some values. I changed the maintainer email and the version number.

Checkinstall is great, and works for 99% percent of the packages that I've tried it with. Emacs is a bit different though — the package that checkinstall creates doesn't include everything that emacs requires, like all the .el files in the lisp folder among other things. So we have to edit the package a bit. For the following code, we'll assume the .deb file that checkinstall creates is just called "emacs_cvs-10202007-1_i386.deb", so adjust for whatever checkinstall called your .deb file:

Extract the data from the .deb

dpkg-deb -x emacs_cvs-10202007-1_i386.deb emacs_cvs-10202007-1_i386 #Or whatever your .deb is called

Extract the control information from the .deb

dpkg-deb -e emacs_cvs-10202007-1_i386.deb emacs_cvs-10202007-1_i386/DEBIAN

Now we need to add some of the files that checkinstall left out of the package

cp -a lisp/* emacs_cvs-10202007-1_i386/usr/local/share/emacs/23.0.50/site-lisp/
mkdir emacs_cvs-10202007-1_i386/usr/local/share/emacs/23.0.50/lisp
mkdir -p emacs_cvs-10202007-1_i386/usr/local/libexec/emacs/23.0.50/i686-pc-linux-gnu
cp -a etc emacs_cvs-10202007-1_i386/usr/local/share/emacs/23.0.50/
mkdir emacs_cvs-10202007-1_i386/etc
ln -s ../usr/local/share/emacs/23.0.50/etc/termcap.src emacs_cvs-10202007-1_i386/etc/termcap

Rebuild the modified package

dpkg-deb -b emacs_cvs-10202007-1_i386 emacs_modified.deb


To totally revolutionize the way you use Emacs, remember to check out my earlier post on Multi-TTY Emacs.

(Historical note May 2008: This post is now over a year old. Multi-TTY Emacs is now part of the main Emacs CVS tree. Most of this post is still applicable, but the build instructions are very much out of date and for the most part are unnecessary as the multi-tty patch has now made it into most distros. It should just be an 'apt-get install emacs-snapshot' away)

Multi-TTY Emacs is one of the most useful peices of software I've seen in a long time. Emacsclient, the underlying interface for multi-tty, has been around for a while. It allows someone to connect to a long running Emacs session and it avoids the long startup time Emacs usually has. However, it only works in a graphical environment. Multi-TTY Emacs allows us to do the same from either a graphical environment or from a simple terminal (tty).

Here's a typical use case: I'm at home and I'm using Emacs from within X-windows. I'm using the wonderful IRC client, ERC, and asking a question in #python, #emacs, #gentoo etc. but no one seems to be paying any attention and no one is answering my question right away. Oh well, it's time for me to head off to class or work. I leave Emacs running at home but since I forgot to enable chat logs, it's rather difficult for me to connect remotely and find out if anyone has responded to my question. With Multi-TTY, I can ssh into my home computer and bring up Emacs in a text console and check my ERC buffer even though I originally started Emacs graphically from X-Windows. Neat.

You can also see a video of it in action

Multi-TTY is contained in the CVS version (23) of Emacs. Version 23 is still a (tiny) bit rough around the edges, so don't blame me if while using it it deletes all your files and destroys your life in general. It won't, but caveat emptor. I am now running multi-tty on Gentoo and Ubuntu Feisty Fawn. Here's how:

Gentoo

On my laptop I run Gentoo Linux. Getting the latest version of Emacs on Gentoo was a breeze! :

• Setup Layman
• Add the emacs overlay: sudo layman -a emacs
• Add the following USE flags for app-editors/emacs-cvs:
  sudo flagedit app-editors/emacs-cvs X Xaw3d alsa gif gzip-el jpeg lesstif png sound spell tiff toolkit-scroll-bars xpm -gtk -hesiod -motif -source.

• GTK support is explicitly turned off as it causes problems with multi-TTY. This is no biggie for me as I always have (menu-bar-mode -1) and (tool-bar-mode -1) set.
• Emerge: sudo emerge emacs-cvs -va
• Tell the system to use the new emacs: sudo eselect emacs set emacs-23-multi-tty

Ubuntu

Ubuntu was much more difficult to setup correctly because the repositories don't have the CVS version yet (emacs-snapshot is 22). So, I've compiled it from source myself and once I clean up my instructions I will publish a HOWTO as well as my .deb package.

Update: See my new Ubuntu specific post for build instructions

Usage

Connecting to an already running emacs session is quite easy:

• In the main emacs session run M-x server-start
• Now just run emacsclient -c and you should see a new window pop up that has all the exact same buffers. If you are on a non-graphical terminal you'll likewise get the same emacs session albeit in text mode.

I found some excellent tips from sness.net for making this even easier. Here are my variations on his methods:

Make a new file somewhere in your path called preload_emacs:
#!/bin/bash
# Usage: preload-emacs []
#
# Preloads the Emacs instance called NAME in a detached screen
# session. Does nothing if the instance is already running. If WAITP
# is non-empty, the function waits until the server starts up and
# creates its socket; otherwise it returns immediately.
# Visit http://www.enigmacurry.com
# based on http://emacslife.blogspot.com/2007/05/multi-tty-emacs.html

if [ ! -e "$screendir"/*."$name" ]; then
if [ -e "$serverdir/$name" ]; then
# Delete leftover socket (for the wait option)
rm "$serverdir/$name"
fi
screen -dmS "emacs-$name" "$emacs" -nw –eval "(setq server-name \"$name\")" -f server-start
fi
if [ ! -z "$waitp" ]; then
while [ ! -e "$serverdir/$name" ]; do sleep 0.1; done
fi


Running preload_emacs -s ryan will now create an emacs server named 'ryan'. It loaded emacs in a screen session so that it will be able to be a long running process (Emacs apparently cannot run without an interface, hence we run it in a screen session to hide it away out of sight.)

To connect, we run emacsclient -c -s ryan. Typing plain old emacs here will load an entirely new Emacs session, which isn't really what we want. To make it a bit easier I created a shell script called "ec" that will load an emacsclient with these settings automatically:


#!/bin/bash -l
SERVERNAME=ryan
#Attempt to connect to an existing server
emacsclient -c -s $SERVERNAME $*
if [ $? -ne 0 ]
then
#Start a new emacs server and connect
preload_emacs $SERVERNAME 0
emacsclient -c -s $SERVERNAME $*
fi

Now when I type ec it will automatically attempt to connect to an already running emacs server named ryan. If it is not running, it will transparently create the server and then connect to it.

If you make "ec" your system's default editor, you'll now be able to use a lightning quick version of emacs for all your editing needs!

… I guess I should have RTFM years ago huh? … I'm such a dolt.

• Ryan