RYAN MCGUIRE

EnigmaCurry.com now powered by Blogofile

Phew!

I've just finished porting this entire site over to my new blog engine, Blogofile. Blogofile is something I've been working on for the last several months, and really it's been pretty much complete for the last two of those months; it turns out that transferring the 100+ posts that I've made on this blog since 2005, first using TikiWiki, then Wordpress (using a myriad of different plugins over the years) was a much larger task than writing Blogofile itself. Or, at least writing Blogofile was a lot more fun, which is why it probably took me so long to do the relatively mundane task of porting EnigmaCurry to the new platform.

Blogofile is best described as a Blog Compiler -- just as a programmer's compiler takes source code and spits out a running program, a blog compiler takes a special blog source code language and spits out raw HTML, the native format of all blogs everywhere. What this means is that any webserver can host the Blogofile based blog, without having to install any special software, not even Blogofile, it's just HTML. You can run Blogofile on any computer that has Python installed, upload the HTML it generates to your webserver, and your webserver remains lean and fast.

You can read more of the great features of Blogofile on it's own site.

One of the biggest motivators in my finishing up moving EnigmaCurry.com over to Blogofile is because, quite frankly, MySQL 5.x blows chunks. It consumes ridiculous amounts of RAM for my pitifully small wordpress database, and when it eventually overwhelms my VPS and forces a restart, the tables are corrupted, especially the mysql user table which requires running "mysqld --skip-grant-tables" in order to get running again just so I can get in to repair the damage. I've seen this happen on two different (but pretty much stock configured) Ubuntu 9.04 servers, so I know it's not an isolated case. I've pledged to myself to never use MySQL in future projects. Do yourself a favor:

sudo apt-get remove mysql-server-5.0

Please don't hesitate to let me know if you see something awry with the site, there's bound to still be a few things missing/corrupted from the transfer.

Ambient Composition 1

I've always wished that I was a musician, I love music. When I was young, my parents even paid for piano lessons, although I gave that up shortly after. Back then I saw it as tedious, and I never saw any real improvement.

I'm hoping to change direction. I seem to always have a tune inside my head and it needs to get out!

I recently bought a new computer powerful enough to run some pretty amazing software synthesizers, so I've started composing some music. Here's a pretty basic one I wrote today:

It's pretty ambient. This is for two reasons:

1. I actually like ambient music, eg. Harold Budd or The Dead Texan.
2. It's got a low barrier to entry, nice long whole notes. :)

Distributing Jython Apps in a Single JAR file

I've been writing a lot of one-off type applications at work lately, which is always kind of a joy for me because these sorts of applications don't need to be maintained or supported in any way, which means I can write them however I want.

So I write them in Python :)

Jython allows me to interface with all the plethora of Java libraries that we use at work, and it lets me program in a language that not only I can tolerate, but one that I respect and love.

But even though these are one-off applications, they still need to be easy to use, and in some cases I won't even be the one running the application. I want these applications to just work damnit.

So, packaging my Jython application into a single executable jar file that contains all of the third party dependancies inside is my goal. I want to send the user the jar file, have them double click on it, and have it immediately start running. It can't get a whole lot easier than that.

The Jython wiki has a page about doing something along these lines. The recipe there called the Jar Method works quite well. The one drawback that it has is that all of the Java dependancies need to be exploded into the main jar root, which when you're dealing with dozens of jar dependancies, it can start to get tedious, messy, and in some cases will even violate the license of a particular library.

One-Jar is a special class loader that can load a jar file that is inside of another jar file, something that the regular class loader from Sun is incapable of doing. Using One-Jar lets my application reside inside of a jar file and contain all my dependancies as seperate jar files inside the main jar file.

I've created a sample project that shows how I normally create a new Jython project hosted inside a single jar file with One-Jar. You'll need the following tools to check out the project:

• A Java SDK (tested with OpenJDK 1.6)
• Apache Ant (tested with version 1.7)
• Git (to checkout the project)

Check out the project like so:

 git clone git://github.com/EnigmaCurry/Single-JAR-Jython-Example.git

Build the project:

cd single-jar-jython-example

ant

Run the example by double clicking it or via the command line:

 java -jar JythonExcelExample.jar

This is just a demonstration app, it doesn't do a whole lot, it outputs an excel file in the current directory listing some computer parts. The point of the application is to show how Jython can integrate with existing Java third-party libraries (in this case Apache POI.)

Instructions for basing your own application on this example are contained inside the README.txt file.

Jython Shell Server -- Adding a REPL to any Java app

I love programming in Python. I get paid to write Java though. Due to Java's verboseness, and lack of a REPL, this can be very frustrating for me.

In Python, the usual way I explore a new library or mock up a new idea is to immediately start coding in Python's interactive interpreter (or REPL). This is often times more efficient than reading (let alone finding) the documentation for the library. I can quickly see if something is going to work before I code inside my larger application.

Java doesn't have an interactive interpreter.. but Jython does! However, setting up Jython, especially interfacing with an already large Java application, can be difficult. One such difficult situation I deal with at work is in Weblogic. With Weblogic, I deploy my application directly to a running Weblogic server, and I never get to see a console in this process, so how am I ever going to run a Jython interactive interpreter?

This morning I got bored and wrote up a quick solution: JythonShellServer. JythonShellServer embeds into any Java application and starts a Telnet server that serves up Jython interactive shells. You can push any Java object that you want to manipulate into Jython's local environment. Run "telnet localhost 7000" and you can use Python code to explore your entire application's running environment.

JythonShellServer works, but I only just wrote it this morning, so consider it alpha quality at the moment. Check out the project page on github.

AutoComplete.el : Python Code Completion in Emacs

A friend of mine and I like to show off to each other little Emacs tips and tricks we learn. Today, he introduced to me the very cool AutoComplete.el package. AutoComplete.el is intriguing to me because, unlike ropemacs mode which I've blogged about before, the completions AutoComplete.el provides are inline with your code in a dropdown box instead of in a seperate window (windows in Emacs are what most people call frames). However, AutoComplete is only a completion framework, it doesn't know anything about Python. Instead, it allows the user to plug into it, feeding it whatever sorts of intelligent completion you want, including Rope.

Setup

The two most important completions I want to integrate into it are Rope and yasnippet. AutoComplete can handle both of them nicely. You'll need to install the very latest development version (as of December '08) of Rope, Ropemacs and Ropemode:

sudo apt-get install mercurial
mkdir /tmp/rope && cd /tmp/rope
hg clone http://bitbucket.org/agr/rope
hg clone http://bitbucket.org/agr/ropemacs
hg clone http://bitbucket.org/agr/ropemode
sudo easy_install rope
ln -s ../ropemode/ropemode ropemacs/
sudo easy_install ropemacs

You'll also need to install Pymacs and Yasnippet if you haven't already:

mkdir -p ~/.emacs.d/vendor && cd ~/.emacs.d/vendor
wget http://pymacs.progiciels-bpi.ca/archives/Pymacs-0.23.tar.gz
tar xfv Pymacs-0.23.tar.gz
cd Pymacs-0.23
make
sudo easy_install .
cd ~/.emacs.d/vendor
wget http://yasnippet.googlecode.com/files/yasnippet-0.5.9.tar.bz2
tar xfv yasnippet-0.5.9.tar.bz2
cd ~/.emacs.d
ln -s vendor/yasnippet-0.5.9/snippets/ .

Make sure Pymacs and Yasnippet get into your load path, in your .emacs:

(add-to-list 'load-path "~/.emacs.d/vendor")
(progn (cd "~/.emacs.d/vendor")
       (normal-top-level-add-subdirs-to-load-path))

Install AutoComplete.el 0.1.0 (0.2.0 is now out, and is not compatible with this post, I'll try and update later):

cd ~/.emacs.d/vendor
wget http://www.emacswiki.org/emacs/?action=browse;id=auto-complete.el;raw=1;revision=5

Now to add some more elisp to your .emacs somewhere:

(require 'python)
(require 'auto-complete)
(require 'yasnippet)

(autoload 'python-mode "python-mode" "Python Mode." t)
(add-to-list 'auto-mode-alist '("\\.py\\'" . python-mode))
(add-to-list 'interpreter-mode-alist '("python" . python-mode))

;; Initialize Pymacs                                                                                           
(autoload 'pymacs-apply "pymacs")
(autoload 'pymacs-call "pymacs")
(autoload 'pymacs-eval "pymacs" nil t)
(autoload 'pymacs-exec "pymacs" nil t)
(autoload 'pymacs-load "pymacs" nil t)
;; Initialize Rope                                                                                             
(pymacs-load "ropemacs" "rope-")
(setq ropemacs-enable-autoimport t)

;; Initialize Yasnippet                                                                                        
;Don't map TAB to yasnippet                                                                                    
;In fact, set it to something we'll never use because                                                          
;we'll only ever trigger it indirectly.                                                                        
(setq yas/trigger-key (kbd "C-c <kp-multiply>"))
(yas/initialize)
(yas/load-directory "~/.emacs.d/snippets")



;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                         
;;; Auto-completion                                                                                            
;;;  Integrates:                                                                                               
;;;   1) Rope                                                                                                  
;;;   2) Yasnippet                                                                                             
;;;   all with AutoComplete.el                                                                                 
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                         
(defun prefix-list-elements (list prefix)
  (let (value)
    (nreverse
     (dolist (element list value)
      (setq value (cons (format "%s%s" prefix element) value))))))
(defvar ac-source-rope
  '((candidates
     . (lambda ()
         (prefix-list-elements (rope-completions) ac-target))))
  "Source for Rope")
(defun ac-python-find ()
  "Python `ac-find-function'."
  (require 'thingatpt)
  (let ((symbol (car-safe (bounds-of-thing-at-point 'symbol))))
    (if (null symbol)
        (if (string= "." (buffer-substring (- (point) 1) (point)))
            (point)
          nil)
      symbol)))
(defun ac-python-candidate ()
  "Python `ac-candidates-function'"
  (let (candidates)
    (dolist (source ac-sources)
      (if (symbolp source)
          (setq source (symbol-value source)))
      (let* ((ac-limit (or (cdr-safe (assq 'limit source)) ac-limit))
             (requires (cdr-safe (assq 'requires source)))
             cand)
        (if (or (null requires)
                (>= (length ac-target) requires))
            (setq cand
                  (delq nil
                        (mapcar (lambda (candidate)
                                  (propertize candidate 'source source))
                                (funcall (cdr (assq 'candidates source)))))))
        (if (and (> ac-limit 1)
                 (> (length cand) ac-limit))
            (setcdr (nthcdr (1- ac-limit) cand) nil))
        (setq candidates (append candidates cand))))
    (delete-dups candidates)))
(add-hook 'python-mode-hook
          (lambda ()
                 (auto-complete-mode 1)
                 (set (make-local-variable 'ac-sources)
                      (append ac-sources '(ac-source-rope) '(ac-source-yasnippet)))
                 (set (make-local-variable 'ac-find-function) 'ac-python-find)
                 (set (make-local-variable 'ac-candidate-function) 'ac-python-candidate)
                 (set (make-local-variable 'ac-auto-start) nil)))

;;Ryan's python specific tab completion                                                                        
(defun ryan-python-tab ()
  ; Try the following:                                                                                         
  ; 1) Do a yasnippet expansion                                                                                
  ; 2) Do a Rope code completion                                                                               
  ; 3) Do an indent                                                                                            
  (interactive)
  (if (eql (ac-start) 0)
      (indent-for-tab-command)))

(defadvice ac-start (before advice-turn-on-auto-start activate)
  (set (make-local-variable 'ac-auto-start) t))
(defadvice ac-cleanup (after advice-turn-off-auto-start activate)
  (set (make-local-variable 'ac-auto-start) nil))

(define-key python-mode-map "\t" 'ryan-python-tab)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                         
;;; End Auto Completion                                                                                        
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Github

These changes are also applied to my Emacs environment on Github. I'm continuously trying to improve my emacs setup, so if you're reading this a few months/years after this post was made, you may want to check there for improvements.

Update Jan 30 2009: I made some modifications to the tab complete order, a regular indent is applied first before autocompletion. Autocompletion is also not applied unless you are at the end of a word. This is useful when you press Tab at the beginning of a line to indent/outdent.

Usage

Once you got everything setup, usage is real easy:

1. Open up a python (.py) file
2. Press TAB when you want to use code completion or to insert a snippet.

The first time you attempt to use code completion you'll be prompted to enter the root of your project directory.

Future

It wasn't too long ago when Python code completion inside Emacs was just a pipedream. So now that we have it, let's make some more pipedreams: I'd like to popup some nice contextual help for method arguments as well as python documentation for the current method/class being completed.

The possibilities are pretty much endless. I love Emacs.