GNU Emacs on Free Range Bits

Automatic Theme Switching in Emacs

Mon, 24 Feb 2025 12:23:47 +0100

I’m a dark theme nerd. Partially because of the hacker aesthetic, but mostly because I prefer to work in dark environments and light themes turn computer screens into floodlights.

However, in well lit rooms (or when working outdoors) dark themes become completely unreadable. Such is the case in my office during the morning hours. Naturally, I figured out how to change the system theme from the command line, immediately updating running GTK and Qt applications. The next step was getting Emacs to respond to the same event.

Theme Switching and D-Bus

Linux and other Unix-like operating systems use D-Bus for interprocess communication. Applications can use D-Bus to register methods and properties which can be accessed and monitored from any D-Bus client. For example, to find out whether the system is currently using a light or dark theme, call the org.freedesktop.portal.Settings.Read method from the command line using the dbus-send tool:

dbus-send --session --print-reply \
  --dest=org.freedesktop.portal.Desktop \
  /org/freedesktop/portal/desktop \
  org.freedesktop.portal.Settings.Read \
  string:org.freedesktop.appearance string:color-scheme

This command will print 1 when the system is using a dark theme and 2 when using a light theme (plus some other stuff you can ignore).

With that information in hand we need to get Emacs to do two things:

Call this method when starting up and set its theme to match the system, and
Monitor the color-scheme property and change its theme automatically when the system theme changes.

Emacs Groundwork

In order to get Emacs to change its theme we’ll need a way to store our preferred dark and light themes, and a way to change the current theme based on the results of a D-Bus call:

(defvar pjones:dark-theme 'ef-maris-dark
  "Default dark theme.")

(defvar pjones:light-theme 'ef-maris-light
  "Default light theme.")

(defun pjones:theme-from-dbus (value)
  "Change the theme based on a D-Bus property.

VALUE should be an integer or an arbitrarily nested list that
contains an integer.  When VALUE is equal to 2 then a light theme
will be selected, otherwise a dark theme will be selected."
  (load-theme (if (= 2 (car (flatten-list value)))
                  pjones:light-theme
                pjones:dark-theme)
              t))

You might be wondering what flatten-list is for. It turns out that D-Bus gives two different types of answers depending on how you requested a property. When invoking a D-Bus method, the result comes back to Emacs as a list. For example, when requesting the current color scheme using the Read method the response will be (1) or (2) depending on whether the current system theme is dark or light.

However, when the property changes and D-Bus notifies clients that are listening for change events, Emacs sees the property as a nested list. In this case the color scheme value will be ((1)) or ((2)). The call to flatten-list ensures that both of these situations look the same to the callback function (pjones:theme-from-dbus).

The only thing left is to convince Emacs to set its theme using information from D-Bus.

Using D-Bus from within Emacs

It should come as no surprise that Emacs ships with its own D-Bus library. To find out what the current color scheme is we can call the Read method using dbus-call-method-asynchronously. And then to be notified when the system theme changes we can use dbus-register-signal to be notified when the SettingChanged event is emitted. The arguments to these functions are a bit crazy but they mirror those given to the dbus-send tool.

(require 'dbus)

;; Set the current theme based on what the system theme is right now:
(dbus-call-method-asynchronously
   :session "org.freedesktop.portal.Desktop"
   "/org/freedesktop/portal/desktop"
   "org.freedesktop.portal.Settings"
   "Read"
   #'pjones:theme-from-dbus
   "org.freedesktop.appearance"
   "color-scheme")

;; Register to be notified when the system theme changes:
(dbus-register-signal
   :session "org.freedesktop.portal.Desktop"
   "/org/freedesktop/portal/desktop"
   "org.freedesktop.portal.Settings"
   "SettingChanged"
   (lambda (path var value)
     (when (and (string-equal path "org.freedesktop.appearance")
                (string-equal var "color-scheme"))
       (pjones:theme-from-dbus value))))

That’s it. Switching the system color scheme should now cause Emacs to update its theme too. And when Emacs starts it should automatically set its theme to match the system theme.

Command Line Theme Switching

I mentioned earlier that I figured out how to change the system theme using the command line. The most reliable way that I have found so far is to use gsettings, which also emits the SettingChanged event via D-Bus:

gsettings set org.gnome.desktop.interface color-scheme prefer-dark
gsettings set org.gnome.desktop.interface gtk-theme Adwaita-dark

# or

gsettings set org.gnome.desktop.interface color-scheme prefer-light
gsettings set org.gnome.desktop.interface gtk-theme Adwaita

The second invocation of gsettings (changing the GTK theme) isn’t strictly necessary. However, I configured Qt to use the GTK theme and this is the only way I could get Qt applications to also change their color scheme as they don’t appear to respect the color-scheme setting.

I also wrapped these commands up in XDG desktop entries so I could invoke them from tools like Rofi.

Appendix: GTK and Qt Themes

It probably matters how I configured GTK and Qt, although these don’t have anything to do with Emacs. Basically, GTK is set to use the Adwaita-dark theme, and Qt is configured to use its gtk3 engine which should follow the current GTK theme. This works for almost all Qt applications, although some refuse to update to the light theme.

All of my GTK/Qt settings are done using Home Manager in this file.

Patching Instead of Pinning

Sat, 06 Jan 2024 11:44:19 -0700

In a previous post I showed how to work around a deadlock between GnuPG and Emacs. As a brief recap, GnuPG 2.4.1 introduced a change in its output which breaks a protocol that Emacs relied on, so I pinned GnuPG to version 2.4.0 on my system to avoid the bug.

Damien Cassou reached out to me and expressed a preferred way of dealing with this issue: patch the current version of GnuPG so it doesn’t contain the bug. This means we get to keep any bug fixes and security updates in the most recent version without having to suffer with the deadlock bug.

I’m going to demonstrate how to apply a patch to GnuPG in nixpkgs, but before that let’s review the Nix code for pinning GnuPG to version 2.4.0.

Pinning the GnuPG Package

In the last post we looked at three different options for pinning GnuPG:

A nixpkgs overlay to replace GnuPG globally
Replace the version of GnuPG that shows up in the system PATH via the NixOS module
Replace the version of GnuPG that shows up in the user environment via the Home Manager module

My preference was for the last option, using Home Manager:

{ pkgs }:

{
  programs.gpg.package = pkgs.gnupg.overrideAttrs (orig: {
    version = "2.4.0";
    src = pkgs.fetchurl {
      url = "mirror://gnupg/gnupg/gnupg-2.4.0.tar.bz2";
      hash = "sha256-HXkVjdAdmSQx3S4/rLif2slxJ/iXhOosthDGAPsMFIM=";
    };
  });
}

However, I didn’t really explain what was going on in that Nix code. I’d like to give that a try now.

Overriding Nix Packages

One way to think about a package written in Nix is as a set of attributes. (Of course, this isn’t true, but it’s a useful analogy and allows us to continue without having to talk about lazy evaluation and fixpoints.) These attribute sets are immutable so we can’t change them. But what we can do is build new packages from existing ones.

This is done with the overrideAttrs function, which allows us to create a new package by replacing attributes in an existing package. The argument to overrideAttrs is a function that is given the existing attributes and should return replacement attributes.

In the Nix code above that pins GnuPG to version 2.4.0, I replaced the version and src attributes. The result is a new package based on the existing GnuPG 2.4.1 package, except that the upstream source code is now coming from the GnuPG 2.4.0 tarball.

This won’t always work, but it does in this case because GnuPG 2.4.0 and 2.4.1 are so similar. If I tried to update the GnuPG package by replacing its source code with that of the Linux kernel things wouldn’t turn out so nicely.

The most interesting thing about overrideAttrs is that we can replace any package attribute we want. I’ve used this to great effect over the years: adding shell commands to the end of install scripts, changing compiler flags, installing files that were missed by make install, to name a few.

And of course, we can use this to apply patches to the source code before building.

Patching the Package Instead

Nix packages have an optional patches attribute which is a list of patch files to apply to the source code before building. And thanks to overrideAttrs we can add patches to that list:

final: prev:

{
  gnupg_plus_960877b = prev.gnupg.overrideAttrs (orig: {
    patches = (orig.patches or [ ]) ++ [
      (prev.fetchurl {
        url = "https://github.com/gpg/gnupg/commit/960877b10f42ba664af4fb29130a3ba48141e64a.diff";
        sha256 = "0pa7rvy9i9w16njxdg6ly5nw3zwy0shv0v23l1mmi0b7jy7ldpvf";
      })
    ];
  });
}

This example is an overlay and is straight out of my Nix configuration. It creates a new package (gnupg_plus_960877b) from the exiting gnupg package by replacing the patches attribute with one that includes the fix for the deadlock bug.

I ensure that I’m including patches from the package I’m modifying by using the original patches list (orig.patches), or if that’s missing, an empty list. Then I append ("++") a list with one element which uses fetchurl to get the patch I want.

The only thing left to do is tell Home Manager to use this new package:

{ pkgs }:

{
  programs.gpg.package = pkgs.gnupg_plus_960877b;
}

Conclusion

Nix and nixpkgs make it very easy to alter an existing package thanks to the overrideAttrs function. Using nixpkgs overlays we can even replace a package globally throughout the package set.

Sometimes we may want to alter a package’s src attribute to downgrade (or upgrade) it to another version. In this case we took Damien Cassou’s advice and applied a patch to the exiting GnuPG package.

Update (February 23, 2024): GnuPG 2.4.4 has been released and it contains the patch mentioned in this post. In other words, GnuPG 2.4.4 and Emacs work nicely together once again.

Archiving in Org Mode

Thu, 04 Jan 2024 12:45:09 -0700

As some of you may know, I practically live inside Emacs. And like a lot of Emacs users a good portion of that time is spend in org-mode, a package that is hard to describe due to its overwhelming large number of features.

One of the many ways I use org-mode is to implement a Getting Things Done (GTD) workflow. That is, managing all of the projects that I’m trying to push forward as well as all of the routine recurring tasks that I have to deal with on a daily basis.

The issue that every org-mode user eventually has to wrestle with is what to do with all those completed projects and obsolete tasks. For some projects the obvious answer is to simply delete them. This is especially true if you’re using a version control system to track your org-mode files. But if you regularly need to refer to completed projects/tasks (i.e. “what was that crazy query I ran last time?”) then archiving is a better option.

Archiving Options in org-mode

In org-mode, archiving is the process of moving a subtree somewhere else when it’s no longer needed¹. The destination for an archived subtree is taken from the org-archive-location variable.

By default, org-mode will move the subtree to another file whose name is derived from the name of the current file by appending “_archive” to the name, but before the extension. That said, like everything in Emacs, the options for configuring the archive destination are virtually limitless.

A really common way to configure archiving is to use a dedicated file, usually archive.org. Whether you use a single archive file or the default strategy of using a file named after the current one, you will ultimately run into the same problem, huge unwieldy org-mode files.

I think there’s a better way to archive projects and tasks: use a lot of small files. Specifically, when I archive a subtree in org-mode I send the subtree to the current day’s journal entry.

Using the org-roam Journal

I maintain my personal knowledge base in org-mode thanks to org-roam. One of its features is the ability to manage journal entries. The journal system in org-roam is simply a collection of subtrees in a one-file-per-day organization scheme. In other words, each day gets a new org-mode file and throughout the day I add headings to the file in order to record certain events.

When I archive a subtree from somewhere in my vast collection of notes, org-mode stores it in one of my journal files. The specific journal file used is based on the date chosen from a prompt when I run the following archiving function:

(defun pjones:org-archive-subtree-to-daily (&optional _find-done)
  "Arhive the current subtree to the roam daily file."
  (interactive)
  (require 'org-roam)
  (when-let* ((today (save-excursion
                       (org-roam-dailies-goto-date nil "d")
                       (buffer-file-name)))
              (org-archive-location
               (concat today "::* Archived From %s")))
    (org-archive-subtree 0)))

You can tell org-mode that you want this to be your archiving function by setting org-archive-default-command:

(custom-set-variables
  '(org-archive-default-command #'pjones:org-archive-subtree-to-daily))

After which you can archive the current subtree with the normal key binding “C-c C-x C-a”. This leaves “C-c C-x C-s” alone if you want to use the built in system which relies on org-archive-location.

(Note: I have both daily and monthly journal files. To avoid being asked which one I want to archive to I pass “d” as the second argument to org-roam-dailies-goto-date. This bypasses the prompt and goes directly to the daily journal file.)

Conclusion

I find it easier to manage many small files verses a small number of huge files. To avoid a massive archive.org file I archive subtrees into my org-roam journal. This makes it a lot easier to find what I’m looking for and provides a bit of context when found. And it pairs perfectly with rg-project and consult-ripgrep.

You can also tag the current subtree with the ARCHIVE tag which darkens the heading and keeps the tree collapsed. This can be done with the org-toggle-archive-tag function or the “C-c C-x a” key binding. ↩︎

Pinning GnuPG with nixpkgs

Sun, 31 Dec 2023 10:49:30 -0700

GnuPG 2.4.1 was released all the way back in May but I failed to notice until I upgraded to NixOS 23.11 a few weeks ago. And I only noticed because Emacs froze when I tried to save an encrypted file. It didn’t take me long to realize it was due to a change in GnuPG.

Emacs and Encrypted Files

When working with encrypted files Emacs goes to great lengths to ensure the unencrypted cleartext is never written to the file system. It does this by invoking GnuPG directly and communicating with it using pipes. Thus, to write to an encrypted file, Emacs sends the cleartext down the pipe and GnuPG writes the encrypted contents to a file.

Just before Emacs sends the cleartext down the pipe to GnuPG it waits for GnuPG to signal that it’s ready to read cleartext. The problem with GnuPG (versions 2.4.1 through 2.4.3) is that it sends this signal after it reads from standard input. In other words, both Emacs and GnuPG are waiting for one another at the same time, resulting in a classic I/O deadlock.

The Workaround

The Emacs maintainers consider this situation a bug in GnuPG. And after a few days the GnuPG maintainers fixed the bug. It will be part of GnuPG 2.4.4 release. Until then the solution is to downgrade GnuPG. The Emacs PROBLEMS file says:

The only known workaround is to downgrade to a version of GnuPG older than 2.4.1 (or, in the future, upgrade to a newer version which solves the problem, when such a fixed version becomes available).

Pinning Packages in nixpkgs

This presents the perfect opportunity to demonstrate several ways that nixpkgs ecosystem allows us to solve the problem of downgrading a package, sometimes called package pinning.

The easiest solution is to use an overlay, a function that takes the current package set and returns a new package set. The overlay below replaces the gnupg package with one pinned at version 2.4.0:

final: prev:

{
  gnupg = prev.gnupg.overrideAttrs (orig: {
    version = "2.4.0";
    src = prev.fetchurl {
      url = "mirror://gnupg/gnupg/gnupg-2.4.0.tar.bz2";
      hash = "sha256-HXkVjdAdmSQx3S4/rLif2slxJ/iXhOosthDGAPsMFIM=";
    };
  });
}

Notice that the overlay function actually takes two arguments. The second argument (prev) is the previous package set, the one we want to alter. Since overlays can be chained together we may be manipulating the original package set or one from a previous overlay.

Sometimes we need access to the package set the way it will be after all overlays have been applied. For example, wanting to use a dependency from a later overlay in the current overlay. In this case the first argument (final) comes in handy, it’s the final package set after all overlays have been applied. (This is possible thanks to Nix being a lazy language and the use of fixpoints.)

Livin’ La Vida Gentoo

Unfortunately, using an overlay to pin GnuPG comes at a cost. It turns out that GnuPG is a dependency in a lot of packages. This means that after applying the overlay, nixpkgs will determine that all of those packages need to be rebuilt, and we won’t be able to take advantage of the binary cache (because the cache was built with GnuPG 2.4.1).

I don’t have the patience of a Gentoo user and compiling a ton of packages from source isn’t my idea of a good time. Besides, it’s a terrible use of the power grid. Thankfully, we have other options.

The NixOS Module

If you use NixOS then the easiest way to install and configure GnuPG is via the NixOS module:

{
  programs.gnupg.enable = true;
}

The NixOS module also has a package option so users can override the GnuPG package. Taking the overlay code from above we can write our own NixOS module and load it into our system configuration (via the normal imports list mechanism):

{ pkgs }:

{
  programs.gnupg.package = pkgs.gnupg.overrideAttrs (orig: {
    version = "2.4.0";
    src = pkgs.fetchurl {
      url = "mirror://gnupg/gnupg/gnupg-2.4.0.tar.bz2";
      hash = "sha256-HXkVjdAdmSQx3S4/rLif2slxJ/iXhOosthDGAPsMFIM=";
    };
  });
}

Similar to the overlay method, pinning GnuPG through the NixOS module affects the version of GnuPG in the system PATH. But unlike with overlays, other packages still see the original version of GnuPG. Since we’re not infecting other packages with a changed GnuPG, nothing needs to be recompiled.

Now, I’m a very happy user of NixOS, but this isn’t my preferred solution. And, of course, it doesn’t work at all for non-NixOS users. The last way we’ll look at pinning GnuPG works for any nixpkgs user and also doesn’t suffer from the overhead of the overlay technique.

The Home Manager Module

I’m a huge fan of Home Manager. It allows nixpkgs users take all the awesomeness of NixOS to other Linux distributions and even to closed operating systems like macOS, Android, and even Windows.

The Home Manager module for GnuPG is very similar to the NixOS module and also includes a package option. That means we can pin GnuPG in our user environment similar to how the NixOS module allowed us to pin it at the system level. The following Nix code can be loaded into your Home Manager configuration through the normal imports list mechanism, just like with NixOS modules:

{ pkgs }:

{
  programs.gpg.package = pkgs.gnupg.overrideAttrs (orig: {
    version = "2.4.0";
    src = pkgs.fetchurl {
      url = "mirror://gnupg/gnupg/gnupg-2.4.0.tar.bz2";
      hash = "sha256-HXkVjdAdmSQx3S4/rLif2slxJ/iXhOosthDGAPsMFIM=";
    };
  });
}

A Warning About Other Workarounds

It’s important to point out that the correct solution to this problem between Emacs and GnuPG is to fix GnuPG.

I’ve seen people on social media sites like Reddit suggesting a hack to Emacs to resolve the problem. Specifically, people are putting the following in their Emacs configuration:

(fset 'epg-wait-for-status 'ignore)

Don’t do this! This is not a valid fix and may lead to file corruption when reading/writing encrypted files.

This bit of hackery replaces the epg-wait-for-status function with one that ignores its arguments and always returns nil. That function is used throughout the EPG library in Emacs, not just when writing files through GnuPG. Again, please don’t do this.

Conclusion

We’ve looked at three different ways of pinning a package in nixpkgs:

Using an overlay function.

This is usually my go-to solution. But in this case GnuPG is used in so many other packages that it caused massive rebuilds.
Pinning via the NixOS programs.gnupg.package option.

This works by only changing the version of GnuPG installed in the system PATH. If you use NixOS (and not Home Manager) this is the way to go.
Pinning via the Home Manager programs.gpg.package option.

This is my preferred solution because it works nearly everywhere thanks to Home Manager. It replaces the version of GnuPG in your user PATH, leaving the system PATH and the rest of nixpkgs alone.

Like many others, my frustration with GnuPG is nearing its breaking point. Over the last 20 years I’ve sent/received no more than a handful of encrypted emails. My use of GnuPG at this point is limited to encrypting files in my password store and signing Git commits. It’s becoming less clear why I put up with the burden of managing my GnuPG key and the occasional upgrade problem.

Perhaps I’ll finally switch to age over the next major holiday break. (I probably won’t find the time, but here’s hoping for the best.)

Update (January 6, 2024): A better solution may be to patch the current version of GnuPG so it doesn’t have the deadlock bug. I explore that option in this post.

Update (February 23, 2024): GnuPG 2.4.4 has been released and fixes the problems mentioned in this post.

Don't be affraid to learn vi or emacs

Wed, 23 Nov 2011 00:00:00 -0700

Over at the thoughtbot blog there’s an article about the myth of the vim learning curve. I think everything said about vi/vim can equally be applied to Emacs. The path to getting up and running isn’t as daunting as you might think:

Go to your shell and type vimtutor. The tutorial that’s presented is excellent and you’ll be through it in no time. Once you’re done, you’ll have the rudiments needed to get your work done. You won’t be fast yet, no; but you’ll be competent.

There’s a similar feature in Emacs. To start the tutorial just type C-h t (that’s Control-h and then t).

Fun stuff for Friday Nov. 18

Fri, 18 Nov 2011 00:00:00 -0700

This past week has been a blur for me so I’m totally ready for the weekend. The kids and I have been making our way through the Alien films and tonight we’re on number 3. I don’t have fond memories of the movie but I’m sure we’ll have fun.

Hopefully you have some fun planned yourself and the links below will help you kick off the weekend.

Anders Evenrud has released what he’s calling OS.js, or what I’d call an operating system emulator. It almost looks like you VNC into his Linux box, but nope, it’s all JS. It even has apps written for it.
Emacs users who want to start using version 24 (to be released this spring) should take a look at Bozhidar Batsov’s Emacs Prelude. It’s a set of configuration files and packages sort of like the Emacs starter kit.
Itching to write code nobody can read? Back after a rather long hiatus is the International Obfuscated C Code Contest!
Looking for a geeky way to put your keys away for the night? Why not build an RJ45 key chain. I recommend wiring it so you can plug it right into a hub or switch.
Videos: Astronauts tripping and falling on the moon (via GeekDad) and Simultaneous time lapse video of the sky over 360 days (via kottke).

Have a fun weekend!

Remote pair programming in a terminal

Mon, 14 Nov 2011 00:00:00 -0700

Pair programming is becoming more popular due to the good old terminal. Instead of having to sit next to your coworker who had an onion sandwich for lunch you can use terminal sharing from across the office or the country. And with services like Ruby Pair that connect people who want to pair program, knowing how to use a terminal multiplexer is worth a few minutes of your time.

The venerable choice for a terminal multiplexer is GNU Screen. Dalibor Nasevic has written an article with instructions on how to configure SSH and Screen to allow remote pair programming. In my opinion, a huge downside to using Screen is having to setuid the process to enable multiuser mode. A much better option is tmux.

If you’re looking for a more modern terminal multiplexer that is much easier to configure, tmux is the answer. Joe Moore has an article about using tmux with Vim for pair programming, but it’s really just a few simple commands:

# Host
tmux -S /tmp/pairprog

# Client
ssh pair-host
tmux -S /tmp/pairprog attach

Remote pair programming with Screen or tmux is also a great excuse to polish your Vim or Emacs skills.

Write HTML/XML faster with Zen Coding

Thu, 20 Oct 2011 00:00:00 -0700

If you find yourself writing a lot of opening and closing tags by hand you really should check out Zen Coding:

Zen Coding is an editor plugin for high-speed HTML, XML, XSL (or any other structured code format) coding and editing. The core of this plugin is a powerful abbreviation engine which allows you to expand expressions–similar to CSS selectors–into HTML code.

A very large variety of text editors are supported by the plug-in, with alternative implementations for editors like Emacs and Vim.

Challenge your text editor fu

Mon, 17 Oct 2011 00:00:00 -0700

Ilya Grigorik has put together an excellent site called VimGolf where developers host challenges to see who can complete a task in Vim with the fewest keystrokes. Here’s an example challenge:

Reformat long lines. Rearrange this ruby method call to put each parameter on its own line. Could become a useful macro.

You are given an input file and an output file with the goal to provide the minimum number of keystrokes to convert the input file to the output file.

Vim not your cup of tea? Tim Visher goes through the VimGolf challenges with Emacs and posts the results on Vimeo. His videos are great to watch if you want to improve your Emacs skills.

Behind the code: project planning

Thu, 24 Jul 2008 00:00:00 -0700

In this new series of articles, Behind the Code, I’ll be looking at the tools and processes that I’ve put into place to make me a more efficient and effective developer. I’m hoping to share my experiences, and while doing that, refine the way I work using your feedback.

Today, I’ll be talking about project planning, the process of breaking a project into tasks, and estimating the effort to complete those tasks. The artifacts generated during this process can then be used to create a proposal for a client, cost estimations for your boss, or even the start of your project documentation.

File Formats and Tools Matter

If there’s one thing I’ve definitely learned the hard way, it’s that the tools I use today aren’t necessarily the tools I’ll be using tomorrow. A better tool might come along, or I might be forced to use another operating system where my preferred tools aren’t available.

I avoid binary proprietary file formats like used car salesmen. It’s so important to me that it’s my first rule:

Rule 1: Use plain text files when possible.

No matter which platform you’re using, you can read and work with text files. If you happen to use a platform with good command line tools (anything Unix based), you have a powerful system available for finding, searching, and transforming text files.

Text files rock, and what’s the best tool to edit them? Being a developer, using a text editor shouldn’t be unfamiliar to you. As a matter of fact, it should be the tool that you’re most comfortable and efficient with. If there’s only one thing you can call yourself an expert in, it should be in using your text editor.

Tools for Project Planning

There are several tools you can use to do project planning, all while working with nothing more than plain text files. My tool of choice is Emacs, using the amazing Org-Mode package.

The tools that you pick will depend completely on you. In this case, the process is slightly more important than the tools. Don’t get me wrong though, a good tool can make a big difference in your workflow.

I like Org-Mode because it provides a lot of functionally, yet at the end of the day everything is just plain text. I can easily keep my project files in a revision control system, and use tools like Ruby and grep.

Breaking a Project Into Features and Tasks

The first step for planning a project is breaking it into smaller pieces. I like to create a shallow hierarchy, where I break a project into features, and then take those features and break them down into tasks.

Features are usually the bigger chunks that your client or boss would know by name. Since features are big chucks, you can’t estimate them correctly, or think clearly about their dependencies without breaking them down further.

By breaking a feature down into tasks, you virtually walk though the implementation of that feature, thinking about dependencies and potential problems that you’ll have along the way.

While breaking a project into features and tasks, don’t think about time estimates, only process. Work through the steps you would actually perform to implement a feature in your mind, recording them in your project plan as tasks.

Below is screenshot from Emacs, showing the outline I’ve created for a fictional social networking application. Org-Mode allows me to fold the outline at any level. Here I’m viewing just the outline headings that show two features, broken down into tasks.

While I’m creating the feature and task breakdown, I leave myself little notes just in case the headings are too vague and a few weeks later I’m not sure what the hell I was talking about.

Below is a fully expanded version of the outline from above, so you can see the notes. Later, these notes can be expanded and used as project documentation or a client proposal.

In Org-Mode, the tab key is used to expand or fold an outline. This makes it very easy to see just the part of the project plan that you’re interested in.

Estimating the Tasks

As I mentioned before, a feature by itself is too vague to estimate. But now that you’ve broken each of the features down into tasks, you should have enough information to estimate the effort of actually implementing those features.

I’m not going to dive too deeply into the art of estimating in this article. That said, put your estimating cap on and let’s get to it.

The best way to estimate your tasks is to begin with the first task, and work through them one at a time. I prefer to break my tasks down into hours and minutes, usually in 30-minute increments.

If you come across a task that will take more than a workday to complete, you need to break the task down into more tasks. It’s important that your tasks are granular enough, I personally don’t like my tasks to take more than 5-hours. Along the same lines, any task that takes less than 30-minutes, however, should be reviewed to ensure it has been thought through enough.

Below is a screenshot showing estimates assigned to each task. I like to switch Org-Mode into column view. Org-Mode makes it easy to see summations of task estimates using its handy column view mode, which turns your hierarchical outline into a table.

In this screenshot, I’ve assigned estimates to each task. The features now show the total estimates for their respective tasks, leaving the top headline (the project name) with a total estimate for the project. Org-Mode automatically summed the time estimates for me, I only assigned estimates to the tasks themselves (in purple below).

Presenting the Project Plan

With your task estimates completed, you’ve collected enough information to present a proposal to a client, or a project plan to your boss.

If you’ve been using a software tool to create your project plan, it probably has an export feature. Org-Mode can export your document to HTML, LaTeX, and even a cleaned up ASCII text file.

Even if you’re not using a tool with an export feature, you’re a software developer right? Write a script that will transform your text file to something useful for your situation.

As an example of a client proposal, I’ve exported the project file into a cleaner PDF file for my fictional client. If this were a real proposal, I would add additional notes, along with details about the cost of the implementation.

Using the Project Plan During Implementation

Why do all the work of creating a proposal, only to let it collect dust? When it comes time to implement the features that you created a spec for, why not turn them into a to-do list?

Using the Org-Mode ability to manage to-do lists, I turned the estimated tasks into to-do items. With Org-Mode I can “clock in” to a task, which starts a timer. When a task is complete, I mark it as being done, and then I can compare the time I spent working to my estimates.

Below is a screenshot where I’ve completely expanded a finished task so that you can see the meta-data that Org-Mode was adding to it when I used the timer on a task and then marked it as complete. You can also see how a headline can be turned into a to-do item.

It’s much more useful to switch back into column view for the comparisons between my estimates and the clocked time. The below screenshot shows my outline back in column view, this time also showing clock time summaries.

This is a good example of how Org-Mode can collect the meta-data in your text file and then present it to you in a more efficient way. You can even edit the data in this table-like view, and Org-Mode will do the right thing.

In addition to column view, Org-Mode has several ways to generate reports from your data. I especially like the agenda view.

With the agenda view, you can see which tasks you’ve scheduled for specific days. I find it very useful to spend a little bit of time at the end of my day, and schedule which tasks I’ll be working on for the following day.

In the morning, I can review my agenda and start working with a clear idea of what I should be doing. This really helps with any tendency to stall or procrastinate during the first hours of the day.

Article Artifacts and Further Reading

I chose to show screenshots above so you could get a feel for how a good tool like Org-Mode can provide a nice interface for project planning. The nice thing about Org-Mode is that you’re just working with a text file, and if you open that file in any other text editor, it doesn’t look foreign.

Below are links to files I created while writing this article. Open the project plan in your favorite text editor and compare it to the screenshots above. Try opening it in Emacs and see what happens.

The Plain Text Org-Mode File (Project File)
A PDF Client Proposal

If you’re looking for more information about Org-Mode, the main web site is a great resource. There is a fantastic manual, and a very responsive mailing list if you have any questions.