Developing command-line utilities

Some people around me have noticed that I frequently use the command-line for development and system administration tasks, in particular when I'm working on Linux and other UNIX-like operating systems.

Typically, when you see me working on something behind my computer, you will most likely observe a screen that looks like this:

The above screenshot shows a KDE plasma desktop session in which I have several terminal screens opened running a command-line shell session.

For example, I do most of my file management tasks on the command-line. Moreover, I'm also a happy Midnight Commander user and I use the editor that comes with it (mcedit) quite a lot as well.

As a matter of fact, mcedit is my favorite editor for the majority of my tasks, so you will never see me picking any side in the well-known Emacs vs vi discussion (or maybe I upset people to say that I also happen to know Vim and never got used to Emacs at all). :-).

Using the command-line

For Linux users who happen to do development, this way of working makes (sort of) sense. To most outsiders, however, the above screenshot looks quite menacing and they often question me why this is a good (or appealing) way of working. Some people even advised me to change my working style, because they consider it to be confusing and inefficient.

Although I must admit that it takes a bit of practice to learn a relevant set of commands and get used to it, the reasons for me to stick to such a working style are the following:

• It's a habit. This is obviously not a very strong argument, but I'm quite used to typing commands when executing system administration and development tasks, and they are quite common in a Linux/Unix world -- much of the documentation that you will find online explain how to do things on the command-line.
  
  Moreover, I have been using a command-line interface since the very first moment I was introduced to a computer. For example, my first computer's operating system shell was a BASIC programming language interpreter, which you even had to use to perform simple tasks, such as loading a program from disk/tape and running it.
  
  
• Speed. This argument may sound counter-intuitive to some people, but I can accomplish many tasks quite quickly on the command-line and probably even faster than using a graphical user interface.
  
  For example, when I need to do a file management task, such as removing all backup files (files having a ~ suffix), I can simply run the following shell command:
  
  

  $ rm *~
  

  
  If I would avoid the shell and use a graphical file manager (e.g. Dolphin, GNOME Files, Windows Explorer, Apple Finder), picking these files and moving them to the trashcan would certainly take much more time and effort.
  
  Moreover, shells (such as bash) have many more useful goodies to speed things up, such as TAB-completion, allowing someone to only partially type a command or filename and let the shell complete it by pressing the TAB-key.
  
  
• Convenient integration. Besides running a single command-line instruction to do a specific task, I also often combine various instructions to accomplish something more difficult. For example, the following chain of commands:
  
  

  $ wc -l $(find . -name \*.h -or -name \*.c) | head -n -1 | sort -n -r
  

  
  comes in handy when I want to generate an overview of lines of code per source file implemented in the C programming language and sort them in reverse order (that is the biggest source file first).
  
  As you may notice, I integrate four command-line instructions: the find instruction seeks for all relevant C source files in the current working directory, the wc -l command calculates the lines of code per file, the head command chops off the last line displaying the total values and the sort command does a numeric sort of the output in reverse order.
  
  Integration is done by pipes (using the | operator) which make the output of one process the input of another and command substitution (using the $(...) construct) that substitutes the invocation by its output.
  
  
• Automation. I often find myself repeating common sets of shell instructions to accomplish certain tasks. I can conveniently turn them into shell scripts to make my life easier.

Recommendations for developing command-line utilities

Something that I consider an appealing property of the automation and integration aspects is that people can develop "their own" commands in nearly any programming language (using any kind of technology) in a straight forward way and easily integrate them with other commands.

I have developed many command-line utilities -- some of them have been developed for private use by my past and current employers (I described one of them in my bachelor's thesis for example), and others are publicly available as free and open source software, such as the tools part of the IFF file format experiments project, Disnix, NiJS and the reengineered npm2nix.

Although developing simple shell scripts as command-line utilities is a straight forward task, implementing good quality command-line tools is often more complicated and time consuming. I typically take the following properties in consideration while developing them:

Interface conventions

There are various ways to influence the behaviour of processes invoked from the command-line. The most common ways are through command-line options, environment variables and files.

I expose anything that is configurable as command-line options. I mostly follow the conventions of the underlying OS/platform:

• On Linux and most other UNIX-like systems (e.g. Mac OS X, FreeBSD) I follow GNU's convention for command line parameters.
  
  For each parameter, I always define a long option (e.g. --help) and for the most common parameters also a short option (e.g. -h). I only implement a purely short option interface if the target platform does not support long options, such as classic UNIX-like operating systems.
• On DOS/Windows, I follow the DOS convention. That is: command line options are single character only and prefixed by a slash, e.g. /h.

I always implement two command-line options, namely a help option displaying a help page that summarizes how the command can be used and a version option displaying the name and version of the package, and optionally a copyright + license statement.

I typically reserve non-option parameters to file names/paths only.

If certain options are crosscutting among multiple command line tools, I also make them configurable through environment variables in addition to command-line parameters.

For example, Disnix exposes each deployment activity (e.g. building a system, distributing and activating it) as a separate command-line tool. Disnix has the ability to deploy a system to multiple target environments (called: a profile). The following chain of command-line invocations to deploy a system make more sense to specify a profile:

$ export DISNIX_PROFILE=testenv
$ manifest=$(disnix-manifest -s services.nix -i infrastructure.nix \
    -d distribution.nix)
$ disnix-distribute $manifest
$ disnix-activate $manifest
$ disnix-set $manifest

than passing -p testenv parameter four times (to each command-line invocation).

I typically use files to process or produce arbitrary sets of data:

• If a tool takes one single file as input or produces one single output or a combination of both, I also allow it to read from the standard input or write to the standard output so that it can be used as a component in a pipe. In most cases supporting these special file descriptors is almost as straight forward as opening arbitrary files.
  
  For example, the ILBM image viewer's primary purpose is just to view an ILBM image file stored on disk. However, because I also allow it to read from the standard input I can also do something like this:
  
  

  $ iffjoin picture1.ILBM picture2.ILBM | ilbmviewer
  

  
  In the above example, I concatenate two ILBM files into a single IFF container file and invoke the viewer to view both of them without storing the immediate result on disk first. This can be useful for a variety of purposes.
  
  
• A tool that produces output writes to the standard output data that can be parsed/processed by another process in a pipeline. All other output, e.g. errors, debug messages, notifications go the the standard error.

Finally, every process should return an exit status when it finishes. By convention, if everything went OK it should return 0, and if some error occurs a non-zero exit status that uniquely identifies the error.

Command-line option parsing

As explained in the previous section, I typically follow the command-line option conventions of the platform. However, parsing them is usually not that straight forward. Some things we must take into account are:

• We must know which parameters are command-line option flags (e.g. starting with -, -- or /) and which are non-option parameters.
• Some command-line options have a required argument, some take an optional argument and some take none.

Luckily, there are many libraries available for a variety of programming languages/platforms implementing such a parser. So far, I have used the following:

• For UNIX programs implemented in C/C++: getopt_long(), getopt().
• For shell scripts: the getopt command-line utility
• Java command-line utilities: jargs
• JavaScript/Node.js: optparse.js

I have never implemented a sophisticated command-line parser myself. The only case in which I ended up implementing a custom parser is in the native Windows and AmigaOS ports of the IFF libraries projects -- I could not find any libraries supporting their native command-line option style. Fortunately, the command-line interfaces were quite simple, so it did not take me that much effort.

Validation and error reporting

Besides parsing the command-line options (and non-options), we must also check whether all mandatory parameters are set, whether they have the right format and set default values for unspecified parameters if needed.

Furthermore, in case of an error regarding the inputs: we must report it to the caller and exit the process with a non-zero exit status.

Documentation

Another important concern while developing a command-line utility (that IMO is often overlooked) is providing documentation about its usage.

For every command-line tool, I typically implement an help option parameter displaying a help page on the terminal. This help page contains the following sections:

• A usage line describing briefly in what ways the command-line tool can be invoked and what their mandatory parameters are.
• A brief description explaining what the tool does.
• The command-line options. For each option, I document the following properties:
  
  
  • The short and long option identifiers.
  • Whether the option requires no parameter, an optional parameter, or a required parameter.
  • A brief description of the option
  • What the default value is, if appropriate
  
  An example of an option that I have documented for the disnix-distribute utility is:
  
  

  -m, --max-concurrent-transfers=NUM  Maximum amount of concurrent closure
                                      transfers. Defauls to: 2
  

  
  that specifies that the amount of concurrent transfers can be specified through the -m short or --max-concurrent-transfers long option, requires a numeric argument and defaults to 2 if the option is unspecified.
  
  
• An environment section describing which environment variables can be configured and their meanings.
• I only document the exit statuses if any of them has a special meaning. Zero in case of success and non-zero in case of a failure is too obvious.

Besides concisely writing a help page, there are more practical issues with documentation -- the help page is typically not the only source of information that describes how command-line utilities can be used.

For projects that are a bit more mature, I also want to provide a manpage of every command-line utility that more or less contain the same stuff as a help page. In large and more complicated projects, such as Disnix, I also provide a Docbook manual that (besides detailed instructions and background information) includes the help pages of the command-line utilities in the appendix.

I used to write these manpages and docbook help pages by hand, but it's quite tedious to write the same stuff multiple times. Moreover, it is even more tedious to keep them all up-to-date and consistent.

Fortunately, we can also generate the latter two artifacts. GNU's help2man utility comes in quite handy to generate a manual page by invoking the --help and --version options of an existing command-line utility. For example, by following GNU's convention for writing help pages, I was able to generate a reasonably good manual page for the disnix-distribute tool, by simply running:

$ help2man --output=disnix-distribute.1 --no-info --name \
  'Distributes intra-dependency closures of services to target machines' \
  --libtool ./disnix-distribute

If needed, additional information can be augmented to the generated manual page.

I also discovered a nice tool called doclifter that allows me to generate Docbook help pages from manual pages. I can run the following command to generate a Docbook man page section from the earlier generated manpage:

$ doclifter -x disnix-distribute.1

The above command-line instruction generates a Docbook 5 XML file (disnix-distribute.1.xml) from the earlier manual page and the result looks quite acceptable to me. The only thing I had to do is manually replacing some xml:id attributes so that a command's help page can be properly referenced from the other Docbook sections.

Modularization

When implementing a command-line utility many concerns need to be implemented besides the primary tasks that it needs to do. In this blog post I have mentioned the following:

• Interface conventions
• Command-line option parsing
• Validation and error reporting
• Documentation

I tend to separate the command-line interface implementing the above concerns into a separate module/file (typically called main.c), unless the job that the command-line tool should do is relatively simple (e.g. it can be expressed in 1-10 lines of code), because I consider modularization a good practice and a module should preferably not grow too big or do too many things.

Reuse

When I implement a toolset of command-line utilities, such as Disnix, I often see that these tools have many common parameters, common validation procedures and a common procedure displaying the tool version.

I typically abstract them away in a common module, file or library so that I don't find myself duplicating them making maintenance more difficult.

Discussion

In this blog post, I have explained that I prefer the command-line for many development and system administration tasks. Moreover, I have also explained that I have developed many command-line utilities.

Creating good quality command-line tools is not straight forward. To make myself and other people that I happen to work with aware of it, I have written down some of the aspects that I take into consideration.

Although the command-line has some appealing properties, it is obviously not perfect. Some command-line tools are weird, may significantly lack quality and cannot be integrated in a straight forward way through pipes. Furthermore, many shells (e.g. bash) implement a programming language having weird features and counter-intuitive traits.

For example, one of my favorite pitfalls in bash is its behaviour when executing shell scripts, such as the following script (to readers that are unfamiliar with the commands: false is a command that always fails):

echo "hello"
false | cat
echo "world"
false
echo "!!!"

When executing the script like this:

$ bash script.bash
Hello
world
!!!

bash simply executes everything despite the fact that two commands fail. However, when I add the -e command line parameter, the execution is supposed to be stopped if any command returns a non-zero exit status:

$ bash -e script.bash
Hello
world

However, there is still one oddity -- the pipe (false | cat) still succeeds, because the exit status of the pipe corresponds to the exit status of the last component of the pipe only! There is another way to check the status of the other components (through $PIPESTATUS), but this feels counter-intuitive to me! Fortunately, bash 3.0 and onwards have an additional setting that makes the behaviour come closer to what I expect:

$ bash -e -o pipefail script.bash
Hello

Despite a number of oddities, the overall idea of the command-line is good IMO: it is a general purpose environment that can be interactive, programmed and extended with custom commands implemented in nearly any programming language that can be integrated with each other in a straight forward manner.

Meanwhile, now that I have made myself aware of some important concerns, I have adapted the development versions of all my free and open source projects to properly reflect them.