2010/12/06

To quote or not to quote

I've been bitten by this twice, so I write it down to avoid another bite.

Short summary: The popular way to pass compiler command-line options by command/parameter substitution does not work with arguments including whitespaces.

Here I'm talking about the typical Makefile idioms such as the following:

  gcc `gauche-config -I` ...

Or like this:

  CFLAGS=`gauche-config -L`
  gcc $(CFLAGS) ...

The commands gauche-config -I and gauche-config -L produce the -I and -L flag(s) to give to the compiler, respectively. Typically there's one -I flag, but there may be more than one -L flags. So it should be interspersed into the command line. That is, we can't quote outside of substitution like this:

  gcc "`gauche-config -L`"

On Windows, Gauche may be installed under a path that contains whitespaces. In fact, Gauche Windows installer uses C:\Program Files\Gauche as the default. To pass such pathnames to -I and -L options, each option must already be quoted right after substitution.

So, initially I naively changed the output of gauche-config to quote the pathname:

  -I"c:\Program Files\Gauche\lib\gauche-0.9\0.9.1_pre2\include"

Here came a twist. The modern way to compile extension modules is to use gauche-package compile command, which takes care of gory details and makes Makefile simpler; you just list the source files and the script takes care of compiling and linking, with proper options. Internally it uses gauche.config module to obtain the same information as output of gauche-config -I and gauche-config -L.

I used Windows installer to install Gauche under C:\Program Files\ and compiled Gauche-gl using it with MinGW/MSYS. Everything worked smoothly. I was satisfied.

Then, a few days later I was testing 0.9.1 prerelease on my Linux box, and found some extension modules didn't compile. Their makefile didn't use gauche-package, but directly invoked gcc with `guache-config -I` for the arguments.

I remembered I had been tripped with the same problem a few years ago and had given up. This time I wanted to solve it once for all. I thought my quoting scheme was wrong, and fiddled with gauche-config output for some time. I couldn't managed it to work. I carefully read the man page of bash. And leaned this:

• Quote processing is done before command/parameter substitution.
• The result of command/parameter substitution is subject of word splitting, unless the argument (before substitution) is quoted.
• Word splitting honor neither quotes, nor escaping (such as backslash). - it is simple string splitting with the characters specified by $IFS.

This means that, as far as we use shell-level command-line substitution, the output of gauche-config cannot contain whitespaces inside each argument. Tools using the same scheme, such as pkg-config, have the same limitation, and in fact it is documented in pkg-config manual.

If we can have intermediate step to preprocess the command line, and passes the quoted pathnames to the shell, it works. It is what gauche-package does. An alternative way could be to invoke shell within Makefile, and let make consturct the command line:

  INCLUDES = $(shell gauche-config -I)
  gcc $(INCLUDES) ...

With this, shell sees already expanded options, so it can process quotes correctly.

However, there may be a case that an extention can't use gauche-package compile because it requires special build process, and also it can't use GNU make. For the backward compatibility, I keep gauche-config -I and gauche-config -L not to quote pathnames. Hence, they are inherently unsafe way to construct a command line.

So, what should be the proper way for the extension makefile and gauche-config to handle pathnames with spaces? I don't know yet. For the time being, I added --incdirs and --archdirs options to gauche-config. They return pathnames separated by colon (or semincolon on Windows), and gauche-package constructs command line arguments from them. I'm not satisfied with it, though.

Tags: extensions, makefile, gauche-config, gauche-package