2018/12/24

Unbalanced unquotes undermine utility

tl;dr - Our design choice of quasirename having implicit quasiquoting was wrong, and we'll fix it.

＊ ＊ ＊

In Common Lisp, backquotes and commas are handled by the reader---this means (1) every comma (and comma-atmark) must have corresponding backquote that's lexically surrounding it, and (2) once S-expression is read, you never see the trace of commas and backquotes.

Scheme took a different approach. Quasiquotes and unquotes are just a shorthand of the form (quasiquote form) and (unquote form). Their interpretation is left to the semantics of these forms.

This opens tempting possibilities to expand usage of these forms. SCSH's extended process form ( https://scsh.net/docu/html/man-Z-H-3.html ) is one example. Its redirection forms are implicitly quasiquoted, and unquote forms in it are evaluated without a corresponding quasiquote.

(define *outfile* "output.txt")

;; Redirect output of my-program to the file named by the value of *outfile*
(run (my-program) (> ,*outfile*))

＊ ＊ ＊

Gauche adopted explicit-renaming macro for the lower hygienic macro layer (ref:er-macro-transformer). While syntax-case provides pattern matching and syntactic wrapping all in one set, ER-macro provides a minimal mechanism to hides underlying macro expansion system. In practice syntax-case is handy, but its features are inseparably tied to it. For example, you can't just simply use its pattern matcher as a runtime library independent from the macro system. We prefer basic tools each of which does one thing well, and building complicated systems combining those orthogonal tools.

For the pattern matcher, we already have mighty-powerful match (ref:util.match). On the other hand, constructing macro output is rather cumbersome with bare ER-macro, as we have to apply the rename procedure to every identifier we want to avoid from name conflict:

(define-syntax when-not
  (er-macro-transformer
    (^[form rename id=?]
      (match form
        [(_ test expr1 expr ...)
         `(,(rename 'if) (,(rename 'not) ,test)
            (,(rename 'begin) ,expr1 ,@expr))]
        [_ (error "malformed when-not:" form)]))))

So we introduced quasirename (ref:quasirename) that works quasiquote with renaming:

(define-syntax when-not
  (er-macro-transformer
    (^[form rename id=?]
      (match form
        [(_ test expr1 expr ...)
         (quasirename rename
           (if (not ,test) (begin ,expr1 ,@expr)))]
        [_ (error "malformed when-not:" form)]))))

Quasirename employs implicit quasiquote. It replaces every identifier in the form with the result of applying rename procedure on it, except the unquoted (and unquoted-spliced) portion which expands to the value of the expression as is. The code can be written almost identical to the legacy macro, except replacing quasiquote with quasirename (and provide the rename procedure).

We're quite happy with it and start rewriting lots of macros using it, then we realized its shortcomings.

＊ ＊ ＊

When quasiquote is nested, corresponding unquote should also be nested. The outermost quasiquote corresponds to the innermost unquote. It is simply implemented by keeping track of nesting levels (when you see quasiquote, increment the nest level; when you see unquote, decrement it; and keep the unquotes except zero-level ones).

For example, suppose you have the following nested quasiquote forms:

(let ((a 'outer))
  `(let ((a 'inner))
     `(list ,',a ,,'a)))

When you unwrap the outer quasiquote form, you get:

   (let ((a 'inner))
     `(list ,'outer ,a))   

And when you unwrap the inner quasiquote you get:

      (list outer inner) 

The nested unquotes may look scary but the rule is simple.

• Count the level of unquote from left to right.
• If you want to evaluate the form in a particular level (except the innermost level), put ', (quote - unquote).
• Or, if you want to keep the form untouched in that level and leave it to be evaluated in higher level, put ,' (unquote - quote).

To make this mechanism work, however, every quasiquote form must know the levels of unquotes in it. Unquote forms that don't have corresponding quasiquotes would trip quasiquote forms.

Using implicit quasiquote in quasirename makes it very difficult, if not impossible, to write a quasiquote form that yields quasirename form, or other combination of nestings.

＊ ＊ ＊

So, what shall we do?

One solution is to recognize quasirename as a built-in syntax just like quasquote; let each one know the other, and count nestings properly.

However, that will make quasirename inherently unportable Gauche-specific syntax. Furthermore, what if we want to add more implicitly quasiquoted forms in future? Do we want to change every quasi-something form expanders?

Another solution is to let quasirename require its second argument to be quasiquoted. That is, this should be the proper form:

(quasirename r
  `(form ...))

and the argument without quasiquote should be invalid.

For the backward compatibility, we could allow the form being without quasiquote for a while. The only incompatible case is that the existing code intended to yield a quasiquoted form. In that case, it should be rewritten to use double quasiquotes.

(quasirename r
  ``(form ...))

Tags: 0.9.8, quasiquote, quasirename, macro

2018/12/22

Upgrading to 0.9.7

0.9.7 is out. (Noticed I didn't annouce 0.9.6 in this blog).

As described in the release notes, this release is not binary compatible---extensions must be recompiled. In case if you run a server with a bunch of Gauche extensions (like me), It's a bit of work.

In case if you forgot what extension modules you've installed, check a directory ${prefix}/share/gauche-0.9/site/lib/.packages/. It contains *.gpd files of extensions you've installed for 0.9.6 and before. For 0.9.7 and later, the gpd files are going into ${prefix}/share/gauche-0.97/site/lib/.packages/. (0.9 and 0.97 suffix in the directory name indicates ABI version.)

Tag: 0.9.7

2018/05/13

Plugged an annoying error behavior

I've been aware of an annoying behavior in Gauche. From time to time, I myself got bitten by it and thought "Gee, it's terrible! I should fix it." But there was always more urgent tasks to finish so it was let untouched.

It is that, when an error is caused by a huge object (e.g. long list or deep tree), Gauche tries to report the offending object entirely, producing huge error message:

*** ERROR: vector required, but got (*TOP* (html (head (title "RSSMix: Recent En
tries") (link (|@| (type "text/css") (rel "stylesheet") (href "wiliki-sample.css
")))) (body (h1 "RSSMix: Recent Entries") (div (|@| (align "right")) "[" (a (|@|
 (href "http://practical-scheme.net/wiliki/wiliki.cgi?WiLiKi:RSSMix")) "What's T
his?") "][" (a (|@| (href "?c=info")) "Sources") "]") (hr) (table (tr (td "2018/
05/13 16:27:40 UTC") (td (a (|@| (href "http://ja.reddit.com/r/lisp_ja/")) "Redd
it - LISP ja") ": " (a (|@| (href "https://ja.reddit.com/r/lisp_ja/comments/8j4y
ff/実行時のデータ型の表現手法_2012/")) "実行時のデータ型の表現手法 (2012)"))) (tr (td "20
18/05/13 15:11:00 UTC") (td (a (|@| (href "http://ja.reddit.com/r/lisp_ja/")) "R
eddit - LISP ja") ": " (a (|@| (href "https://ja.reddit.com/r/lisp_ja/comments/8
j4ez7/evolution_in_lisps_qiita/")) "Evolution In LISPs - Qiita"))) (tr (td "2018
...

It is especially bad when you're running gosh in *scheme* buffer of Emacs with font-lock mode. Emacs starts to parse this huge S-expression and does nothing else---even not accepting keystrokes.

Yesterday I hit it again and had to kill Emacs. That was the last straw.

It turns out it's so simple that I wonder why I didn't already do it long time ago.

--- a/src/libexc.scm
+++ b/src/libexc.scm
@@ -73,7 +73,7 @@
             (values '() (list exc)))
         (let1 name (condition-type-name exc)
           (if (condition-has-type? exc <message-condition>)
-            (format out "*** ~a: ~a\n" name (~ exc'message))
+            (format out "*** ~a: ~,,,,200:a\n" name (~ exc'message))
             (format out "*** ~a\n" name)))
         (for-each (cut report-mixin-condition <> out) mixins)))))

Now the error message is truncated if it's too long.

*** ERROR: vector required, but got ((*TOP* (html (head (title "RSSMix:
 Recent Entries") (link (|@| (type "text/css") (rel "stylesheet") (href
 "wiliki-sample.css")))) (body (h1 "RSSMix: Recent Entries") (div ...
Stack Trace:
_______________________________________
  0  (vector-ref zz 1)
        at "(standard input)":4

Tag: 0.9.6

2018/04/08

Easier installation

Distributing a standalone binary executable is the easiest way to deliver Gauche applications to the users. However, what if you need your code to be built on the user's machine? Maybe your code is part of larger system and the client needs to build it on their site.

Once upon a time, delivering a source tarball and asking the user to run ./configure && make && make install wasn't a big deal. It was so much easier than before, when you had to read instructions cafefully and edit Makefiles according to your environment. However, the world has moved on.

So I wrote a small shell script get-gauche.

If you trust me enough, you can ask the user to do this:

    curl -s https://raw.githubusercontent.com/shirok/getgauche/master/get-gauche.sh | /bin/bash

It works as follows:

• Ask the user where to install Gauche
• Check if the latest version of Gauche is already installed; if so, do nothing.
• Check if the user has write permission to the install destination; if not, ask the user if it's ok to use 'sudo' for installation.
• Download the official tarball of the latest release, compile and run check, then install it.

You can also download the get-gauche script and run it. The script can accept a bunch of command-line options to customize the behavior. See https://github.com/shirok/get-gauche for the details.

One instance I used it was like this: I wanted to use Gauche for testing and various management work in the product. I included get-gauche.sh in the source tree, and in the Makefile I invoked it to install Gauche under the build tree.

Tags: get-gauche, Installation

2018/04/05

Static linking and standalone executables

One of the most frequent-asked feature for Gauche is the ability to compile a Scheme program and produce an executable file. Well, it finally comes in the upcoming 0.9.6 release!

Already in the repo, the build process creates a static library of libgauche by default. There's also a script build-standalone that takes Scheme script source files and spits a stand-alone binary executable---meaning, you can copy the single file to another machine and just run it, without having Gauche runtime there, given that the other machine is the same architecture. (Note: It statically links libgauche but still depends on quasi-standard dynamic libraries such as libz.so.)

The detailed explanation is in the "Building standalone executables" section of the manual. There's a sample source in examples/standalone you can play with.

If you just want to use the feature, that's all you need to know. The following is the discussion behind the scene...

＊ ＊ ＊

The ability to produce standalone executable is somewhat considered a distinguishing feature of programming language implementation that separates itself from interpreters or scripting languages.

However, we'd say such a feature has nothing to do with the language being interpreted or scripted. It is purely for the convenience of distributing Gauche programs.

In fact, what build-standalone does is to take Scheme source files and generates a C code snippet with the Scheme code embedded as a C string literal, which is just evaled when the binary is run. There's no performance advantage in the binary form. The static version of libgauche also contains all of library code written in Scheme as C string literals, to be evaled on demand.

In fact, it is a trade-off; a standalone binary is easier to distribute, but you lose flexibility of scripting that the user can easily modify the source and run immediately. Also, the size of stand-alone binaries is not small (16MB on Linux x86_64), because it carries around entire Gauche runtime in it. (It's Lisp's curse---because of eval, we can't statically pick only required code.)

To distribute Gauche applications as Scheme scripts, you need Gauche runtime installed on the target machines. But that's the same for most languages---you need Java runtime to run Java applications, or even need libc to run C applications. If you deploy a bunch of Gauche applications in a limited number of users, it might still be better to install Gauche runtime on each target machine and distribute Scheme files, rather than build a standalone binary for each of applications.

That said, the standalone executable feature will expand the use of Gauche, I hope.

(And I do plan to improve ahead-of-time compilation, employing more optimizations, so standalone binary may have performance advantage in some day.)

Tags: 0.9.6, build-standalone