2020/05/30

C API and promise

This ate up my whole afternoon so I write it down not to fall into it again.

I've got a really weird bug. We have a parameter (say P). P has a default value, but it may not be available at the initialization time. Basically, what we want is to delay evaluation of EXPR below until the value of P is actually taken:

(define P (make-parameter EXPR))

Simply wrapping EXPR with delay did't cut it, for the user of P expected it to contain a value which wasn't a promise. We couldn't go to every place where P was used to wrap it with force.

So we added a special flag in the parameter, which applies force on the value whenever the value is taken. The feature isn't available from the Scheme world, though. It's only through C API, for we're not sure if such feature is a good idea yet.

Anyway, P got such a flag, so we could also say (P (delay EXPR)) to alter the value of P, with the actual computation of EXPR is delayed. And it seemed working.

However, we ran into an issue when some code takes the value of P from C API. The internal of parameter object is a bit complicated, but you can assume there's an C API that retrieves the value of the given parameter. Through C API, however, P's value looked like #<closure ...>, whereas when I took P's value from the Scheme world, it returned the value of EXPR.

I started tracking it down and it was like a rabbit hole. Scheme interface eventually calls the internal Scheme procedure %primitive-parameter-ref, which directly calls C API Scm_PrimitiveParameterRef. I inserted a debug stub to show the result of C call. The C API returns the mysterious closure, yet in the Scheme world it returns the desired value. Does Gauche runtime intercept the return value from C world to Scheme world? Nope. It's directly returned to the Scheme world. I have no idea where this #<closure...> came from, neither how the value changes to the desired one.

Furthermore, I found that if I evaluate (P) second time, C API returns the desired value. But no code is called to actually replacing P's value!

I poke around C stub generators, VM code, parameter code,... in vain. Finally, I opened up the source of Scm_Force, the C API for force. And BANG! The answer was there.

C runtime doesn't like call/cc. C procedures return either exactly once, or never. So, when you call back Scheme code from C, you have to choose one of these two strategies:

• Restrict the called Scheme code to returns at most once. If a continuation captured within the Scheme code is invoked again later, and tries to return to the C code again, an error is thrown.
• Split your C code to two, before the callback (A) and after the callback (B). Both A and B are ordinary C function. A arranges B to be called after the Scheme callback returns. Effectively, you write it as a continuation-passing style. With this, a continuation captured within the Scheme callback can be re-invoked, which just calls B again.

Most of Gauche runtime in C adopts the latter strategy, so that call/cc works seamlessly. By convention, the C API functions that use the strategy are named Scm_VM***. The caller of such C API can't expect to get the final result as the C return value, since such function may need more calculation (Scheme code and B part) to get the final result.

Scm_Force is that type of function, too. I only forgot to name it as Scm_VMForce.

Scm_PrimitiveParameterRef casually called Scm_Force when it has the delayed evaluation flag, expecting that it returns the final value. But in fact, Scm_Force can only be used in conjunction of Scheme VM to obtain the final result.

Tag: BugStories