2011/07/11

Generic file locking---is it really about files?

Gauche provides POSIX file locking through gauche.fcntl module, but the feature isn't available on all platforms. Providing a locking API that's always available has been on my TODO list for years.

And it's finally in. In file.util module, now you have with-lock-file. It can be used like this:

  (with-lock-file lock-file-name
    (lambda () ... do something exclusively ...))

It uses a classic inter-process locking relying on the exclusive file creation. Optionally you can use atomicity of mkdir.

The function make sure if you throw an exception in the thunk it unlocks the file, but if the application dies unexpectedly, it leaves the lock file (or directory). If it finds a lock file that's very old, you may allow it to steal the lock, assuming that the creator of the lock file died without cleaning it up. It takes quite a few parameters to configure how to handle various situations.

★ ★ ★

Originally I planned to provide a lock API that abstracts underlying lock mecanism---either POSIX fcntl file locking or using lock files (or directory/symlink/...). In fact, I attempted to write one several times. However, the more I think about it, the less the idea become appealing. POSIX fcntl locking and lockfile-based locking differ so much and attempts to hide the difference always lead to nowhere.

The biggest obstacle was that hiding POSIX file locking under the hood turned out to be an extremely difficult job, if not impossible. That is, if a locking library is using POSIX file lock down below, the user of the library must be very aware of it.

That's because, to use POSIX lock robustly, you have to make sure to meet two conditions at least: (1) Making sure the fcntl lock is actually supported on a particular file, and (2) making sure that the file to lock won't be accessed by other libraries/threads in the same process. (The latter condition came from the fact that POSIX fcntl lock is per process, not per file descriptor. See, for example, A tale of two standards and On the Brokenness of File Locking for the detailed discussions.)

These conditions aren't necessarily a show-stopper for applications---at an application level, you may be able to choose a file to lock that's likely to satisfy them. At a library level, however, it is not trivial to check if the given file is really on a filesystem that supports POSIX lock, and it is probably impossible to make sure the file won't be accessed by other parts of the program.

There are also a conceptual difference between POSIX fcntl lock and lock-file based locking---the former is to lock a file, but the latter is not really about a file. It's a way to mutex multiple processes. Of course you can use shared file lock as inter-process mutex and vice versa, but conflating these two makes me feel uneasy.

So, with-lock-file ended up to be based on the lock-file/directory. It's not very fast (involves multiple syscalls and file access) but I expect it to cover most of trivial cases to avoid race between processes.

Tags: file.util, gauche.fcntl, FileSystem, InterProcessCommunication