CastleCops® - Put virtual filesystems to work

Cameron Laird
Vice president, Phaseit, Inc.
April 29, 2003

A virtual filesystem (VFS) is an abstraction with surprisingly productive uses. Several popular languages now support VFS constructs, and Cameron Laird shows you what they are good for. "You won't understand how useful it is until you've tried it for yourself." When someone offers me that line, my usual reaction is to think the speaker doesn't understand the feature in question well enough to explain it.

In the case of virtual filesystems, though, my own experience tells me that the speaker is right. Jeffrey Hobbs is a Senior Developer with ActiveState Corp., and both of us have seen how unexpectedly powerful working with a virtual filesystem (VFS) can be.

Simple idea, big consequences
The idea behind a VFS is simple: it represents as a filesystem something that is not a filesystem. Filesystem here means a "conventional Linux-like filesystem": a tree or hierarchy of directly accessible directories and (ordinary) files. The concept should intrigue anyone working with Linux, of course, simply because so much of Linux's own character comes from the representation of devices, tables, and other objects within the UNIX filesystem. UNIX is founded on the principle that everything, or at least plenty of things, are files; VFS generalizes this to view as much as possible as a filesystem.

Note: Linux kernel engineers also speak of VFS, but in a different sense. This month's column is not about the Linux virtual filesystem switch, which dispatches filesystem drivers for ext2, ext3, reiserfs, and so on.

One way to think about VFS is that it's another example of a technology or concept that is "blurring the line between OSes and high-level language environments," in the words of independent developer Jean-Claude Wippler. Among other things, exposure in an application development language of a "system service" makes portability easier, for the operating system simply vanishes from view.

So, what kinds of things aren't filesystems, but are useful to represent that way? Lots of them: files accessible through FTP, HTTP, WebDAV, or other networking protocols; the contents of a .zip, CVS (concurrent versions system), or other archive file; database tables; projections of real filesystems restricted by security or other constraints; and many more.

You can easily see how such resources will map onto filesystems in a natural way. Suppose an example.zip archives these files:

first
subdir1/second
subdir1/third
subdir2/fourth

This archive was probably created as the direct (partial) image of an existing filesystem tree, and is naturally represented by the rooted tree:

./first
./subdir1/second
./subdir1/third
./subdir2/fourth

Plenty of industrial-strength products rely on VFS. The architecture of such DB2 features as its journalling filesystem rests on a VFS model. The well-known Zope application server provides a slightly more challenging example of a VFS. Zope's "acquisition" concept maps programmatic objects to URLs. In Zope, invocation of a method such as:

context.myproject.object1.method1(year 
= "1999")

corresponds to an HTTP request for the URL:http://myzope.com/myproject/object1/method1?year=1999

Do you see the benefit? VFS is also like UNIX's "everything is a file" concept, in that it's easy to understand and imitate the idea, but it might take years to appreciate how much it simplifies application design.

Consider an example. Suppose you have written a text editor; it provides means for accessing individual files, reading them, modifying them, and writing them back to storage. If you drop in a filesystem "virtualizer," you suddenly can use all the same code to navigate an FTP or ZIP archive, select individual items, modify them, and save them. Do you have a browser or backup utility or security scanner or version-control system that serves you well when it operates on local files? Virtualize its filesystem access and it immediately acts on .tar files, old tape reels, and corporate resources accessible only through a virtual private network (VPN). Vendors like to sell such add-ons for thousands of dollars. VFS gives them for free.

Or almost free. Programmers don't have anything new to learn; they just keep doing the same filesystem opens, closes, and so on they've always used. And that's the point: all the code looks as it always has. The only difficulty is that few language run-time libraries currently support full-blown VFS. Among the common difficulties is that drivers are often read-only, either because write capabilities require more delicate programming, or simply because write operations don't have a place in such protocols as HTTP.

Who has VFS?
The language with the best support for VFS is Tcl, Hobbs' specialty at ActiveState. In other languages, including Java and Perl, existing implementations of VFS are "impure" in that they provide new methods such as Perl's vfsopen to supplement core library entry points. In Tcl's release 8.4, in contrast, "Tcl's filesystem is completely virtual filesystem aware," as the community page on VFS explains. Among other things, this means that virtual file resources can be used syntactically any place ordinary files are recognized. Classic Tcl allowed this:image create -file myimage.gif

Tcl 8.4 extends this naturally to:

image create -file 
ftp://myserver.com/myimage.gif

Independent consultant Matt Newman first implemented a robust VFS for Tcl in the late 1990s to facilitate development he did on behalf of large financial corporations. Late in 2000, Vince Darley, a scientist with Eurobios, prepared an ambitious rewrite of Tcl's filesystem application programming interface (API) that, among other advantages, gives VFS hooks.

Article Continues
IBM


	2002-2008 � Computer Cops LLC dba CastleCops®. All rights reserved. Acceptable Use Policy. Use signifies your agreement. 87ppm 0.125s (0.007s) Making cybercriminals unhappy since 2002*