A Virtual Operating Environment for the Web

This is a progress report on our "Electronic Filing Cabinet", or "EFC" for short, in which I’ll share our experiences in developing this prototype system. We’ll look at the technologies and techniques, either already used or anticipated for use later in the project; problems encountered, some directly solved, some worked around. This system is a work-in-progress, with the prototype about 70% completed.

This written version follows the presentation given at CAUSE ’97. The presentation slides themselves are available at the following web URL, in Powerpoint format:

The Advanced Technology Group is a small working group of the Office of Information Technologies at the University of Virginia. Our work is both in research and development, and also involves forecasting and technology transfer.

Quick characterization

Let me introduce the Electronic Filing Cabinet with a broad characterization showing two uses. The first use could be characterized as proactive, strategic, or long-term. This is to ease the office burden in handling paper documents by providing an electronic, archival storage. The Electronic Filing Cabinet is not intended to eliminate paper documents, or even necessarily to reduce their volume. But we do hope to help get a grip on handling of paper documents. In this use, paper documents are scanned to electronic form and stored on a server for convenient access from network stations other than the scanning station.

The second use is best conveyed as a scenario. A report is received in a central office. Several senior managers, with offices scattered around a campus, each need to review the report. An operator scans the report, and it is imported as an Electronic Filing Cabinet document. Here the EFC is used here not so much for storage as for distribution. Managers receive notification of the report’s availability, through usual email. Each manager can separately read the report online through the EFC. The manager’s workstation need not be on the same local network as the scanning station or server. This second use can be seen to be more reactive, tactical, or short-term than the first. The Electronic Filing Cabinet is intended to encompass both types of need: as archive, and for notification and distribution. These two uses characterize the functionality now in place.

The remainder of this report lays out the project into three areas: Functionality, Design, and Lessons learned.

Early scope

Initially, we implemented a single datatype, scanned image documents. We deployed this functionality in a single user office, our organizational parent, the University’s Office of Information Technologies. This put our prototype code into a pilot test and to good use. The production character of inputting data proved good for both the pilot office and for our development group. For them, this allowed a growth in their base of documents and in metadata. (Metadata is "data about data", attribute information about documents which is made useful by abstracting it outside of the document itself. Document title and author are two examples of metadata.) For us, we got to share our craft with others and gain experience in usage, user interface, and implementation technology.

Functional Goals

The preceding sections gives the system’s functionality, narrowly seen and available today.

What will this scale and generalize into?: We have in mind a general file repository, including any file type; whose content is organized into a broad hierarchy of categories; searchable by both content and metadata; and with print and other export functionality. This is an archive system without versioning. We are concerned here more with storage and retrieval of documents, than with their creation, update, or rendering / viewing. These functions we leave to applications to provide, including web browsers and their plugins and helper apps. The EFC is a repository of documents.

The file type categorization made familiar by the Internet and guiding our work is MIME.

Handle various data types

Initially and currently, the EFC handles a single datatype, scanned image documents. The binding together of images of individual pages is done at the server, page images bound together with server code. This was to avoid a monolithic file type, with all pages in a single file, eg., multipage TIFF or PDF file.

But the prototype will generalize to allow MIME types generally, and perhaps also URL references ("bookmarks") to material on the World Wide Web (www). So the general case which we aim at is a repository of MIME files, whose import is facilitated by EFC discovery or verification of file type, but whose display is handled by software augmenting EFC functionality (browsers, plugins, helper apps). The EFC leaves those responsibilities to that other software, and concentrates on storage, categorization, and delivery of content.

Handle various protocols

Initially and currently, the EFC provides two ways to import files into its storage: the FTP and NFS protocols. In either case, the user moves files into a drive or directory on the network specific to the EFC and generally not on the user system. (NFS is Unix technology similar to Novell or Microsoft network drives.) These are general protocols, with no association to specific file types.

But the prototype will grow to include other manners of bringing content into the EFC repository. These will continue to be useful in bringing in content generally. HTTP is such a general-purpose protocol, and electronic mail can serve so generally with MIME capability. But mail usually conveys a specific file type, a mail message. And fax always conveys a paged image set. These protocols will add their general and specific utility to the Electronic Filing Cabinet.

Why not buy?

We sought and seek an alternative to commercial systems which is inexpensive (avoiding per-seat costs); and both non-proprietary and based on open standards. We seek to fully expose the implementation to allow flexibility and customization. Additionally, lack of an available system, with search over both content and metadata, led to a decision to build a prototype system.

Web-based functionality

Document import into the EFC is through several protocols, from a scanning station or other network workstation originating the content files, to the server implementing the EFC. In this, it is an Internet application. But after document creation, most or all functions are specifically web-based, with content provided by the server to any workstation with a web browser. Note specifically that not all workstations need to have scanners, and that multiple concurrent access is provided.

Design Goals

The EFC is a modularized, scalable system, with replacable components. We are using off-the-shelf components where possible, including web search, OCR, and fax servers. These should be independent of the web server or browser used. Additionally, the EFC itself can be loosely coupled to other external systems.

Authentication

We have developed an authentication mechanism, independent of the web server and of the rest of the Electronic Filing Cabinet. It is already reused in another of our projects. This "web wrap" functionality front-ends the CGI scripts implementing the EFC at the server. It verifies user identity before the EFC gets control. This follows the CGI implementation of our prototype version, in which HTML pages are generated dynamically by a template mechanism. The authority against which user IDs and passwords are checked is configurable; it could be based on the operating system of the server platform, eg., Unix /etc/passwd file, or on a file specific to the application (EFC).

To form an application session across individual Webwrap invocations (separate consecutive HTTP request/response interactions), state must be preserved. (Here, what must be retained is that the user can continue to exercise the privileges of her user ID / password without continually keying that in.) The HTTP authentication header mechanism does not provide for authentication time-out, and we had a negative view of Netscape’s "cookie" mechanism. Moreover, hidden form fields are transmitted from the browser to the server only on a form submit, but not on the user following a link. So the Webwrap security session data, and any EFC application session data, is included as pathinfo at the end of every URL on a generated page

Whenever user identity is not given in a followed URL or when enough time has elapsed since last use to amount to a session timeout, Webwrap presents the user with a login screen instead of the requested EFC page. After successfully completing the security form on this page with user ID and password, Webwrap lets the browser user through to the Electronic Filing Cabinet without challenge.

Import and user Inbox

Scanned documents generally comprise several separate files, each of which is the image of a scanned page. EFC software running on the server could see files appearing as they are moved into the EFC import directory using NFS or FTP protocols. Determination that all of the files from a document have arrived would be difficult. This motivated the concept of a per-user Inbox, which is more or less a loading dock for candidate EFC documents. This the user can view through the EFC web interface. Here the user can trigger import of the document into the EFC proper, implicitly saying "all the document files are present".

Currently, a provisional (quicker) programmatic determination of file type is done before the Inbox is presented to the user. On selection of files to import, additional (more lengthy) verification of file type is performed. [We plan to move all file type detection into a background process.]

Basic repository structure

Once imported into the EFC, a document resides in a folder hierarchy. A folder is simply a container for a set of documents, and can contain nested folders. Content resides as a folder, a document, or a page. The content itself can be viewed, or the metadata describing it. In the case of a folder, content is a list of the items it contains. The EFC user interface supports this viewing and a limited set of actions: folder creation; items moved from one to another folder; item deletion.

Document creation

Documents are created by user action on the Inbox page. The files which are moved by NFS or FTP into the EFC import directory are not EFC documents until after this user action. Minimal metadata is provided programmatically. From the user point-of-view, this metadata is an initial document title extracted from the original document filenames. The original per-page TIFF files from the scanning software are kept as archive files; these will later be used with a print function. A background conversion process creates additional GIF format files for use with web viewing. [We anticipate moving to PNG format, as this is provided in upcoming version 5 browsers.]

Views

The various user views of EFC documents available through the web interface are shown in the Powerpoint slides of the CAUSE talk. Generally, clicking on the user ineterface component (link) representing one of several EFC objects progresses the view to be of that object. So, a web page showing various folders allows the user to view the contents of any of the folders individually; a web page showing the various pages of a document allows the user to view an individual page; and so on, this similar to drilling down in the familiar folder/document or directory/file operating system metaphor. The view at the document level includes thumbnails of pages, and at the page level the image of the page itself. The page can be viewed at a reduced resolution or at the full resolution of scanning.

Storage and Maintenance

Again, the EFC is an archival storage. We’ve implemented this as flat files for either content or metadata, which arrangement exposes the implementation to system development and maintenance in a way that a database would not. (Tools are already available to work on the files individually.) Filenames include unique and arbitrary object identification, not including in the filename metadata describing the contents. Folders are implemented similarly to documents.

Metadata

We modeled our metadata implementation on the Dublin Core standard. This is a SGML-based , and will lead gracefully into a conversion to XML as this standard becoming prevalent. Metadata is available from import into the EFC, both from content and filename. Additional user editing of metadata is supported in the user interface. When the web search capability is implemented, metadata will enable discovery of documents without regard to folder association.

Because creation of working GIF files from the original TIFF files occurs in the background, metadata is available for user editing immediately after import of the files into an EFC document.

Image conversion issues and solutions

Image conversion speed is memory-size dependent, as the software requires the image to be resident in memory (RAM). Our first server was an old Sun IPX, whose 60 MB RAM limited image conversion speed to about 10-12 minutes per page! In moving to an Ultrasparc 2 with 128 MB, we cut this time to 26 seconds per page. As a typical document scanned in our trial has 30 pages, this results in 13 minutes server processing per document. This is for creation of 2 resolutions (30% and 100%) in GIF format of each scanned page.

Image storage issues and solutions

Each page requires about 400 KB of disk storage. After 58 days of active scanning, our 10,000 pages currently require 4 GB of disk. In an active week, about 1 GB more disk is required. For comparison, this is equivalent to more than 100 high-density disks daily. Our current system has 16 GB disk capacity. Our interim solution to the disk needs created by successful scanning is to add more disk. We did this once, adding one 9 GB drive in a multibay unit allowing for expansion. We anticipate a lasting solution which automatically stages disk storage to a backing tape store. This may in fact be in place before additional disk need be physically added.

OCR issues and solutions

Our pilot office is predominately Macintosh based. We chose OmniPage as perhaps the best a scanning solution on that platform. We encountered numerous problems, some of which were relieved by upgrading to a newer, larger Macintosh. We tried unsuccessfully to script around other problems; unfortunately, OmniPage does not expose enough of its functionality to AppleScript to be useful. Other problems were also not easily overcome, including severe failure requiring system reboot on OCRing some pages. This problem was somewhat related to the complexity of the page, with complex tables and blank pages equally causing problems. The problems seemed related also to how long the scanning application had been operating, as if OCR operation had a memory leak. We have temporarily stopped OCR processing on the scanning workstation; simple image scanning works well and is proceeding. Meanwhile, we are acquiring a server-based OCR solution. We view OCR as a necessity for automatic metadata discovery.

Prototype timeline and remaining work

The "application-within-an-application" complexion of building a browser app forces a conservative approach to development in order to enhance portability among browsers and servers. This has led to a CGI-based system, with HTML forms and common image delivery at the browser. This will continue through the pilot phase of the system. Datatypes and import mechanisms will be added, including server-side OCR and import of mail and fax messages. We anticipate a completed prototype using this technology by spring, 1998, including web searching, email notification of new content, and some print functionality.