Visual Interfaces to Wearable Computers

Visual Interfaces to Wearable Computers Mediated and Diminished Reality AndersPearson 2001-05-07 The rising field of wearable computing brings with it a unique set of requirements as well as unique possibilities for visual interfaces. This paper explores these requirements and possibilities, surveying the fundamental research and the state of the art in this field. Several different aspects and implementations of visual interfaces for wearable computers are covered, focusing on the "Eyetap" system for "Mediated and Diminished Reality" as implemented and described by Prof. Steve Mann. The theoretical underpinnings of the system and the engineering decisions driving its implementation are discussed as well as the current and potential applications. Introduction/Background In order to understand how and why the described visual interfaces to wearable computers were developed and how they can be used, a certain amount of background on wearable computing and related fields is necessary. This section provides a brief overview of the field, its history and a general description of where current research and technologies stand. This should provide the reader with the context necessary to understand the later sections and, in particular, to understand the requirements, constraints, and applications for visual interfaces in this field. Wearable Computers

"A wearable computer is a computer that is subsumed into the personal space of the user, controlled by the user, and has both operational and interactional constancy, i.e. is always on and always accessible. Most notably, it is a device that is always with the user, and into which the user can always enter commands and execute a set of such entered commands, and in which the user can do so while walking around or doing other activities. The most salient aspect of computers, in general, (whether wearable or not) is their reconfigurability and their generality, e.g. that their function can be made to vary widely, depending on the instructions provided for program execution. With the wearable computer (WearComp), this is no exception, e.g. the wearable computer is more than just a wristwatch or regular eyeglasses: it has the full functionality of a computer system but in addition to being a fully featured computer, it is also inextricably intertwined with the wearer. This is what sets the wearable computer apart from other wearable devices such as wristwatches, regular eyeglasses, wearable radios, etc.. Unlike these other wearable devices that are not programmable (reconfigurable), the wearable computer is as reconfigurable as the familiar desktop or mainframe computer."

from: Definition of "Wearable Computer" (http://wearables.about.com/gadgets/wearables/bldefinition.htm) Simply put, a wearable computer is a computer system designed to be worn by and to interact with the user on a constant basis. Most definitions of wearable computers are specifically worded to distinguish the wearable computer from other technologies that it may be confused with. A wearable computer differs from laptop computers and Personal Digital Assistants (PDA's) such as palm pilots by its constant interface with the user. In order to use a laptop or PDA, it must be explicitly opened and turned on; a wearable is always on and has some form of steady stream to the user through which it can convey information. Laptops or PDAs also typically consume the user's full attention while in use; the user must be able to position the screen and keyboard where they can be accessed and would have a very difficult time performing other complicated actions while using one. Contrarily, a wearable is designed to be unobtrusive, not demanding the user's full attention under ordinary circumstances; a user should be able to use the wearable while performing other, complex, physical or mental tasks such as jogging, fixing a jet engine, or carrying on a conversation with a colleague. This constant information stream from the wearable to the user is most often in the form of a head mounted display (HMD) that is either translucent, allowing the user to see normally through it, or only visible in one eye or in peripheral vision. Again, a primary goal of the design of wearable computer systems is unobtrusiveness. Recent advances in display and miniaturization technology have made the HMD a very feasable and effective output device due to their small size, low weight, low power consumption, unobtrusiveness, and ability to display fairly large amounts of information in an easy to read format. While HMDs are definately the most popular output device for wearable computers, they are not the only ones in use; audio (text-to-speech) output devices and even tactile output devices are all in current use and have had no small amount of success in certain applications. One of the main goals of wearable computing is not to simply provide continuous access to a desktop computer and its functionality, but to create hybrid systems of user and computer that allow for a synergistic effect to take over; rather than attempting to emulate human intelligence in the computer, as is a common goal of research in Artificial Intelligence, the goal of wearable computing is to produce a synergistic human-machine system with the computer handling tasks that computers are well suited for such as providing ready access to large amounts of information and the human handling higher level decisions and providing overall direction. Generally, making the computer as much of a prosthetic extension of the person as possible. This human-computer system is made possible by several design factors of the wearable. It is essential that there is operational and interactional constancy. Ie, the computer runs continuously, and is "always ready" to interact with the user. As mentioned earlier, it must be unobstructive or unmonopolizing of the user's attention. It cannot cut the user off completely from the outside world; one must be able to attend to other matters while using the wearable. It must be continuously observable by the user: it can get their attention at any time. It must be controllable by the user; the user can gain control of it at any time with minimal effort in order to enter or execute commands. And, it must be location and environmentally aware. Wearables generally consist of several seperate pieces carried on the user in either a specially modified vest, or some type of closely worn backpack: the main processor and storage unit (often based on PC/104 technology or simply the modified internals of standard laptop or sub-notebook computer), batteries, one or more output devices (most commonly, the HMD described earlier), and one or more input devices. One-handed chorded keying devices appear particularly popular for the main input device of wearable computers but other types of input devices are not uncommon; microphones, bodily sensors that allow the wearable to monitor things like heart-rate and respiration, and motion detectors have all been used for different applications. Most interestingly from the point of view of this paper, is the work that has been done with wearables that feature a camera (typically video, but occasionally still as well) as an integral input device. The Eyetap System An Eyetap is a system designed by Steve Mann for interposing a computer into the wearer's field of vision allowing it to mediate what the wearer sees. It absorbs, quantifies, and resynthesizes the light the user would normally see. When the light is resynthesized under computer control, visual information can be blocked, modified, or added in real-time before it is presented to the user. Typically, an Eyetap system resembles an ordinary pair of eyeglasses or sunglasses and consists of three parts: an input sensor array, typically a digital camera or similar device a diverter which redirects eyeward bound light into the input sensor an "aremac" which reconstructs at least some of the light from the diverter after it has been processed by the wearable computer (the term "aremac" is "camera" spelled backwards since it is basically a camera in reverse). Several different implementations of the Eyetap have been implemented exploring different focus models. In one kind of implementation, a focus tracking mechanism is used to reconstruct rays of diverted light in the same depth plane as imaged by the camera. In another implemenation, the aremac has an unlimited or extended depth of focus so that the eye itself can focus on different objects. The second model has proven easier to implement and more robust and is therefore, more frequently used. The aremac has focus linked to the measurement system (e.g. "camera") focus such that the measurement system's focus controls the aramac focus, so that objects seen depicted on the aremac of the device appear to be at the same distance from the user of the device as the real objects so depicted. Such a linked focus gives rise to a more natural viewfinder experience, as well as reduced eyestrain. Reduced eyestrain is important because the device is intended to be worn continually. In stereo versions of the proposed device, there are two cameras or measurement systems and two aremacs that each regenerate the respective outputs of the camera or measurement systems. The apparatus is usually concealed in dark sunglasses that obstruct vision except for what the apparatus allows to pass through. Because the device absorbs, quantifies, processes, and reconstructs light passing through it, there are extensive applications in Mediated Reality. Mediated/Diminished Reality Mediated Reality is created when virtual, or computer generated, information is mixed with what what the user would otherwise normally see. Mediated, Augmented, and Diminished reality arose out of research into Virtual Reality. Virtual Reality creates a completely computer generated environment for the user. Augmented Reality takes the user's existing environment and adds computer generated information to it. Diminished Reality is similar to Augmented Reality except that it involves programmatically subtracting information; it filters the environment, altering real objects replacing them with virtual ones, or rendering them imperceptable. Mediated Reality is the combination of Augmented and Diminished Reality. Mediated Reality differs from Virtual Reality in the sense that Mediated Reality allows the visual perception of reality to be augmented, or, more generally computationally altered whereas Virtual Reality typically creates a wholly artificial environment for the user. In a sense, a mediated reality system serves as a shell or membrane encapsulating the user and controlling the flow of information between the user and the environment. This membrane can act as a sensory enhancement device, adding computer generated information (Augmented Reality) or an information filter, blocking out material the user might not wish to experience, whether it be offensive advertising, or simply a desire to replace existing media with different media (Diminished Reality). Whenever the subject of wearable computers or Mediated Reality is brought up, a typical complaint is that it would merely increase the degree of "information overload" in the lives of its users. Since the human brain can only make sense of so much information at once, this is a legitimate concern. Diminished Reality is a reaction to this fear. It is an application of Mediated Reality that primarily involves subtracting information from the user's field of vision; typically removing "noise" in the form of unnecessary or distracting information. As will be described in a later section, one application of Diminished Reality is the identification and blocking of Billboard type 2-dimensional advertisements, blotting them out or reusing the visual space for more useful information. Mathematical Underpinnings In order for the Eyetap system and Mediated/Diminished Reality to be implemented, several problems had to be solved. In order for the Eyetap system to function properly, overlaying information on or blocking sections of the user's field of view, it must be able to accurately and efficiently track the user's head movements. Some similar systems use external head-tracking devices for this purpose. Since one of the goals of the Eyetap system is compactness and, eventually, complete stealth, there are limitations on how many devices can be head mounted without drawing attention to the user. Currently, the display and camera portions of the Eyetap can be embedded entirely in a pair of normal looking sunglasses. It is necessary that the head-tracking it employs be designed such that it doesn't rely on any special apparatus being present in the environment. Therefore, a camera-based head-tracking method was developed based on the VideoOrbits algorithm which performs head-tracking, visually, based on a natural environment, and works without the need for object recognition, relying instead on algebraic projective geometry. User location often provides valuable clues to the user's context. By identifying the user's current task, the computer can assist actively in that task by displaying timely information or automatically reserving resources that may be needed. However, a wearable computer might also take a more passive role, simply determining the importance of potential interruptions (phone, e-mail, paging, etc.) and presenting the interruption in the most beneficial manner possible. For example, while driving alone in an automobile, the system might alert the user with a spoken summary of an e-mail. However, during a conversation, the wearable computer may present the name of a potential caller unobtrusively in the user's head-up display. Aside from context, there are many other reasons that it would be advantageous for the wearable unit to be aware of the user's precise location. Eg, the wearable could function as a sort of interactive map, directing the user to a desired destination. Similarly, there are many potential applications of wearables that involve superimposing additional information over the user's vision in manners that require knowledge of both the user's exact position as well as their orientation. The wearable's awareness of location is a common and fundamental requirement. Today, most outdoor positioning is performed in relation to the Global Positioning System (GPS). Differential systems can obtain accuracies of less than one meter, and update rates of one second are common. However, indoor systems require different methods. Current systems such as active badges and beacon architectures require increased infrastructure for higher accuracy. This increased infrastructure implies increased installation and maintenance. For a general purpose wearable system that is to function in any environment and not just one or two specially prepared buildings, some kind of visual based system for inferring the user's location is desirable. Algebraic Projective Geometry and the Chirplet Transform Most indoor and outdoor environments contain a great deal of periodicity. Repeating lights, windows, bricks, and similar repeating motifs abound. When an picture is taken, this periodicity is usually not captured exactly except in the rare case where the film plane is exactly parallel to the surface. This inherent abundance of periodicity can be used quite reliably to infer the orientation of the camera towards planar surfaces. This information, when combined in the time dimension, can then be used to further infer direction of motion of the camera relative to its environment. The Algebraic Projective Geometry that makes this possible works by taking into account the periodicity in perspective or the so-called 'p-chirp' (a special kind of chirp comprising projected harmonic components). "Chirping" refers to the effect of increasing or decreasing spatial frequency with respect to spatial location. The p-chirp is the effect that is seen when one looks at a planar surface with a regular periodic pattern on its surface from an angle other than exactly parallel to it. Eg, if a photograph is taken facing straight down a railroad track, the ties closer to the camera (toward the bottom of the image) will be spaced farther apart on the photograph than those ties that are farther away from the camera. Another example is shown in figure 1. chirp.gif 490 493 view of projective geometry acting on a period structure figure 1: Here is a picture (a) containing periodicity-in-perspective. If a single raster is plotted across the center of the image, it is observed that it is nearly periodic but that the period changes (in this case, decreases) from left to right (or equivalently, the frequency increases from left to right). The most dominant p-chirp component is plotted in (c), below the plot of the raster. (image from wearcam.org) The general method used to determine the orientation of the camera to a plane in its environment is also shown in figure 1. From the image, a single raster across is taken creating a vector of the intensities of each pixel in the raster. From this vector, a "fundamental frequency" (but increasing or decreasing across the image) is calculated. This process is repeated going vertically across the image. From these "fundamental frequencies" and the rates at which increases and decreases (its chirp rate), the relative orientation of the camera can be calculated with ordinary projective geometry. This chirping phenomenon is implicit in the proposed system, whether or not there is periodicity in the subject matter. The only requirement is that there be some distinct texture upon a flat surface in the scene. These fully-automatic methods are also relatively immune to deviations from the assumptions of static scene and no parallax. VideoOrbits Camera Based Head-Tracking In order for an augmented or mediated reality system to be very useful, the virtual information or light as seen through the display must be properly registered and aligned within the user's field of view. Furthermore, as mentioned earlier, it is unadvantageous to rely on external mechanical or electrical devices to track the user's head; an image based method of head-tracking is preferred. This image based method is based on the VideoOrbits algorithm (Mann, 1993). The VideoOrbits algorithm performs head-tracking, visually, based on a natural environment, and works without the need for object recognition. Instead it is based on algebraic projective geometry, and a featureless means of estimating the change in spatial coordinates arising from movement of the wearer's head. Direct featureless methods are used for estimating the 8 parameters of an "exact" projective (homographic) coordinate transformation between two or more images of the same scene. The approach is "exact" for two cases of static scenes: (1) images taken from the same location of an arbitrary 3-D scene, with a camera that is free to pan, tilt, rotate about its optical axis, and zoom or (2) images of a flat scene taken from arbitrary locations. Previous methods for inter-image combination have been based on affine transformations (which lack the degrees of freedom to "exactly" characterize such phenomena as camera pan and tilt) and/or which have relied upon finding points of correspondence between the image frames. Eg, the affine model cannot capture camera pan and tilt, and therefore cannot properly express the "keystoning" and "chirping" we see in the real world. The camera in the wearable system simply tracks itself based on its view of objects in the environment. The algorithm provides an estimate of the true projective coordinate transformation for successive image pairs. The system is designed with a frame-rate high enough that successive pairs of images may be estimated to be taken from the same camera position, with only the pan, tilt, and zoom as differences.The algorithm typically runs at 5-10 frames per second on a general-purpose computer but the simple structure of the algorithm makes it easy to implement in hardware for the higher frame-rates needed for full-motion video. Location Sensing While the VideoOrbits algorithm can robustly handle head-tracking, it still does not provide any indication of the wearer's overall spatial location. As previously mentioned, GPS units do not work in many locations and other external labelling or location sensing methods are impractical. Once again, a camera-based system is needed. This problem is inherently more difficult than the head-tracking problem. Nevertheless, several reasonably promising approaches have been discovered. One such approach, developed at the MIT media lab, and tested in a game setting, is described herein. Since the system makes use of training data and a fixed environment, it is not suited for general use, but it appears to be a promising step in the right direction. "Patrol" is a game played by MIT students every weekend in a campus building. The participants are divided into teams denoted by colored head bands. Each participant starts with a rubber suction dart gun and a small number of darts. The teams converge on the basement, mezzanine, and first floors to hunt each other. If shot with a dart, the participant removes his head band, waits for fighting to finish, and proceeds to the second floor before replacing his head band and returning. The Patrol environment consists of 14 rooms that are defined by their strategic importance to the players. The rooms' boundaries were not chosen to simplify the vision task but are based on the long standing conventions of game play. The playing areas include hallways, stairwells, classrooms, and mirror image copies of these classrooms whose similarities and "institutional" decor make the recognition task difficult. However, four of the possible rooms have relatively distinct coloration and luminance combinations, though two of these are not often traveled. For the location and context sensing task, players are outfitted with specially designed hats, each containing two video cameras; one facing forward, in the direction of the player's gaze, and one looking directly downward across the player's nose. The mean colors of three video patches are used to construct a feature vector in real-time. One patch is taken from approximately the center of the image of the forward looking camera. The means of the red, green, blue, and luminance pixel values are determined, creating a four element vector. This patch varies significantly due to the continual head motion of the player. The next patch is derived from the downward looking camera in the area just to the front of the player and out of range of average hand and foot motion. This patch represents the coloration of the floors. Finally, since the nose is always in the same place relative to the downward looking camera, a patch is sampled from the nose. This patch provides a hint at lighting variations as the player moves through a room. Combined, these patches provide a 12 element feature vector. Hidden Markov models (HMM's) were used to represent the environment due to their potential language structure and excellent discrimination ability for varying time domain processes. The previous known location of the user helps to limit their current possible location. By observing the video stream over several minutes and knowing the physical layout of the building, many possible paths may be hypothesized and the most probable chosen based on the observed data. Prior knowledge about the mean time spent in each area may also be used to weight the probability of a given hypothesis. HMM's fully exploit these attributes. To improve the current system, optical flow or inertial sensors could limit frame processing to those times when the player is moving forward. This would eliminate much of the variation, often caused by stand-offs and firefights, between examples of moving through a room. Similarly, the current system could be combined with optical flow to compensate for drift in inertial trackers and pedometers. Windowing the test data to the size of a few average rooms could improve HMM accuracies as well. Additionally, instead of the average color of video patches, color histograms could be used as feature vectors. The model created by the HMM location system above can also be used for prediction. For example, the computer can weight the importance of incoming information depending on where it believes the player will move next. An encounter among other players several rooms away may be relevant if the player is moving rapidly in that direction. In addition, if the player is shot, the computer may predict the most likely next area for the enemy to visit and alert the player's team as appropriate. Applications What makes the field of wearable computing so exciting is the myriad of potential applications. Several applications of wearable computers that utilize one or more of the techniques described will now be briefly described. HOLZER Homographically Obliterating Labels by Zeroing, Enhancement, or Replacement HOLZER is a diminished reality system developed by Steve Mann that filters billboards and other advertising material out of the user's visual field, replacing them with blank spaces or other, more pleasing, images. Useless, annoying and potentially dangerously distracting visual "noise" such as advertisements can be identified based on their general shape as 2-dimensional planes with the help of algebraic projective geometry and distinguished from other desirable planes using more sophisticated pattern recognition techiques (typically hidden markov models or neural networks). The visual real-estate that these advertisements take up can either be blocked out -- replaced with a featureless, solid plane -- or reused for the communication of more important information. Figures 2 and 3 demonstrate the HOLZER system in operation. holzer.gif 224 312 holzer 1 Figure 2: (a),(b) Billboards, advertising, and other visual detritus form annoying, and sometimes dangerous clutter at the sides of busy roadways and highways. holzer2.gif 224 312 holzer 2 Figure 3: (c),(d) Successive frames of video processed by the Eye Tap system using the VideoOrbits planetracker. The advertisement is filtered out, to reduce visual clutter in the scene. In its place is a useful message that can help the user of the Eye Tap system keep their attention on the road, and on not getting lost. (images from about.eyetap.org) A.Butz T.Höllerer S.Feiner B.MacIntyre C.Beshers Proc. IWAR '99 (Int. Workshop on Augmented Reality), San Francisco, CA, October 20-21, 1999 Enveloping Users and Computers in a Collaborative 3D Augmented Reality. 35-44 S.Feiner B.MacIntyre M.Haupt E.Solomon Proc. UIST '93 (ACM Symp. on User Interface Software and Technology), Atlanta, GA, November 3-5, 1993 Windows on the world: 2D windows for 3D augmented reality. 145-155 S.Feiner Proc. IWAR '99 (Int. Workshop on Augmented Reality), San Francisco, CA, October 20-21, 1999 The Importance of Being Mobile: Some Social Consequences of Wearable Augmented Reality Systems. 145-148 T.Höllerer S.Feiner J.Pavlik Proc. ISWC '99 (Third Int. Symp. on Wearable Computers), San Francisco, CA, October 18-19, 1999 Situated Documentaries: Embedding Multimedia Presentations in the Real World. 79-86 B.Horn B.Schunk Artificial Intelligence, 17, 1981. Determining Optical Flow. 185-203 T.Jebara C.Eyster J.Weaver T.Starner A.Pentland Proceedings of the International Symposium on Wearable Computers , Cambridge, Massachusetts, October 1997. Stochasticks: Augmenting the Billiards Experience with Probabilistic Vision and Wearable Computers S.Mann TR 361, M.I.T. Media Lab Perceptual Computing Section; Also appears in AAAI Fall Symposium on Developing Assistive Technology for People with Disabilities, 9-11 November 1996, MIT; http://wearcam.org/vmp.htm, Cambridge, Massachusetts, February 2 1996. Wearable, tetherless computer-mediated reality: WearCam as a wearable face-recognizer, and other applications for the disabled. S.Mann IEEE Computer; http://wearcam.org/ieeecomputer.htm, Feb 1997, 30(2) Wearable computing: A first step toward personal imaging. 25-32 S.Mann R. W.Picard TR 338, Massachusetts Institute of Technology, Cambridge, Massachusetts, See http://hi.eecg.toronto.edu/tip.html 1995. Video orbits of the projective group; a simple approach to featureless estimation of parameters. S.Mann Proceedings of the IEEE, 86(11), Nov 1998. Humanistic intelligence/humanistic computing: `wearcomp' as a new framework for intelligent signal processing. 2123-2151+cover S.Mann {IEEE} Trans. Image Proc. Comparametric Equations with Practical Applications in Quantigraphic Image Processing 1389-1406 R. Y.Tsai IEEE Trans. Accoust., Speech, and Sig. Proc., ASSP(29), December 1981. Estimating Three-Dimensional Motion Parameters of a Rigid Planar Patch I. 1147-1152 A.Webster S.Feiner B.MacIntyre Proc. ASCE Third Congress on Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996 Augmented reality in architectural construction, inspection and renovation. 913-919