Paper Reading #2: MirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop

Introduction

MirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop was coauthored by Hrvoje Benko, a researcher at Microsoft Research’s Natural Interaction Research group and Columbia University graduate, Ricardo Jota, a post-doctoral fellow at the University of Toronto working under Daniel Wigdor, and Andrew D. Wilson, a principle researcher at Microsoft Research and MIT graduate.

Summary

“MirageTable is an interactive system designed to merge real and virtual worlds into a single spatially registered experience on top of a table.” Using a depth camera (Kinect), a stereo projector, a stereo sync emitter, a curved screen, and shutter glasses, MirageTable is able to provide a seamless 3D AR experience that not only allows users to interact with virtual objects, but also allows the user to scan real objects into the virtual world. This is all done without gloves, wearable trackers, or other cumbersome gear. To demonstrate the capabilities and limitations of their system, the authors devised “three application examples: virtual 3D model creation, interactive gaming with real and virtual objects, and a 3D teleconferencing experience that not only presents a 3D view of the remote person, but also a seamless 3D shared task space.” MirageTable’s ability to provide a correct 3D perspective view, ability to acquire and relay mesh data in real-time, and ability to perform high-fidelity physical interactions with virtual objects based on both virtual and real geometry - such as the user’s hand, a virtual block, or a real book - combine to provide a high-quality, unique experience. 

MirageTable projects a correct 3D perspective using head tracking (done by the Kinect) and stereoscopic projective texturing. Projective texturing allows 3D virtual objects to be correctly placed in the scene alongside real objects by accounting and correcting for occlusions. This is done by rendering the scene from the perspective of each of the user’s eyes, taking into account real geometry as well as virtual content, and projecting these renderings onto captured real-world geometry. Then, a second rendering is performed for each eye from the perspective of the projector. The result is a 3D virtual image that appears correct from the eye of the user.

Acquisition and replay of mesh in real-time was achieved using a depth camera as a continuous 3D digitizer and a custom vertex shader run on a GPU. They did not restrict digitization to anything specific, but instead capture anything that occupies physical space. While replay of the mesh constructed by the depth camera in real-time was not challenging due to advanced GPU technology, mesh acquisition was limited to what parts of the real-world object were visible to the camera. In other words, MirageTable makes a great 3D mirror, but is limited in its ability to fully digitize a physical model without rotating the object, using mirrors, or expanding the system to multiple cameras.

MirageTable aimed to minimize the differences between real and virtual objects within the system by simulating realistic physical interactions that extend from the real world to the virtual space. This is where MirageTable falls short due to technological limitations. Depth cameras aren’t able to accurately infer grasping forces, real-time deformable geometry is too computationally complex, and the limitations of mesh acquisition extend to this capability as well. The authors did, however, approximate the physics of captured geometry using proxy particles and Nvidia’s PhysX game engine.

Related Works

The authors freely admit that no singular component of their system is particularly novel, but that the novelty of their experiment lies in the successful blending of components to create a high-quality system unlike any other ever implemented. Their contributions consist of their “system design and implementation, three prototype applications, and a user study on 3D perception and image quality in [their] system.” The authors were primarily influenced by two projects conducted prior to their experiment: Office of the Future and LightSpace. In addition to these projects and the works citied in their paper, the following un-cited related works help place MirageTable in the context of other research into artificial reality and related topics:

1. A Survey of Augmented Reality by Ronald T. Azuma
2. Recent Advances in Augmented Reality by Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, & Blair MacIntyre
3. Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System by Hirokazu Kato & Mark Billinghurst
4. Efficient Model-based 3D Tracking of Hand Articulations using Kinect by Iason Oikonomidis, Nikolaos Kyriazis & Antonis A. Argyros
5. Human Detection Using Depth Information by Kinect by Lu Xia, Chia-Chih Chen & J. K. Aggarwal
6. The Studierstube Augmented Reality Project by Dieter Schmalstieg, Anton Fuhrmann, Gerd Hesina Zsolt Szalavári, L. Miguel Encarnação, Michael Gervautz & Werner Purgathofer
7. Collaborative Augmented Reality by Mark Billinghurst & Hirokazu Kato
8. CounterIntelligence: Augmented Reality Kitchen by Leonardo Bonanni, Chia-Hsun Lee & Ted Selker
9. Perceptual Issues in Augmented Reality by David Drascic & Paul Milgram
10. MIND-WARPING: Towards Creating a Compelling Collaborative Augmented Reality Game by Thad Starner, Bastian Leibe, Brad Singletary & Jarrell Pair 

A Survey of Augmented Reality is an older study conducted in 1997 that covers a wide variety of AR applications explored at the time of writing, but most relevant is the paper’s discussion of the future of AR. The paper states that the two most pressing problems in the field of AR are registration and sensing and that future approaches to addressing these issues will incorporate perceptual studies and real-time computing. MirageTable certainly fits this description and effectively meets the challenges of registration and sensing in a novel way. Recent Advances in Augmented Reality is a followup study conducted in 2001 to update A Survey of Augmented Reality. This study greatly enhances the first study by addressing the rapid technological advancements that occurred just prior to the 21st century. In doing so, the authors realize that the problems faced by the future of AR are technological limitations, user interface limitations, and social acceptance. The advancements from 1997 to 2001 did wonders to improve AR and the authors recognize that future advancements will likely do the same - and they obviously have in the case of MirageTable.

Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System is a study conducted in 1999 that addresses the same issue as MirageTable’s teleconferencing application in a similar way. MirageTable utilizes many advancements made since 1999 to accomplish a much more robust solution without the use of cumbersome headgear - note that the shutter glasses used in MirageTable’s system are only necessary if the user desires a 3D experience. Efficient Model-based 3D Tracking of Hand Articulations using Kinect is a study that utilizes the capabilities of Microsoft’s Kinect to conduct marker-less hand articulation tracking - a more complex application of Kinect’s capabilities than MirageTable’s shutter glasses tracking. Human Detection Using Depth Information by Kinect explores methods of conducting human-tracking using Kinect hardware. Both of these studies expand on the technical and computational challenges associated with real-time depth tracking - a challenge very central to MirageTable’s functionality that isn’t covered in detail in the original paper.

The Studierstube Augmented Reality Project was conducted in 2002 with the goal of creating a 3D user interface metaphor for augmented reality as powerful as 2D’s ubiquitous desktop metaphor. This project, much like MirageTable, uses projection and 3D elements to approach the task of providing a real world solution to collaborative AR computing systems and was influenced by the Office of the Future project at UNC. Collaborative Augmented Reality, another paper by Billinghurst & Kato, was also conducted in 2002 with the same basic goal as MirageTable: to blend reality and virtuality to allow users to see each other alongside virtual objects. Unlike related works previously discussed, these works run parallel to MirageTable and could be seen as an alternative approach to the same basic challenge. These papers provide a good context for what MirageTable might have been were it developed ten years ago and demonstrate the field of AR’s ongoing interest in developing collaborative 3D systems.

CounterIntelligence: Augmented Reality Kitchen is an AR system based on projection of information onto novel surfaces, much like MirageTable uses curved surfaces to project a seamless artificial surface extending from a real desk. While this paper’s testing scenario and implementation are not directly useful in relation to MirageTable, this paper does serve to highlight the significant contribution that is MirageTable’s user study and other potential applications and extensions of MirageTable’s projection capabilities. Another paper that emphasizes the importance of testing user perception in AR applications is Perceptual Issues in Augmented Reality, a paper written in 1996 that explores the challenges of displaying AR data in graphics in relation to depth perception and stereoscopic imaging. Both of these issues are specifically addressed in MirageTable’s user study of 3D perception and it’s effect on image quality and subsequently, user preference.

Finally, we look at MIND-WARPING: Towards Creating a Compelling Collaborative Augmented Reality Game, another paper that addresses one of the challenges taken on by MirageTable - AR gaming. MIND-WARPING addresses many of the same systematic problems faced by MirageTable: user perception, novel input collection, and collaborative functionality. This paper tackles these issues well, but not as successfully as MirageTable’s implementation. This serves to support the authors of MirageTable’s assertion that their overall system is novel due to its quality and successful incorporation of tested ideas into a novel implementation.

Evaluation

Now that MirageTable has been placed in context with previous related work, it is appropriate to address the methods used to evaluate its success. The authors developed three prototype applications to test the overall system as a whole and to assess the MirageTable’s real-world viability. In addition, a detailed user study of projective texturing quality and perception was done to access the success of their novel application on projectors and stereoscopic imaging.

Virtual 3D model creation was a viable application that allowed real-world objects to be scanned, copied, and manipulated in the virtual space. Testing done with architects demonstrated that MirageTable was able to construct complex virtual models despite the mesh acquisition and interaction limitations previously discussed. Their gaming application was similarly successful. 3D shape approximation was utilized to overcome some of the system’s limitations and allow for passable gameplay, but not without flaws and only with approximately symmetrical objects. The teleconferencing application was the most successful application. The curvature of the screen created a seamless environment which, coupled with 3D imaging and a shared virtual workspace, provided a truly unique experience at or near consumer product quality. All testing done using the prototype applications was informal and only served to demonstrate the capabilities of their system as a whole, the only quantitative measures taken concerned the accuracy, quality, and perception of MirageTable’s 3D projections.

Two experiments were performed to assess MirageTable’s projection capabilities: an evaluation of image quality degradation and an evaluation of user’s depth perception. First, the authors assessed how projected 3D images were impacted by a variety of irregular surfaces using RMS testing. They found that geometric distortions lead to relatively small differences in projection, indicating the success of their texturing technique. Color played a much larger factor than geometry in image distortion. The second experiment was conducted to assess user’s perceptions of the results found in experiment one.

Users viewed virtual balls on a variety of backgrounds and textures and made judgement ratings of the depth of each ball using a tape attached to the table. Overall, six conditions and four depths were analyzed with four repetitions for each participant. The results showed that color played a negligible factor in affecting user’s perception and that the base case resulted in an accurate depth assessment as well. The only two scenarios in which user perception was seriously impacted were the drop (surface at two different levels) and wave (surface riddled with peaks and valleys) conditions. While 3D perception is possible on geometrically distorted backgrounds, it can be less accurate depending on the geometry.

Discussion

I really enjoyed this paper. MirageTable is a very innovative system that combines the best of many other attempts at creating 3D AR. The individual components of their system lack in novelty, but the system as a whole is quite unique. This comes across in their evaluation, which is almost entirely limited to informal systematic testing. Many of the individual components of MirageTable have already been devised and tested, and since their goal is to create a system that is viable for the real-world, this approach is valid. The exception to this is their projective texturing technique, which is a novel approach to projector-based AR and they do test this contribution in detail. MirageTable’s only failings are from lack of technological capability - I would be interested to see what a new implementation of MirageTable would be like in years to come.