Using Augmented Reality to Visualise Architecture Designs in an Outdoor Environment

Bruce Thomas, Wayne Piekarski, and Bernard Gunther

University of South Australia

Mawson Lakes, SA 5095

Australia

{thomas, piekarski, gunther}@cs.unisa.edu.au

Abstract

This paper presents the use of a wearable computer system to visualise outdoor architectural features using augmented reality. The paper examines the question -

How does one visualise a design for a building, modification to a building, or extension to an existing building relative to its physical surroundings?

The solution presented to this problem is to use a mobile augmented reality platform to visualise the design in spatial context of its final physical surroundings. The paper describes the mobile augmented reality platform TINMITH2 used in the investigation. The operation of the system is described through a detailed example of the system in operation. The system was used to visualise a simple extension to a building on one of the University of South Australia campuses.

1 Introduction

The advent of wearable computers (Bass et al. 1997; Mann 1998; Thorp 1998) and light-weight head mounted displays (HMDs) has made it feasible to use augmented reality (AR) applications outdoors. AR is the process of a user viewing the physical world and virtual information simultaneously, where the virtual information is overlaid and aligned with the physical world view (Azuma 1997; Starner Schiele, & Pentland 1998). Many of the existing applications of AR, such as heads-up displays in aviation, surgery and maintenance work, are characterized by requiring precise tracking in small operating regions.

However, through coupling global positioning system (GPS) receivers and digital compasses with 3D graphical models, we can create spatially aware computer systems for mobile users working outdoors. We anticipate outdoor users wishing hands-free operation, and thus related AR applications are especially well supported by wearable computers and non-traditional input devices. An interesting application of these AR enhanced computer systems is the visualisation of new architectural designs. This visualisation is performed at the site where the architectural design will be built. This paper presents a wearable computer system TINMITH2 to support such visualisation.

1.1 The Problem

How does one visualise a design for a building, modification to a building, or extension to an existing building relative to its physical surroundings? In the past, technical drawings would have been made and models built. With the introduction of Computer Aided Design (CAD) packages, this process has been extended to visualise the design of the building fully rendered as a 3D graphical model on a graphics workstation. Changes can be made while the customer is in the design studio and the outcome of decisions can be visualised during this process. With the advent of Virtual Reality (VR), more ambitious visualisations were made possible. VR enables customers and designers to view a design in an immerse environment (Brooks 1986; Mine & Weber 1995), with the use of a VR head-mounted display. People are placed in a simulation and simulate a walk-through the new design. They can visualise and naturally move through the layout of the building in 3D. Tracking the user's head allows for intuitive movement of their head to change the viewing direction. Treadmills allow users to move by walking through a design while still physically inside the design studio. Together, tracking and treadmills allow users to sense the size and position of features in a new design.

However, how can a user place a new building or extension in context with the existing surrounds? Digitally enhanced photographs can show the placement of a building with respect to one vantage point. Models may be built to provide more vantage points, but these are expensive and time consuming to create, and offer no better than an artificial rendition of the site.

1.2 The Solution

One solution is to allow a user to walk around the site where the new building is to be constructed and visualise this new artefact in the spatial context of the existing environment. AR may be employed as a technique to provide this visualisation.

AR has been used before in visualising interior design information. Webster et al. (1996) are developing AR systems to improve methods for the construction, inspection, and renovation of architectural structures. Their initial experimental AR system shows the location of columns behind a finished wall, the location of re-bars inside one of the columns, and a structural analysis of the column.

Like other researchers, Azuma (1999) and Feiner et al. (1997), we are taking this use of AR from the indoor setting and placing it in the outdoor environment. The TINMITH2 system has been employed as a mobile AR platform to display architectural designs in an outdoor environment.

1.3 Structure of the Paper

The paper is broken down into five sections as follows:

Section 2: a description of the use of mobile AR platforms,
Section 3: an overview of the role of our mobile AR system in the design process,
Section 4: through the use of an example, a description of the operation of the system,
Section 5: a look at some of the important hardware/software implementation issues, and
Conclusion: concluding remarks of using AR to visualise architecture designs in an outdoor environment, with some thoughts on future directions of research.

2 Using a Mobile Augmented Reality Platform

We are investigating the exploration and development of computer technology that truly takes computers out into the field, where computer applications are geographically aware and designed to interact with users in their world, not just in the confines of the computer's artificial reality (Piekarski et al. 1999a; Piekarski et al. 1999b). The key to making this practical is augmented reality technology. Figure 1 depicts how AR works: the user's normal visual stimulus of the physical world is combined with computer generated images. One form of an optical combiner may be a half-silvered mirror; the half-silvered mirrors are embedded in the HMD along with an image projection device to supply the computer generated images, a see-through HMD.

Figure 1. How augmented reality works.

Unlike VR, where the computer generates the entire user environment, AR places the computer in a relatively unobtrusive, assistance role. Using a wearable computer with a see-through HMD allows people to move freely while working. Through the use of GPS technology the computer gains an additional and important input, the user's location, and thus computer applications gain spatial awareness that remains synchronised with their own awareness.

The power of AR systems lies in their ability to help us visualise normally hidden or abstract features, such as pipes and boundaries, respectively, as shown in the underground cable example depicted in Figure 2. By providing information in a 3D form, in scale with surroundings, AR systems provide significant benefits:

objects can be located more rapidly, especially in featureless terrain, thus saving time and costs;
the location of an object can be determined with accuracy;
previously invisible features, such as boundaries, become visible without the use of physical markers;
overlaying more than one information source allows the relationship between objects to be determined easily; and
features can be viewed from orientations that are more appropriate to the task than a map or drawing may allow.

Figure 2. Example of visualising underground cables.

Our first application of this technology was to develop a mobile AR user interface for terrestrial navigation (Thomas et al. 1998; Piekarski et al. 1999a). A second application was integrating a stationary VR system, MetaVR (1999), with a mobile AR system (Piekarski, Gunther & Thomas 1999), to support collaborative tasks (Thomas & Tyerman 1997). This application facilitates potentially two way interaction: communication of 3D model information to the mobile user, and, conversely, detailed information back to the VR system for updating and modifying its 3D model (Bauer et al. 1998; Carlsson & Hagsand 1993; Funkhouser 1995).

3 Role of a Mobile Architectural Visualisation System in the Design Process

Our system helps people visualise architectural designs in their physical outdoor context. The mobile system we have built was designed to meet the following objectives:

Architectural designs should originate from standard CAD packages and be stored in standard interchange file formats;
Architectural designs will be displayed relative to their physical site placement;
The entire system will be developed with off-the-shelf technology; and
The user interface must be easy and intuitive to use.

Our system must conform to current architectural design methodologies. The first being, the system must be able import information from standard architectural design software packages. The core of the system is the ability to visualise or see aspects of the architectural design in the field, providing the user with a feeling of how the architectural artefact will fill or change the physical space. The targeted end users of the system are architects, engineers, designers, and clients.

4 Operation of the System

As a means to describe the operation of the system, we are going to present the steps of preparing, installing, and visualising a simple extension to a building on the Levels Campus of the University of South Australia. The model of the external features of the original building is shown in Figures 3, 4, and 5. The current system is able to display the entire model as shown in these figures in real time. The extension will be represented as an outlined 3D rectangular cube, shown in the bottom left of Figure 6. The major steps of preparing, installing, and visualising the simple extension are:

drafting the new design,
transferring the new design to TINMITH2,
3D augment reality viewing,
registration of AR images, and
2D overview to help with alignment.

Figure 3. Top down view of the original exterior building design.

Figure 4. Side view of the original exterior building design.

Figure 5. Bird's eye view of the original exterior building design.

4.1 Drafting the New Design

We designed the extension using AUTO-CAD for Windows95, but any CAD package which exports the Drawing Interchange Format (DFX) may be used. DFX files enable the interchange of drawings between AutoCAD and other programs (Minnesota CADWorks 1997). The new design includes external features of the new extension and key features of the existing architecture. Figures 6, 7, and 8 depict the extension and key external features of the building near the location of the extension. The extension is the the square addition to the building at the bottom of Figure 6.

We chose to include only key features in the AR image, because using the entire building may cause confusion when viewed as an AR image. Informal testing has shown complex AR images are more difficult to align with buildings. We believe this is caused by the presentation of a very complex pattern (the line drawing of the design) to be matched with the physical building. A common first strategy for users to overcome this problem is to attempt to identify unique features of presented in the AR image and match these features with the same unique features of the building. To simplify the visualisation task for the user, we only used a small portion of the entire building with which to establish an easier to use frame of reference.

Figure 6. Top-down view of the new extension and key features and extension design.

Figure 7. Side view of the new extension and key features and extension design.

Figure 8. Bird's eye view of the new extension and key features and extension design.

When drafting the design, we first enter the new extension into a copy of the model shown in Figure 5. Once this had been performed, the features which are deemed not to be key features of the existing building are erased from the model. The CAD model now has the graphical images which will be depicted via the see-through display as AR information. The model is then exploded, that is to say, grouped objects and facets are converted into single lines. We expect that the next generation of system, TINMITH3, will be able to render groups of objects and facets. The line model is then saved in the DFX file format.

4.2 Transferring the New Design to TINMITH2

TINMITH2 first stores 3D models as 3D lines in a PostGreSQL (1998) database. The 3D DFX model of the extension is processed to build a script which defines a set of SQL insertion commands to load the model into the PostGreSQL database. Using a database engine allows other forms of data to be attached to different features of the model by extending the relations in the database schema. The script is then executed on the TINMITH2 system, and the model is inserted into the database.

4.3 3D Augmented Reality View

We will first describe viewing the AR images, and then describe some techniques to overcome the errors introduced into the system by the electronic compass and the differential GPS. The section presents the alternative 3D and 2D interfaces the system provides for the user.

4.3.1 HMD View of Augmented Reality Images

Figure 9 depicts what a user sees while viewing the design through the HMD. The 3D immerse display superimposes the design on the wearer's field of view. When looking through this display, the user is able to register the display with the outline of the key features of the building within the accuracy limits of the differential GPS and digital compass. The figure depicts (as green lines) the outline of the key features of the existing building.

Figure 9. Augmented reality view of the building design.

The proposed extension is also depicted with green lines. The user is able to walk around the proposed extension to gain a feel of size and shape of the extension. This form of visualisation enables the user to determine the overall context of the new extension. Issues such as the extension blocking views or placement near unmarked land features scan be quickly assessed. Multiple plans may be viewed in sequence or simultaneously to compare and contrast different design ideas. The simple cube design presented as an example may be modified to include external features such as windows, doors, ledges, and over-hanging roofs.

Although the current version of TINMITH2 does not support full 3D rendering or the use of texture maps to provide a more realistic view of the extension, these modifications are currently being implemented in the next version of the system. Other possible modifications could include animation of articulated parts of the building (doors swinging, people walking, or windows opening), and implementing a model of sunlight to depict shadows during the day.

4.3.2 Registration Errors in Outdoor Augmented Reality

A known problem is that the alignment of the overlaid 3D graphics in context with the physical world can not be guaranteed to be accurate at all times. The GPS used in this system is moderately priced and accurate to one to two metres in open spaces, while the digital compass is accurate to 0.5 degrees. In operation, we were able to maintain approximately one to five metre accuracy given a good fix on six or more GPS satellites. The accuracy of the GPS is determined by the number and position of the visible satellites. Standing next to a building can and normally will reduce the accuracy of the positioning information, and in our case the accuracy would drop to 3 to 5 metres, making the registration of the design with the physical building difficult. For example the building is on the order of ten metres tall, and standing five metres away from where the system believes the user is standing will cause a large scaling distortion.

To overlay the information in the correct manner, the system must be able to compensate for errors in its position and orientation information. TINMITH2 provides a simple but robust solution to the problem of visual registration in AR, the first step in achieving practical 3D data visualisation.

4.3.3 Correcting Registration Due to Errors in Angle Information

Our solution to the registration problem due to errors in the angle information is to manually align a number of key features from the building with the virtual images. Once the user is viewing the existing building from their desired position, the digital compass and GPS systems are turned off; thus freezing the position and orientation of the image of the design on the HMD. As shown in Figure 10, the virtual and physical roof lines are being aligned. (The virtual roof line in the figure is slightly lower than the physical one.) This allows the user to adjust their head position until the key features are properly aligned. Typically, this can be accomplished in a matter of seconds.

Figure 10. Second view the design.

4.3.4 Correcting Registration Due to Errors in Position Information

The 2D interface of TINMITH2 incorporates a first person perspective, a 2D map view, and traditional non-spatially aware information on one display, as shown in Figure 11. To overcome problems due to large errors in position information (+/- five metres), the user accesses the 2D map to determine a more accurate fix of their position. As before, the user firsts positions the design to display in the centre of the HMD, and then turns the GPS and compass systems off. The user then moves to the position where the system believes they are standing. Using the 2D map of the footprint of the building, the user may accurately determine where the system believes they are standing. The current position of the user is indicated by the end of a red line, as shown in Figure 12. The user moves to the position relative to the design shown in yellow. The 2D map view may zoomed in or out help obtain a more accurate fix on the user's location. Figure 13 indicates the user's position relative to the building's features.

Figure 11. Overview of the 2D map.

Figure 12. Close up of the positioning information on the 2D map.

Figure 13. User's position relative to the building's key features.
The various pieces of textual information are placed around the display to show position, GPS accuracy, date and time, and as in Figure 14. Shown in the figure are outlines of objects in the environment. The building within the blue circle is the footprint of the building shown in the photo. The second model depicted in the figure indicates a building behind to the left of the user. Every visual cue displayed by the HMD is rotated in real time as the user rotates their head.

Figure 14. Information provided to the user via the HMD.

5 The TINMITH2 Wearable Computer System

Our wearable computer system, TINMITH2 (Piekarski et al. 1999a), a is a mobile AR platform developed at the Wearable Computer Laboratory in the University of South Australia. Figure 15 pictures the current hardware platform.

Figure 15. TINMITH2.

5.1 Hardware Components

TINMITH2 is built upon a Toshiba 320CDS notebook running the freely available LinuxOS. The laptop is about the size of an A4 book. A Sony PLM-100 transparent display, worn on the user's head as shown in Figure 16, allows the video output of the computer to superimpose images over the real world. A Phoenix miniature keyboard attached to the forearm enables the user to interact with the system and enter commands.

Figure 16. Using a Sony PLM-100 transparent display.

To provide positioning information for our application, a GPS module (with differential receiver) connects to the laptop. A TCM2 three-axis digital compass is attached to the display positioned on top of the user's head, to allow the computer to determine the orientation of the wearer's head. All of the equipment is attached to a rigid backpack, along with batteries and antennae. The prototype hardware and software system is fully functional in outdoor environments.

5.2 Implementation of the System Architecture

To support a wide range of AR applications, we developed a highly modular architecture. The software system is broken up into various modules that communicate with each other using the connection oriented TCP/IP protocol.

5.2.1 Communications

To interconnect modules, we used a client-server style architecture. The server is a data source for other modules, which subscribe to it. The entire system operates asynchronously and is data driven; if there is no new data in the system, no action will be taken by any of the software modules. Each of the modules are implemented as separate Unix processes, which communicate via kernel network services. This allows modules to be distributed over multiple processors on one or more machines.

5.2.2 Dynamic configuration from a DBMS

Most software tends to use statically compiled controls, or possibly a configuration file, but our system loads all system parameters such as the 2D maps, 3D object models, location of modules, port numbers, device names, and screen configuration into a collection of relational database tables. When the software initializes, it queries the database and loads the values required. The database proved to be very powerful because it can be changed remotely over the wireless network. This feature was found to be useful when testing outdoors, for example, for tuning the various display options such as colours and font sizes.

6 Conclusion

In conclusion, this work has shown augmented reality can be used to visualise architecture designs in an outdoor environment. A design of an extension to an existing building on the Levels campus of the University of South Australia was successfully depicted with our mobile augmented reality platform. Through informal testing, the system was shown to provide the user with a sense of space and feeling of the size and location for the extension. As an example of gaining a feeling for a design, the simple extension was designed with a height of only 3 metres. The informal testing showed this design flaw immediately. It is not suggested a professional designer would not detect such a design flaw. We are suggesting novices to the field of design might not see flaw on a CAD system, but they would be able to detect it out in the field.

Known problems of registration can be effectively overcome through allowing the user to use their natural ability to align similar visual patterns together.

We are going the continue to investigate the visualisation of architecture designs. Features we are investigating to incorporate into the next generation of system, TINMITH3, are as follows:

full 3D rendering of design models, with texture mapping,
rendering groups of objects and facets,
incorporate additional file formats,
animation of articulated parts of the building (doors swinging, people walking, or windows opening), and
addition of a model of sun light to depict shadows during the day.

The research issues which need to addressed to effectively use augmented reality in an outdoor setting are as follows:

These systems require accurate and timely tracking in the outdoors. The tracking with limited infrastructure in an outdoor environment is a challenging problem.
These systems potentially will use low bandwidth networks. The network and distributive problems are being investigated but many of these solutions assume a traditional laboratory setting. Wireless computing is looking at issues of providing Internet like capabilities to the user on the street, or even the outback. The bandwidth requirement for sending digital images, audio, and video signals is still higher than current technology can provide.
A wearable computer requires a computing system with a set of limited resources. The HMDs require the use of small displays. The local secondary storage and computing power are both limited.
An augmented reality interface requires a novel form of information presentation. Traditional user interface technology is inadequate for seethrough or small above the eye displays. In the case of seethrough displays, a major portion of the screen should not occlude the physical world. There is a need for more information with less pixels. This is possible due to the fact that the system's task is to augment the user's field of view, and not to provide all the contextual information. The registration of overlay images on a user's display is a key issue. We envision a traditional laptop computer to cope with tasks such as word processing.
The outdoor environments limits choice of user input devices. With the user in an outdoor environment, many input devices designed for an office environment are no longer functional. Hands free viewing of the information space allows the user to perform different tasks while viewing the information in their HMD.

Acknowledgements

The authors would like to acknowledge the assistance of Victor Demczuk and Franco Principe of Land Operations Division at Defence Science and Technology Organisation, Salisbury, South Australia. We would like to thank Dan Makovec for technical assistance. Finally thank you to Judy Thomas and John Marist for proof reading our paper.

References

Azuma, R. 1997, `Survey of augmented reality', Presence: Teleoperators and Virtual Environments, vol. 6, no. 4.

Azuma, R. 1999, `The challenge of making augmented reality work outdoors', in Mixed Reality: Merging Real and Virtual Worlds, Springer-Verlag, chap. 21, pp. 379-390.

Bass, L., Kasabach, C., Martin, R., Siewiorek, D., Smailagic, A. & Stivoric, J. 1997, `The design of a wearable computer', in CHI'97 Looking to the Future, ACM SIGCHI, pp. 139-146.

Bauer, M., Heiber, T., Kortuem, G., & Segall, Z. 1998, `A collaborative wearable system with remote sensing', in Proceedings 2nd. Int'l Symp. on Wearable Computers, pp. 10-17, Oct.

Brooks, Jr., F.P. 1986, `Walkthrough-A Dynamic Graphics System for Simulating Virtual Buildings'. 1986 Workshop on Interactive 3D Graphics. S. Pizer and F. Crow, eds. University of North Carolina at Chapel Hill, Oct.

Carlsson, C. and Hagsand, O. 1993, `Dive a platform for multiuser virtual environments', Computers and Graphics, pp. 663-669.

Feiner, S., MacIntyre, B., Hollerer, T. and Webster, A. 1997, `A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment', in Proceedings 1st. Int'l Symp. on Wearable Computers, pp. 74-81, Oct.

Funkhouser, T.A. 1995, `Ring: A client-server system for multiuser virtual environments', in Computer Graphics, pp. 85-92, April.

Mann, S. 1998, `WearCam (The Wearable Camera): Personal imaging systems for longterm use in wearable tetherless computer-mediated reality and personal photo/video-graphic memory prosthesis', in Proceedings 2nd. Int'l Symp. on Wearable Computers, pp. 124-131, Oct.

MetaVR, Inc. 1999 MetaVR Virtual Reality Scene Generator http://www.metavr.com, .

Mine M. and Weber, H. 1995 Large Models for Virtual Environments: A Review of Work by the Architectural Walkthrough Project at UNC. Presence vol. 5, no. 1, pp. 136-145.

Minnesota CADWorks 1999 , Incorporated, AutoCAD Release 14 DXF File Format, http://www.mcwi.com/dxf14/dxf14_000.html.

Piekarski, W., Hepworth, D., Demczuk, V., Thomas, B. and Gunther, B. 1999a, `A mobile augmented reality user interface for terrestrial navigation', in Proc. 22nd Australasian Computer Science Conference, Auckland, NZ, pp 122-133, Jan.

Piekarski W., Thomas B., Hepworth D., Gunther B., and Demczuk V. 1999b. `An Architecture for Outdoor Wearable Computers to Support Augmented Reality and Multimedia Applications', Proceedings of the Third International Conference on Knowledge-Based Intelligent Information Engineering Systems, Aug., Adelaide.

Piekarski W.,Gunther B. , and Thomas B. 1999, `Integrating Virtual and Augmented Realities in an Outdoor Application' Proceedings of the Second IEEE and ACM International Workshop on Augmented Reality (IWAR) '99 Oct., San Francisco, CA (USA)

PostGreSQL 1998, v6.4.2 Database Engine http://www.postgresql.org, .

Starner, T., Schiele, B. and Pentland, A. 1998, `Visual contextual awareness in wearable computing', in Proceedings 2nd. Int'l Symp. on Wearable Computers, pp. 50-57, Oct.

Thomas, B., Demczuk, V., Piekarski, W., Hepworth, D. and Gunther D. 1998, `A wearable computer system with augmented reality to support terrestrial navigation', in Proceedings 2nd. Int'l Symp. on Wearable Computers, pp. 168-171, Oct.

Thomas, B. and Tyerman, S. 1997, `Collaboration issues for augmented realities in an outdoor environment'. in Proceedings INTERACT97 combined Workshop on CSCW in HCI, Sydney, pp. 29-33, Jan.

Thorp, E. 1998, `The invention of the first wearable computer', in Proceedings 2nd. Int'l Symp. on Wearable Computers, pp. 4-8, Oct.

Webster, A., Feiner, S., MacIntyre, B., Massie, B. and Krueger, T. 1996, `Augmented reality in architectural construction, inspection and renovation', in Proc. Third ASCE Congress for Computing in Civil Engineering, Jun. 17-18 Anaheim, CA