Augmented Reality Applications. Challenges and Future Trends

Augmented Reality Applications. Challenges and Future Trends Augmented Reality (AR) Abstract Augmented reality presents a mix of real world information and additional computer generated information to a user. Such a mix of information is usually presented by superimposing two distinct sets of images on to a monitor or a head mounted display which enables a viewer to observe a mix of the real world and computer generated text or images. Even though the visual sense is not the only sense which can be used to present and perceive computer altered reality, visual presentations are the most used in augmented reality systems. The additional presentations that are generated by a computer can substantially assist users in a number of ways including provision of textual information about what is being viewed, presentation of hidden information or images associated with a visual scene as well as the results of planned alterations to reality such as the presentation of the results of a construction or a demolition of physical structures. Augmented reality may be considered as being a b ranch of virtual reality research, but unlike virtual reality, in which everything that is presented to a user is generated by a computer, augmented reality only presents an alteration to the reality that is being sensed. There are very many interesting applications of augmented reality which are proving to be useful to human users. From assisting surgeons to perform delicate surgery, helping archaeologists collect excavation data to assisting rescuers after a natural disaster or aiding soldiers in urban combat, augmented reality systems are achieving even greater success as technological advances on a broad front enable even more sophisticated systems to be designed. Miniaturisation and the design of rugged, low power consumption components have resulted in portability. This essay presents a discussion of the potential and the problems of augmented reality systems. Contents (Jump to) 1. Introduction 2. A Description of the Operation of an Augmented Reality System 3. The User Benefits of an Augmented Reality System 4. Portable Augmented Reality Systems in Wearable Computers 5. The Outdoor Use of an Augmented Reality System 6. The Potential Sources of Error in an Augmented Reality System 7. Conclusion References / Bibliography List of Figures The Generation of Visual Augmented Reality using Head Mounted Displays, Cameras and Diverters for the Mixing of Real and Virtual Spaces Managing Multiple Spaces for Generation of Augmented Reality Sources of Error in Augmented Reality Systems 1. Introduction Augmented reality refers to the utilisation of computers to modify reality, usually in order to provide additional assistance when a human user is interacting with the real environment. The real world around humans provides a wealth of information which the human user must absorb and process through their senses. The most useful and informative of all human senses is the sense of vision and a huge amount of information about the ambiance is required to be sensed as well as processed by the human visual system. Computers are useful because they can provide an overlay of information to assist with the human processing of the information which they perceive through their senses, mostly the visual sense. [1] As an example, a human who is walking around an urban area can be provided additional information through a head mounted display which superimposes textual information about identification of buildings and other important landmarks so that the human is provided additional guidance ab out identification of buildings and landmarks or other useful information about what they are viewing. [2] Obviously, the computer too has to sense the environment in order to provide any additional information and this is done through a camera that captures the same view as the human. The camera provides images which are identified by a computer and virtual scenes are created in the virtual world generated by a computer. The computer program then generates additional assistive information that is presented to the human user in order to assist them in their interactions with their environment. Augmented reality software program is required to process information which a computer senses through sensors and the real space is transformed into a virtual space within the computer, with computer software identifying, recognising and transforming the external inputs from the real world in order to transform these inputs into assistive outputs for the user of an augmented reality system. Au gmented reality can, therefore, be considered to be a variation of virtual reality which completely immerses a user inside a synthetic environment. Augmented reality makes it possible for a human user to view the world with virtual objects that are generated by a computer and then superimposed or merged with it. The characteristics of virtual reality systems include a combination of the real and the virtual, interactions in real time and registration in 3 – D. [3] Augmented reality has proven itself to be useful in a wide variety of applications. From assisting surgeons to perform delicate surgery to assisting soldiers in battlefield or helping with education as well as assisting archaeologists gather field data on a site, virtual reality has proven to be of great benefit to humans in a very wide range of applications. [4] However, virtual reality technology is still evolving and with advances in computer science, even greater sophistication is likely to be possible in a large number of applications. Advances in wearable computing, software, miniaturisation, display technologies as well as sensors and radio networking have had a profound impact on the application of augmented reality. [5] Augmented reality systems are now far less bulky, portable and can be worn by a human user who may want to be assisted in a variety of ways in order to perform a task. Systems making use of augmented reality are now often used outdoors in harsh weather condit ions. Despite the possibilities, virtual reality implementations do have some limitations as well as difficulties with implementation. Registration errors or problems associated with sensing or bandwidth of the display devices can cause deterioration in the performance of augmented reality systems. Objects in the real and the virtual world have to be aligned correctly in relation to each other otherwise the illusion which is created as a result of the two worlds coming together will not be comfortable for humans. Sensing devices which provide data inputs for the augmented reality computer have to be able to track the field of vision of a user. Sensing errors, mechanical alignments, incorrect viewing parameters such as field of view or tracker –to- eye position or orientation as well as optical distortion in the display system are some of the problems which can affect the performance of an augmented reality system. [6] In this essay, an attempt has been made to discuss various aspects of augmented reality systems, including perspectives associated with their construction, portability, user benefits for those using augmented reality, outdoor use of augmented reality systems and the problem of errors in such systems. It is hoped that the readers will benefit from such a discussion and gain a broader understanding of the current state of augmented reality. The next few sections of this essay discuss the previously mentioned issues. 2. A Description of the Operation of an Augmented Reality System Augmented reality systems usually add to the information that is received from the real world. In augmented reality vision systems, objects from the virtual world generated by a computer can be added to the scene from the real world. However, augmented reality systems can also remove sensations from the real world and as an example; it is possible to remove a table which a human using the augmented reality system may be viewing as a result of the computer painting it over. Although augmented reality usually refers to a computer modifying the real information that is presented to the human vision system, vision is not the only sense which a computer can interfere with. A system in which multi – sensory input is provided may turn on a heat lamp when a user approaches a spot in the virtual environment that is exposed to the sun or the smell of roses may be directed to a user when they approach a rose garden in mixed environment. A computer can also direct that a fan be switched o n in order to provide a flow of air to a user when they approach a spot in the virtual environment that is exposed to wind. [7] Thus, augmented reality involves modification of reality that is presented to a user by a computer which also senses reality and which has created a virtual model based on what has been sensed. Examples of the application of augmented reality include superimposition of internal information over external surfaces or the augmentation of viewed environment with informational labels. [8] When thinking about augmented reality, it is appropriate to think about the physical space, the virtual space and the measured space. The concept of physical and virtual space should be clear, however the concept of measured space refers to the representation of the physical space inside a computer and the manner in which sensory data is used to present the real space to a user. The generation of augmented and virtual spaces require the joining together or overlapping of the space maps associated with the physical and the virtual spaces, with the interaction being managed by computers and people using their sensory capabilities. Virtual space does not necessarily represent the physical space and this space is a model of the physical space that is created the real world and its views as perceived by sensors. Virtual space contains inputs from the physical space which are incorporated into the electronic space by the computer. The generation of augmented reality involves superimposing electronic generated space on to the real objects and space. The virtual space inside a computer must correspond to the real space which has to be mapped into the computer using some sort of a global reference such as a global positioning system. [9] Relative measurements of objects in physical space are transferred to the computer so that the electronic space may be generated, even though it may not be known where the physical space is with regard to the rest of the universe. Thus, a car’s interior dimensions are relative to each other, although it may not be known where the car is in the universe. A close mapping may exist between the physical and the virtual space if the electronic space is required to be a close representation of the physical space. It may be required to represent different parameters from the real space into a virtual space, such as the variation of light intensity or temperature. Some event in the physical space may also be required to generate a trigger and some how alter the electronic virtual space. Different types of sensors may be used to provide information about the physical space such as the dimensions of the physical space being measured by ultrasound or sonar. Sensor inputs are required to be fed into a computer, processed and then used to present a reflection in the electronic space. Triggers from the physical space are also measured by sensors and as an example; the presence of a human object in an area of the physical space may be registered by a capacitive sensor which will transmit this information into the computer. The computer that is being used may then take some action to process this trigger and generate some outputs or variations in the electronic space. Virtual spaces that are generated inside a computer after some processing of the data related to physical spaces are translated and made available in the real world by projection involving video, audio, tactile or hepatic devices or even by using the sense of smell. The virtual world of the computer is projected at some point or location in the physical world which is known as the point of projection. Device such as a screen, a virtual reality goggle or PDA etc are used for such projections. Projections of the virtual world into the real world through devices present users with an illusion of occupying some part of the physical space such as the space behind a projection screen or in front of a holographic plate. In purely virtual reality environments, the sensing of an object from the physical space may have an impact on the projection of the virtual space into the physical space, however, the object that is being sensed will not be a part of the virtual space. In mixed or augmented spaces, however, the sensed object will be incorporated into the virtual space and hence the link between the object that is being sensed and the projection is important. Location in space is usually measured in terms of some sort of coordinates which could be Euclidean coordinates such as polar, spherical or ordinance grid coordinates or alternatively, the location may be relative to a zone with objects being detected as being in a zone such as a room or a part of a room. Information about position may also be relational with some object being detected as being close to another object. The level of accuracy with which measurements in the physical space need to be made and information in the virtual space need to be displayed will vary for different applications, but the requirements related to accuracy will determine the amount of data that needs to be exchanged between the physical and virtual spaces. [10] As an example, head mounted displays and cameras that are mounted on these displays can be used to present computer modified reality to a user. The cameras sense reality and feed a video signal to a computer which is then modified by the computer according to the programmed algorithms for the generation of virtual space involving identification of objects, addition of data to images, image manipulation or object cancellation etc. The desired additions or modifications to the human view of reality are then projected on to the human eye through head mounted monitors or optical diversion and mixing of the real and virtual space. This is illustrated in the diagrams below. The Generation of Visual Augmented Reality using Head Mounted Displays, Cameras and Diverters for the Mixing of Real and Virtual Spaces [11] [12] Managing Multiple Spaces for Generation of Augmented Reality [13] Apart from optical see – through augmented reality displays, it is also possible to generate augmented reality using monitors and video see-through displays. Optical see-through displays in which the real world images are mixed with the virtual reality images that are generated by a computer using optical mixing are different from video see-through displays which project images that are desired to be seen by a human user on to the eye using a video display without any optical mixing from the real world. [14] The next section of this essay takes a look at the user benefits of an augmented reality system. 3. The User Benefits of an Augmented Reality System Augmented reality systems attempt to present a world which consists of a combination of computer generated imagery or other sensory inputs and real world perceptions that are available to a human user. Thus, augmented reality systems can present an enhanced view of the world to the users and the enhancements provide additional information which can be of benefit to the user in a variety of ways. The additional information which can be superimposed on the perception of a user can take a variety of forms depending on the application or the intended use of the augmented reality application. As an example, an augmented reality system may be designed to superimpose a pipeline system which has been designed for an industrial setting, or the system may assist a surgeon by superimposing internal imagery of a patient’s diseased organs on the image of their exterior anatomy, assisting the surgeon to perform surgery. All augmented reality systems assist humans to perform a task by enabli ng the human user to visualise, readily access additional information or to superimpose objects that are not visible on to real views. Augmented reality systems can enhance human understanding of what they are able to perceive and thus humans are assisted in solving problems which may be difficult if they were not provided any additional pieces of information. However, the applications of augmented reality are very broad and this technology has proven to be useful in very many applications. Thus, augmented reality may be used for entertainment or gaming, providing additional input on views of sporting events as well as assisting humans in more serious endeavours such as the battlefield, archaeology or architecture or urban design etc. Objects that are superimposed on real world views using augmented reality may be required to be presented in correct perspective depth as well as being accurately positioned with respect to other real or virtual objects and this can assist in human use rs being able to perform delicate work due to the additional understanding that they are able to acquire as a result of using augmented reality systems. Miniaturisation of computing elements and advances in wireless as well as general technological advances on a broad front have made it possible for augmented reality systems to be miniaturised and to be made wearable or portable, adding to their general usefulness both indoors and outdoors. Thus, augmented reality systems are able to assist, entertain, inform or aid humans by enhancing their perceptions in a wide variety of applications. [15] [16] [17] [18] The next section of this essay presents a brief discussion on portable augmented reality systems in wearable computers. This topic is important because portable and wearable systems have added considerably to the utility of augmented reality systems. 4. Portable Augmented Reality Systems in Wearable Computers Portability is required if an augmented reality user has to change their position in the world. Unlike virtual reality, augmented reality users cannot change their position by changing their location in the computer generated virtual world and have to be able to move around in the physical world. Portability is essential for a wide variety of augmented reality applications because users may be required to roam around in large areas. However, such requirement means that the augmented reality equipment has to be self contained as well as portable and that it should be possible to track users outdoors. The requirements for being able to track a user means that markers or communication relays will have to be utilised, especially if the computational requirements associated with augmented reality cannot be fully catered for on the portable end of the equipment and signals have to be relayed to a desktop, mainframe or networked computer. Because augmented reality does not replace the real world as virtual reality does, therefore, the resolution requirements associated with display devices or sensors are generally less stringent then those for virtual reality applications. Full colour displays are usually required for virtual reality applications, but monochrome displays may be sufficient for augmented reality. Requirements associated with rendering are also not a problem with augmented reality applications because images are only required to add to the real world visual inputs. However, tracking and sensing requirements for augmented reality applications are far more stringent then those for virtual reality applications because objects in the virtual space have to be correctly aligned with objects in the real world. As a result of the requirement to accurately track a user’s head and objects in the surrounding area, a large volume of tracking data is likely to be generated which must be processed in real time. If a portable or wearable computer is incapable of catering to these processing requirements, then reliable communications links must carry the tracking or associated data to remote computers. Thus, trackers or sensors are required to have a longer range, greater input variety and bandwidth as well as better accuracy. [19] [20] [21] Advances in portable augmented reality systems have been made possible as a result of faster, more rugged and better computing devices which consume low power. Wearable computers are required to be comfortable, safe and adjustable for an individual’s personal preferences. The user interfaces for such systems have to be user friendly as well as being capable of operating correctly in a harsh and noisy environment. Gesture and hand motion based interfaces have shown a promise in wearable computing, but such systems are required to be able to interface with a user in a reliable manner. Software for augmented reality substantially increases in complexity with complex applications. Short range wireless communication technologies such as Bluetooth have made it possible for sensors to be readily placed on all parts of the human body. Thus, it may be concluded that portable augmented reality systems are required and useful but their widespread use and acceptance will depend on the rel iability, ruggedness, comfort as well as computing power which can be packed into such systems. However, technological advances on a broad front have resulted in many portable augmented reality systems being developed including systems for military, field archaeology or navigational assistance to name a few. [22] [23] The human factors requirements associated with wearable computing demand that aspects related to safety, ergonomics, anthropometry and ease of use or usability be considered in the design of portable or wearable augmented reality systems. Human users should be able to comprehend information that is being presented to them without any perceptual interference. The design of the wearable system should be based on ergonomic studies with good visually coupled display design. The user interface design should promote reliability in interactions with the system and usability for the human user. The augmented reality system is required to minimise the cognitive load on the human user and not add to the problems. It should be possible to indulge in team collaboration through the wearable augmented reality system. [24] The design of usable interfaces for human interactions is of the greatest importance in the design of wearable augmented reality system design. There are many usability guidelines which should be considered for designs and the quality of the design for usability can also be used to evaluate an augmented reality system. The design of a user interface should take into consideration factors including the level of user experience, support for interactions with other users, number and location of users, ease of calibration, support for body centred interactions, reduced relative latency and provide the user with a tracking system that is accurate to a small faction of a degree in orientation and a few millimetres in position. The overall system processing delay should be minimised and there should be a level of consistency in the visual or sensory cues that are presented to the user. It is desirable that the errors which affect the performance of an augmented reality system and which are discussed in section 6 are minimised. The considerations associated with a good design of a wearable augmented reality system are too numerous to be tabulated here, but a detailed discussion about these issues is presented in Gabbard, Joseph. L. â€Å"Researching Usability Design and Evaluation Guidelines for Augmented Reality (AR) Systems†. Virginia Tech. 2001. [25] In the next section of this essay, a discussion is presented about the outdoor use of an augmented reality system. 5. The Outdoor Use of an Augmented Reality System Augmented reality systems are increasingly being used outdoors. Whereas, it is obvious that the use of augmented reality is gaining greater acceptance because there is a need for applications to assist humans with various activities outdoors, it is also important to understand that the outdoor use of augmented reality presents some technical challenges. Examples of the outdoor use of augmented reality include assistance with rescue operations after a natural disaster such as an earthquake or assisting with military operations in urban environment. [26] [27] Both of the previously mentioned applications require the augmented reality equipment to operate in a harsh environment in which any existing communications infrastructure is likely to have been obliterated or in danger of being destroyed. The military application can assist soldiers in identifying buildings, getting a feel for their interior and to be informed about potential threats such as the location of snipers from battlefie ld intelligence network, with such information being conveniently displayed on their field of vision. However, communication links are important because sophisticated computations on data cannot be performed on portable augmented reality units. Disaster relief can be aided by augmented reality as a result of rescuers being provided with on the spot information about buildings, terrain and the likely results of any planned rescue operations such as the likely impact of clearing a path through rubble. Although the additional information that is presented by outdoor augmented reality systems is likely to be of great benefit, it has to be realised that the generation of such information is only possible after extensive digital surveys of the areas for which augmented reality is being used are available. The likely design of such augmented reality systems will also mean that not all computation or data storage is capable of being performed on the portable units which are provided to the field users. The need to establish and maintain communication links as well as sophisticated computer equipment with which portable units are connected through communication links as well as the requirements to collect detailed data such as airborne laser scanning data under conditions of war or immediately after a natural disaster are some of the difficulties associated with the previously outdoor augmented reality applications. The equipment has to be extremely rugged and reliable as well as capable of being worn over long periods. In the military applications, the communication system which consists of fixed transmitters can be vulnerable and there is a requirement to maintain the large bandwidth communication links between the portable units and a central computer so that the augmented reality system can superimpose virtual scenes on real world information with accuracy as well as reliability. It will be undesirable to have augmented reality system malfunctions in situations whi ch require that users concentrate on other critical tasks rather then fixing any idiosyncrasies associated with the augmented reality equipment. Generally speaking, it is difficult for the existing tracking technology to track a user with sufficient accuracy when a portable augmented reality system is being used outdoors. [28] The next section of the essay presents a discussion about errors in augmented reality systems. 6. The Potential Sources of Error in an Augmented Reality System An important problem which has been observed in augmented reality systems is that objects which have been generated by the computer in a virtual environment for superimposition on to the view of the real environment do not remain correctly aligned or locked onto the real world objects. The degree of misalignment can vary as the user changes their field of view and thus, with a changing field of vision due to a user moving their head, objects that are being viewed through the augmented reality displays can appear to swim as a user moves their head. [29] The errors which create problems in augmented reality systems can be broadly classified as static or dynamic errors. Static errors refer to those sources of error which can cause problems with the presentation of augmented reality when the user and the objects within the environment are completely still. Dynamic errors do not have an impact until there is a movement of the viewpoint or the objects. Some of the sources of error which ca n have an impact on the visual performance of augmented reality systems are as follows: [30] Static Errors Optical distortion caused by distortions or imperfections in the camera and lens systems especially in objects which are away from the optical axis. Errors in the tracking and sensing system which result a distorted output from such systems. Mechanical misalignments that exist within the components of the augmented reality system such as combiners, optics or monitors due to factors such as not having a sufficiently rigid mechanical frame. Incorrect viewing parameters due to a lack of calibration that has an impact on how the reported head or camera locations are translated into viewing matrices which are used by scene generators for drawing images that are required to be superimposed on to the real world inputs. Dynamic Errors End –to- end system delays cause problems because each component in the augmented reality system require some time to produce an output for a given input. Any change in the position or orientation of a user will take some time before being processed and registered on the augmented reality picture that is presented to a viewer. Delays associated with the tracking subsystem, communication links and scene generation as well as time delays that are associated with frame buffers will prevent an immediate updating of the augmented reality picture that is presented to the viewer. A certain level of delay or lag is acceptable and these delays will not cause a serious deterioration in the augmented reality presentation. Typi