ABSTRACT
Augmentation reality technology offers virtual information in addition to that of the real environment and thus opens new possibilities in various fields. The medical applications of augmentation reality are generally concentrated on surgery types, including neurosurgery, laparoscopic surgery and plastic surgery. Augmentation reality technology is also widely used in medical education and training. In dentistry, oral and maxillofacial surgery is the primary area of use, where dental implant placement and orthognathic surgery are the most frequent applications. Recent technological advancements are enabling new applications of restorative dentistry, orthodontics and endodontics. This review briefly summarizes the history, definitions, features, and components of augmented reality technology and discusses its applications and future perspectives in dentistry.
KEYWORDS: Augmented reality; virtual reality; image-guided surgery
1. Introduction
Augmented reality (AR) is a type of technology in which an environment is enhanced through the process of superimposing computer-generated virtual content over real structure, enhancing the sensory perception of reality. The term ‘augmented reality’ is generally believed to have been coined by Caudell and Mizell, who worked for Boeing in the 1990s. However, the origin of AR dates back to the time of World War II. At this time, the de Havilland Mosquito night fighter, a British aircraft, was equipped with an airborne radar navigation system. The first system that was recognized as AR was developed by Sutherland in 1968 using headmounted display (HMD). Since its inception, AR has attracted much interest in diverse fields. In contrast, virtual reality (VR) can be described as the combination of multiple technologies allowing users to interact with virtual entities in real time. The two main features of VR are immersion and interaction. Immersion refers to the sense of presence in a virtual environment, while interaction refers to the user’s power of modification. In a sense, AR can be considered as a subcategory of VR.
However, in contrast to a VR system, which provides only a computer-generated environment that elicits a user’s sense of being present, an AR system mixes real and virtual elements to augment sensory perception in the real environment. Currently, an AR platform is defined as a system that has several of the following features: it combines virtual and real objects in a single real environment, it runs interactively in real time and it registers virtual and real objects reciprocally. To generate a successful AR system, the following essential functionalities should be operated and closely integrated with one another: real and virtual data sources, tracking, registration techniques, visualization processing, display types, perception locations and feedback mechanisms.
To date, AR has been applied in industry, sightseeing, marketing, entertainment, maintenance, architecture, education and medicine is not an exception. The Star Trek series of the 1960s forecast the future application of AR in the field of medicine. Surgeons are typically always looking for better technologies to improve patient outcomes and were early adopters of AR. For instance, neurosurgery, urology, spine surgery, laparoscopic surgery in many fields, endoscopic surgery, hepatobiliary and pancreatic surgery and cancer surgery have all tested the feasibility of AR-aided surgeries. In addition to surgical fields, AR has been applied to the treatment of psychologic disorders, cognitive impairment and motor rehabilitation. Furthermore, AR is widely used in medical education including but not limited to anatomy and surgery, which is sometimes performed in the form of telementoring . Dentistry shares various features of other medical subspecialties but also has unique characteristics. It is another field in which AR technology can be successfully utilized to enhance diagnosis, treatment and education outcomes. The aim of this article was to review the contemporary literature regarding AR in dentistry and to discuss future trends in AR technology.
Applications in oral and maxillofacial surgery
The oral and maxillofacial regions are heavily populated with many surgically and anatomically important structures, including large nerves and vessels. The intricate anatomy of the craniofacial structures requires sophisticated presurgical planning with high precision. AR technology is well suited for this purpose and also fits the currently favored minimally invasive philosophy of maxillofacial surgery. Based on AR principles, where the real operative site of a patient is provided with graphic information that was extracted and modified from a data source, AR navigation systems have been effectively employed.
Previously acquired diagnostic images such as radiographs, computed tomography scans, magnetic resonance imaging scans and angiography can also be good sources of additionally integrated AR information. In contrast to conventional image-guided surgery, where a surgeon must look away from the operative field to view the informative data, AR guidance systems provide real-time intraoperative information with real surgical fields. It is ideal to offer three-dimensional presentations on the patient’s body rather than a separate screen because perception of the real body is more intuitive and avoids confusion. Since the display type dominates the perception location, head-mounted displays or microscopic eyepieces are typically recommended, mimicking Google Glass. Generally, computer-generated additional information is overlaid onto a real surgical field and thus resides between the patient’s body and the surgeon’s eye.
Dental implant placement
AR technology has been shown to substantially improve a wide range of procedures. A dental implant positioning system with a graphically superimposed suggested position on the patient was introduced as early as 1995. Tele planning and surgical navigation of implant placement can also be achieved using AR systems. An AR toolkit using HMD and marker tracking was described by Kato and Billinghurst. The AR surgical navigation systems for implant placement were introduced using retinal imaging display rather than watching the monitor because looking away from small oral surgical site during the operation is quite dangerous. Sometimes, useless information hampers surgeons’ decisions. During implant placement, AR systems can act as automatic information filters that selectively display only the most relevant information to surgeons, thereby helping them concentrate most fully on the implant placement. Lin et al. evaluated the accuracy of AR-based dental implant placement and compared it with that of planning and placing the site in vitro. They found that integrating the surgical template with the AR system significantly reduced placement deviation. This result implies that the accuracy of AR systems still needs to be improved. Interestingly, the AR system used in dental implantology is considered to be cost-effective due to its ability to reduce time and additional costs.
Orthognathic surgery
Orthognathic surgery is one of the most widely used fields of AR applications among the other dental fields. In 1997, Wagner et al. reported augmentation in facial skeleton osteotomy via partial visual immersion using a head-mounted display (HMD). The technology enriched visual access to the invisible anatomy offered continuous observation and was very helpful to doctor’s fluent surgery. A feasibility study in 2006 reported the successful utilization of an augmented reality tool, the ‘X-scope,’ in five patients for translocation of the maxilla after Le Fort I osteotomy. Maxilla positioning was performed within 1mm of the originally planned location with the simultaneous use of intra-operative splints.
In another feasibility study of mandibular angle oblique split osteotomy, occlusal splints were evaluated as registration markers of AR tools in the ‘AR Toolkit.’ The results from 15 patients demonstrated that occlusal splints can serve as powerful and precise markers for AR systems. In order to overlay the 3D information with high accuracy, meticulous registration techniques must be incorporated. Registration is primarily based on face surface matching, superficial fiducial markers and skull implanted or tooth-fixed fiducials to which the occlusal splint belongs.
For maxillofacial surgery, Fushima and Kobayashi suggested a mixed reality-based system using a dental cast model and a 3D maxillofacial mesh model. Manual refinements are frequently needed. Recently, vision-based markerless registration was introduced in a pilot study, in which teeth themselves are used for registration instead of references or fiducial markers. For real-time presentation, tracking must be performed in exact fashion and the supporting visualization processing should be fast enough to avoid any lags or delays. Suenaga et al. reported high accuracy of AR in a pilot study. Visualization processing can be performed in several different ways, such as texture maps, surface mesh, wireframes and transparencies. This processing is important for surgeons to obtain accurate 3D perceptions. Badiali et al. reported that waferless AR-assisted maxillary repositioning surgery demonstrated only small errors that were within acceptable limits. The integration accuracy of computer-generated imaging was also shown to be more reliable than that of real-life vision alone.
Other surgical applications
AR systems have also been introduced to cranio-maxillofacial reconstructive surgery and cleft lip repair surgery has been simulated with AR technology. In addition, studies have been conducted for maxillofacial complex fracture reduction. In another work, AR was utilized for precise positioning of an intraoral distraction osteogenesis device in patients with hemifacial macrosomia. For training purposes, immersive workbench systems were developed and reported for Le Fort I osteotomies. The systems provided sawing, drilling and plate fixation functions with haptic feedback. Teleconsultation and telecommunication have also been reported in cranio-maxillofacial surgery using AR technology, resulting in improved surgery quality.
Other clinical applications in dentistry
Aside from the surgical applications of AR, few studies have investigated the clinical applications of AR in dentistry.
However, interest in this field has grown in parallel with related technological advances. In orthodontics, guided bracket placement was performed using a light-weight AR system; this system was shown to be practical and feasible. One interesting endodontic application of AR technology is automated real-time detection of root canal orifices. The authors used k-nearest algorithms to restrict segmentation and Euclidean distance-based segmentation was applied to determine the orifice locations. For restorative dentistry, AR is still utilized for educational purposes only, which will be discussed in the next section.
Dental education
Along with VR, AR is of great interest to medical education fields, including dentistry (Figure 1). By blending digital elements with a real learning environment, AR provides new educational opportunities. In medical fields, laparoscopic surgery, neurosurgical procedures and echocardiography are the three area to which AR is most frequently applied. Anatomy is of interest to both medicine and dentistry. Traditional anatomy teaching usually entails the use of an atlas and a cadaver, which requires great time and labour. The values of AR are highlighted when few cadavers are available. Moreover, AR has a special advantage when teaching live anatomy because it is possible to visualize internal structures of the user’s own body using augmented mirrored images. Furthermore, AR can be used in real time by incorporating digital ultrasound. Several VR systems have been developed for dental education such as the DenSim simulator (Image Navigation Ltd., New York, NY) and the Virtual Reality Dental Training System (Novint Technologies, Inc. and Harvard School of Dental Medicine, Albuquerque, NM). Since the tooth structures are composed of materials of different hardness including enamel, dentin and pulp, multi-layered virtual tooth models were developed for more reality. Augmented video and audio rendering has been used to help dental students learn how to prepare abutment for all-ceramic restorations through the ‘VirDenT’ project. In Brazil, students prepared for gold only design by creating learning objects using AR and testing them.
The utility of traditional teaching versus AR for Class I and II cavity preparation has been compared and no significant difference was found between the two regarding the level of knowledge. However, students trained by AR showed better depth results, improved extent of Class I cavities and better divergence of the proximal walls of Class II cavities.
Discussion
The recent interest in AR can be said to occur after the release of Google Glass. Open-source development platforms have led to enthusiastic inventor participation. VR and AR are comprehensive technologies that are made up of many components: computation machines, operating software, display devices and tracking sensors. Currently, VR still lacks true realism, although immersion is a basic feature of both VR and AR. Unlike VR, AR can be characterized by incorporating additional information into physical reality. Thus, AR is of greater benefit in real operations compared with VR, since AR enables ‘seeing through’ reality. Thus, it is essential to embed relevant technological and procedural knowledge for operators to access in a way that does not damage their sense of reality. Rapid advances in computational paradigms have enabled AR to be used in clinical situations, as evidenced by processor miniaturization.
While surgeons usually confront superfluous information in the operating room, hidden information that cannot be seen by the naked eye is often of great importance. Therefore, surgeons are familiar with using many technical aids. Often, surgeons must translate, interpret and combine information gathered from the real surgical target on the patient’s body. Such data include surgical plans, CT results, MRI results and anatomical knowledge. AR will potentially enable surgeons to concentrate on the surgery at hand, resulting in better outcomes. Due to this potential advantage, AR is believed to have a wide range of applications in many surgical areas, even in dentistry.
As related technologies evolve, the areas of AR application are expanding. Potential users are becoming more interested in the systems are becoming able to meet expectations. The ‘head-up see-through’ display, HMD, is a relevant example of a convenient user-friendly interface for an AR system to go over a hurdle of inconvenience. More elaborate tracking and registration techniques have also made the systems more precise and accurate. These developments have led to systems that are appropriate for clinical situation not just for educational purposes, highlighting the importance of accuracy for better outcomes and patient safety. Applications in restorative dentistry require more accurate registration, where gaps or errors on the micrometer scale could be harmful. In addition, fast computing devices and supporting software are essential. Since surgeons move freely with their display devices, the overlaid images should be able to coincide three-dimensionally without any lag. Moreover, orientation, scale and depth perception should be properly maintained. This is particularly relevant during dental procedures such as implant installation, because patient can move their heads to a certain extent and sometimes abruptly because they are just under local anesthesia in usual situations (Figure 2). Furthermore, crowding of the operational view must be avoided and can be achieved through visualization processing techniques, such as transparency or color coding.
Regarding smile width, 58% of the sample showed dental expansion (CI: 58.0 ± 13.77%). Excessive expansion (>5 mm) was observed in 8% of cases (CI: 8.0 ± 7.84%). See Figure 2.
The advantages of AR systems are clear. Preoperative planning can provide realistic outcome predictions. Intraoperative navigation provides surgeons with better chances of achieving good results and for mitigating potential risks. As educational tools, AR simulators can offer enhanced opportunities to students and medical residents.
However, a number of uncertainties limit the extent of AR use. Most of these uncertainties are anticipated to be solved by technological progress. Legal issues regarding confidential data management are also expected to be solved. Another issue is cost/benefit evaluation from the perspectives of patients and doctors since AR systems are still expensive for daily practice in dental clinics. Supplementation with other technology seems to enhance the function of current AR systems. Photon emission tomography, near infra-red spectroscopy and the use of dyes, such as indocyanine green have been used with AR systems to identify sentinel nodes and tissue vascularity. Haptic force feedback and robotics are also promising avenues to combine with AR technology.
Conclusions
In this review, the history, features, main components and applications of AR were briefly stated and compared with those of VR. Contemporary literature regarding the use of AR in dentistry has mainly focused on the applications of AR to cranio-maxillofacial surgery, such as surgical preplanning and navigation systems. AR technology has also expanded to other areas of dentistry, such as restorative dentistry, orthodontics and endodontics; this expansion has been expedited by technological advances. Simulators for education or training are well-suited to AR technology, especially in anatomy, surgery and restorative dentistry. Although the feasibility of AR application in dentistry has been confirmed, it has not been established whether AR is superior to conventional methods. With other state-of-the-art technologies, such as haptics and robotics, AR applications in dentistry are expected to become more prevalent in the near future.
Source
https://pubmed.ncbi.nlm.nih.gov/29465283/
References
[1] Tang JB, Amadio PC, Boyer MI, et al. Current practice of primary flexor tendon repair: a global view. Hand Clin. 2013;29:179–189.
[2] Hugues O, Fuchs P, Nannipieri O. New augmented reality taxonomy: technologies and features of augmented environment. Berlin: Springer; 2011.
[3] Carmigniani J, Furht B. Augmented reality: an overview. In: Furht B, editor. Handbook of augmented reality. New York (NY): Springer New York; 2011. p. 3–46.
[4] Vaughan-Nichols SJ. Augmented reality: no longer a novelty? Computer. 2009;42:19–22.
[5] Feiner SK. Augmented reality: a new way of seeing. Sci Am. 2002;286:48–55.
[6] McCloy R, Stone R. Science, medicine, and the future. Virtual reality in surgery. BMJ. 2001;323:912–915.
[7] Ryan ML. Immersion vs. Interactivity: virtual reality and literary theory. Baltimore (MD): Project MUSE; 1999.
[8] Albuha Al-Mussawi RM, Farid F. Computer-based technologies in dentistry: types and applications. J Dent (Tehran). 2016;13: 215–222.
[9] Riva G, Mantovani F, Gaggioli A. Presence and rehabilitation: toward second-generation virtual reality applications in neuropsychology. J Neuroeng Rehabil. 2004;1:9.
[10] Zinser MJ, Mischkowski RA, Dreiseidler T, et al. Computer-assisted orthognathic surgery: waferless maxillary positioning, versatility, and accuracy of an image-guided visualisation display. Br J Oral Maxillofac Surg. 2013;51:827–833.
[11] Zhou F, Duh HBL, Billinghurst M. Trends in augmented reality tracking, interaction and display: a review of ten years of ISMAR. Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. Washington (DC): IEEE Computer Society; 2008. p. 193–202.
[12] Azuma R, Baillot Y, Behringer R, et al. Recent advances in augmented reality. IEEE Comput Grap Appl. 2001;21:34–47.
[13] Meola A, Cutolo F, Carbone M, et al. Augmented reality in neurosurgery: a systematic review. Neurosurg Rev. 2016;40:537–548.
[14] Reidsma D, Katayose H, Nijholt A. Advances in computer entertainment. Proceedings of the10th International Conference, ACE 2013 Bekel, 2013 Nov 12–15; The Netherlands. Berlin: Springer; 2013.
[15] Chicchi Giglioli IA, Pallavicini F, Pedroli E, et al. Augmented reality: a brand new challenge for the assessment and treatment of psychological disorders. Comput Math Methods Med. 2015;2015: 862942.
[16] Berryman DR. Augmented reality: a review. Med Ref Serv Q. 2012;31:212–218.
[17] Alberti O, Dorward NL, Kitchen ND, et al. Neuronavigation-impact on operating time. Stereotact Funct Neurosurg. 1997;68:44–48.
[18] Cabrilo I, Bijlenga P, Schaller K. Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir. 2014;156:1769–1774.
[19] Cabrilo I, Bijlenga P, Schaller K. Augmented reality in the surgery of cerebral aneurysms: a technical report. Neurosurgery. 2014; 10(2):252–260.
[20] Cabrilo I, Sarrafzadeh A, Bijlenga P, et al. Augmented realityassisted skull base surgery. Neurochirurgie. 2014;60:304–306.