Contents lists available at ScienceDirect

Journal of Dentistry

journal homepage: www.elsevier.com/locate/jdent

Short communication

Mapping intraoral photographs on virtual teeth model

Walter Y.H. Lama, Richard T.C. Hsungb, Leo Y.Y. Chengc, Edmond H.N. Powa,⁎

a Prosthodontics, Faculty of Dentistry, The University of Hong Kong, Hong Kong Special Administrative Region, Chinab School of Information Engineering, Guangdong University of Technology, Guangdong, Chinac Dental Technology, Prosthodontics, Faculty of Dentistry, The University of Hong Kong, Hong Kong Special Administrative Region,China

A R T I C L E I N F O

Keywords:3D modelAesthetic dentistryImage-to-geometry registrationOptical scanningPhotographyShade matchingSurface scanningVirtual patient model

A B S T R A C T

Objectives: Tooth shade is crucial to patient satisfaction in aesthetic dentistry. Two-dimensional (2D) clinicalphotographs have been widely used to record patients’ tooth shade for creation of beautiful smile. However, inthe digital workflow, 3D virtual teeth models in open file format contain only mesh information with no colour.This paper describes the mapping of colour information from the intraoral photographs onto the virtual teethmodel.Methods: Intraoral photographs of occlusal and buccal views were taken using digital single lens reflex (SLR)camera with macro lens and ring flash. Photographs were calibrated for its colour and white balance. Virtualmodels were generated by scanning teeth/stone casts using an intraoral/model scanner. Intraoral photographswere mapped onto the virtual model using the image-to-geometry registration method by locating correspondingfeature points in the 2D and 3D images.Results: Virtual teeth models of dentate (with and without crowding) and partially dentate patients weremapped with intraoral photographs. The resultant models are open file format and can be viewed and ma-nipulated in dental or generic CAD/CAM software. Moreover, RGB (Red Green Blue) colour information anderror of registration can be retrieved.Conclusion: Image-to-geometry registration allows mapping of colour information in the 2D intraoral photo-graphs on 3D virtual teeth models. The proposed method is applicable across scanning systems and the colouredmodel can be generated from stone casts and intraoral photographs.Clinical significance: Virtual teeth model with colour information facilitates shade matching and creation ofbeautiful smile.

1. Introduction

Creation of a beautiful smile is generally considered as the ultimategoal of aesthetic dentistry [1]. Tooth shade, among all the factorsgoverning dental aesthetics, plays an important role in determining thefinal aesthetic outcome [2–4]. It has been found to be one of the mostimportant factors in patient satisfaction [4,5]. Many patients are in-terested in having their teeth whiter and expect dental restorationsmatch or blend with their natural teeth. Therefore, treatment planningin the aesthetic zone should include a thorough assessment of patient’stooth shade.

Two-dimensional (2D) intraoral photographs taken by single lensreflex (SLR) camera have been used for shade matching [6,7] and theyhave the advantage of providing colour map as well as surface textureof teeth [8,9]. However, these photographs should be calibrated fortheir colour and white balance for precise shade matching [10].

In computer-aided design and manufacturing (CAD/CAM) dentistry,the dentition should be digital scanned before further processing. Whilemany intraoral scanners generate three-dimensional (3D) virtualmodels in open file format such as STL (STereoLithography), thesemodels contain mesh information only and do not have colour in-formation [11]. On the other hand, intraoral scanners that can capturecolour data export encrypted (closed file format) 3D models that canonly be viewed and manipulated by proprietary CAD software. More-over, tooth colour information cannot be obtained if stone casts aredigitally scanned. This paper describes a workflow to generate a virtualteeth model with colour information in an open file format. Image-to-geometry registration was used and this is an established technique formapping 2D surface texture and colour onto a 3D model [20,21].

https://doi.org/10.1016/j.jdent.2018.09.009Received 7 August 2018; Received in revised form 21 September 2018; Accepted 25 September 2018

⁎ Corresponding author at: Prince Philip Dental Hospital, Room 3B17, 34 Hospital Road, Sai Ying Pun, Hong Kong Special Administrative Region, China.E-mail address: ehnpow@hku.hk (E.H.N. Pow).

Journal of Dentistry 79 (2018) 107–110

0300-5712/ © 2018 Elsevier Ltd. All rights reserved.

T

2. Materials and methods

Digital Single Lens Reflex (SLR) camera (EOS 500D, Canon) wasequipped with a macro lens (EF 100mm f/2.8, Canon) and a ring flash(Macro Ring Lite MR-14EX, Canon) for intraoral photography.Maxillary occlusal, mandibular occlusal, right and left buccal viewswith teeth apart were taken for mapping the maxillary and mandibularvirtual models (Fig. 1a). Occlusal views may be taken using occlusalmirror and the mirror images were flipped using Paint (Windows 10,Microsoft). If only the anterior teeth are interested, one frontal viewwith teeth apart may be adequate (Fig. 1b). The colour and whitebalance of photographs can be calibrated, in an open source photo-graphy editing software (GNU Image Manipulation Program; v2.8; TheGIMP Team, www.gimp.org), with reference to a commercial colourchecker (ColorChecker Passport, X-Rite) or grey card (R-27, Kodak)[5,22].

Dentitions were scanned using an intraoral scanner (True definitionscanner; 3M ESPE) and 3D models with open file format such as STL(STereoLithography) were generated which contained only mesh in-formation. Alternatively, stone casts could be scanned using a struc-tured light 3D scanner (DAVID SLS-3, Hewlett-Packard). However,tooth colour information cannot be obtained if stone casts are digitallyscanned. The virtual teeth model may be “water-leak” at its base andcould be closed with a flat plane model (video 1) in an open sourcemesh processing software (MeshLab, v1.3.3; Visual Computing Lab ofthe ISTI-CNR, www.meshlab.net) to make it “water-tight” (Fig. 2 andvideo 2).

During mapping the 2D intraoral photographs onto the 3D virtualteeth model, the maxillary virtual model was firstly moved and re-scaled to match one photographic view [12]. At least four corre-sponding feature points were selected on both the photograph andvirtual model (Fig. 3 and video 3) [13,14]. These feature points usuallylocated at the cusp tip, incisal angle and/or gingival margin wherethere is an abrupt change in contour. Image-to-geometry registrationwas then performed in that view angle. Error of registration would beshown by the mesh processing software (Fig. 4) and alternative/more(up to 7–8) feature points could be selected to improve the accuracy ofregistration. Registration was repeated for all view angles and for themandibular model. As a result, the intraoral photographs were pro-jected on the virtual models and the file would be saved as OBJ (Objectfile), PLY (Polygon File Format) or WRL (Virtual Reality ModellingLanguage) format which had colour texture in addition to the meshinformation (video 4).

3. Results

Three sets of virtual teeth models including dentate (Fig. 5a and b),dentate with crowding (Fig. 5c) and partial dentate with tetracyclinestaining at mandibular teeth (Fig. 5d) were mapped with intraoral

Fig. 1. a) Occlusal and buccal intraoral photographs for mapping the maxillaryand mandibular virtual models. These views should be in-focus with soft tissuesadequately retracted. Buccal views should cover from the last tooth of one sideto across the dental midline to facilitate image merging of two buccal views.The maxillary and mandibular arches should be kept apart so the incisal/cusptips can be captured and used as feature points in the mapping. b) Frontal in-traoral photograph with teeth apart for mapping the anterior teeth.

Fig. 2. Prepare “watertight” virtual model. A. Maxillary virtual model with topsurface missing (dark) and is “water-leak”; B. Align an upper plane model (red)of matched vertex density (e.g. 1 vertex per 0.1 mm) to the top surface ofmaxillary virtual model; C. Trim the plane model (red) at the boundary ofmaxillary virtual model; D. “Watertight” virtual model. (For interpretation ofthe references to colour in this figure legend, the reader is referred to the webversion of this article).

Fig. 3. Matching virtual teeth models to different view angles of intraoralphotographs. In both images, corresponding feature points (encircled in redcolour for better visualization) were located on cusp tip, incisal angle and/orgingival margin. (For interpretation of the references to colour in this figurelegend, the reader is referred to the web version of this article).

W.Y.H. Lam et al. Journal of Dentistry 79 (2018) 107–110

108

photographs. Colour information in RGB (Red Green Blue) could beretrieved from these mapped models by the mesh processing software(Fig. 6).

4. Discussion

Virtual teeth model with colour information facilitates aestheticplanning and treatment execution. Digital shade matching is shown tobe more accurate and useful than the conventional chairside methodwhen digital workflow is used in practice [8]. Using the proposedtechnique, clinicians can retrieve the RGB colour information from thevirtual teeth models and can be converted to CIELAB colour for clinicaluse [15,16]. Additional information such as the surface texture andcolour distribution are also provided by these models. Since dentalaesthetics should be in harmony with patient’s facial appearances [17],virtual teeth models could be combined with 3D facial photographs togenerate virtual patient models (VPM) [18–20] for 3D digital smiledesign [21]. These VPMs may also be integrated to virtual articulatorfor creation of smile design that is in harmony with the occlusion[22–25] (Fig. 7). Many 3D faces are now with colour information andtherefore virtual teeth models with colour information are preferred.

The proposed technique is applicable to all 3D scanning systemsthat create open source virtual models [26] and to patients withcrowding and/or missing teeth. 3D models may be obtained by scan-ning of teeth or stone casts. The resultant virtual models have open fileformat and can be viewed/manipulated in dental or generic CAD/CAM

software. This is particular advantageous for clinicians and researcherswho request additional CAD and/or 3D processing/measuring functionsthat are not provided by dental CAD/CAM software. Extra clinicalprocedures are not required since intraoral photography and teethscanning/impressions are part of patients’ dental record and dentistsare competent to these procedures. Open source software are availablefor the mapping procedure, however training is needed for their uses.The colour calibration and the mapping procedures takes around 10and 15min respectively. However these procedures can be performedby non-clinical supporting staffs such as dental technicians and do notincrease the chairside time. The coloured models are at most as goodquality as the intraoral photographs and every efforts should be madeto ensure high quality photographs were taken (Fig. 1). Lastly, it shouldbe cautious in using this technique for edentulous patients or patientswith extensive loss of dentition where insufficient feature points can befound for the mapping.

5. Conclusions

A technique for mapping intraoral photographs on virtual teethmodel is presented. Image-to-geometry registration is an establishedtechnique for mapping 2D surface texture and colour onto a 3D model.This method is applicable across scanning systems and the resultantteeth models are open file format and can be viewed and manipulatedin dental or generic CAD/CAM software.

Fig. 4. Error of image-to-geometric registration shown (indicated by orange arrow for better visualization) in the mesh processing software.

Fig. 5. Virtual teeth models mapped with intraoral photographs. A. Dentate model mapped with occlusal and buccal views. B. Same dentate model mapped withfrontal view only. C. Dentate with crowding. D. Partial dentate with tetracycline staining at mandibular teeth.

W.Y.H. Lam et al. Journal of Dentistry 79 (2018) 107–110

109

Funding sources

This research did not receive any specific grant from fundingagencies in the public, commercial, or not-for-profit sectors.

Appendix A. Supplementary data

Supplementary material related to this article can be found, in theonline version, at doi:https://doi.org/10.1016/j.jdent.2018.09.009.

References

[1] A. Joiner, Tooth colour: a review of the literature, J. Dent. 32 (2004) 3–12.[2] A. Della Bona, A.A. Barrett, V. Rosa, C. Pinzetta, Visual and instrumental agreement

in dental shade selection: three distinct observer populations and shade matchingprotocols, Dent. Mater. 25 (2) (2009) 276–281.

[3] J. Jørnung, Ø. Fardal, Perceptions of patients’ smiles: a comparison of patients’ anddentists’ opinions, J. Am. Dent. Assoc. 138 (12) (2007) 1544–1553.

[4] G.R. Samorodnitzky-Naveh, S.B. Geiger, L. Levin, Patients’ satisfaction with dentalesthetics, J. Am. Dent. Assoc. 138 (6) (2007) 805–808.

[5] M.M. Tin-Oo, N. Saddki, N. Hassan, Factors influencing patient satisfaction withdental appearance and treatments they desire to improve aesthetics, BMC OralHealth 11 (1) (2011) 6.

[6] A.G. Wee, D.T. Lindsey, S. Kuo, W.M. Johnston, Color accuracy of commercial

digital cameras for use in dentistry, Dent. Mater. 22 (6) (2006) 553–559.[7] W. Tam, H. Lee, Dental shade matching using a digital camera, J. Dent. 40 (2012)

e3–e10.[8] F. Jarad, M. Russell, B. Moss, The use of digital imaging for colour matching and

communication in restorative dentistry, Br. Dent. J. 199 (1) (2005) 43.[9] S.J. Chu, R.D. Trushkowsky, R.D. Paravina, Dental color matching instruments and

systems, Rev. Clin. Res. Aspects J. Dent. 38 (2010) e2–e16.[10] J. Fondriest, Shade matching in restorative dentistry: the science and strategies, Int.

J. Periodontics Restorative Dent. 23 (5) (2003) 467–480.[11] S. Ting‐shu, S. Jian, Intraoral digital impression technique: a review, J.

Prosthodont. 24 (4) (2015) 313–321.[12] M. Corsini, M. Dellepiane, F. Ponchio, R. Scopigno, Image‐to‐geometry registration:

a mutual information method exploiting illumination‐related geometric properties,computer graphics forum, Wiley Online Lib. (2009) 1755–1764.

[13] M. Sottile, M. Dellepiane, P. Cignoni, R. Scopigno, Mutual correspondences: anhybrid method for image-to-geometry registration, Eurographics Italian ChapterConference, (2010), pp. 81–88.

[14] R.T.C. Hsung, W.Y.H. Lam, E.H.N. Pow, Image to Geometry Registration for VirtualDental Models, Digital Signal Processing, Shanghai, China.

[15] C. Connolly, T. Fleiss, A study of efficiency and accuracy in the transformation fromRGB to CIELAB color space, IEEE Trans. Image Process. 6 (7) (1997) 1046–1048.

[16] S. Westland, C. Ripamonti, V. Cheung, Computational Colour Science UsingMATLAB, John Wiley & Sons, 2012.

[17] M.R. Mack, Perspective of facial esthetics in dental treatment planning, J. Prosthet.Dent. 75 (2) (1996) 169–176.

[18] W.Y. Lam, R.T. Hsung, W.W. Choi, H.W. Luk, E.H. Pow, A 2-part facebow for CAD-CAM dentistry, J. Prosthet. Dent. 116 (6) (2016) 843–847.

[19] E. Solaberrieta, A. Garmendia, R. Minguez, A. Brizuela, G. Pradies, Virtual facebowtechnique, J. Prosthet. Dent. 114 (6) (2015) 751–755.

[20] T. Joda, G.O. Gallucci, The virtual patient in dental medicine, Clin. Oral ImplantsRes. 26 (6) (2015) 725–726.

[21] M.B. Ackerman, J.L. Ackerman, Smile analysis and design in the digital era, J. Clin.Orthod. 36 (4) (2002) 221–236.

[22] M. Zimmermann, A. Mehl, Virtual smile design systems: a current review, Int. J.Comput. Dent. 18 (4) (2015) 303–317.

[23] W.-S. Lin, B.T. Harris, K. Phasuk, D.R. Llop, D. Morton, Integrating a facial scan,virtual smile design, and 3D virtual patient for treatment with CAD-CAM ceramicveneers: a clinical report, J. Prosthet. Dent. 119 (Feb. (2)) (2018) 200–205, https://doi.org/10.1016/j.prosdent.2017.03.007 Epub 2017 Jun 13.

[24] B. Hassan, B.G. Gonzalez, A. Tahmaseb, M. Greven, D. Wismeijer, A digital ap-proach integrating facial scanning in a CAD-CAM workflow for complete-mouthimplant-supported rehabilitation of patients with edentulism: a pilot clinical study,J. Prosthet. Dent. 117 (4) (2017) 486–492.

[25] W.Y. Lam, R.T. Hsung, W.W. Choi, H.W. Luk, L.Y. Cheng, E.H. Pow, A clinicaltechnique for virtual articulator mounting with natural head position by using ca-librated stereophotogrammetry, J. Prosthet. Dent. 119 (June (6)) (2018) 902–908,https://doi.org/10.1016/j.prosdent.2017.07.026 Epub 2017 Sep 29.

[26] A. Ireland, C. McNamara, M. Clover, K. House, N. Wenger, M. Barbour,K. Alemzadeh, L. Zhang, J. Sandy, 3D surface imaging in dentistry–what we arelooking at, Br. Dent. J. 205 (7) (2008) 387.

Fig. 6. RGB colour information of a facet of virtual model (magnified for better visualization) shown in the mesh processing software.

Fig. 7. Virtual patient model (VPM) shown in a dental CAD/CAM softwareExocad (Exocad GmbH). This VPM composed of 3D face, virtual teeth modeland virtual articulator. Virtual teeth models with colour information are pre-ferred when the 3D face has colour.

W.Y.H. Lam et al. Journal of Dentistry 79 (2018) 107–110

110