Decoding crystallography from high-resolution electron ... · was applied for each class and...

SC I ENCE ADVANCES | R E S EARCH ART I C L E

1Idaho National Laboratory, Nuclear Science and Technology Division, Idaho Falls, ID,USA. 2Department of Electrical and Computer Engineering, Scientific Computing Imag-ing Institute, University of Utah, Salt Lake City, UT, USA. 3Oak RidgeNational Laboratory,Center for Nanophase Materials Science, Oak Ridge, TN, USA.*Corresponding author. Email: [email protected]

Aguiar et al., Sci. Adv. 2019;5 : eaaw1949 30 October 2019

Copyright © 2019

The Authors, some

rights reserved;

exclusive licensee

American Association

for the Advancement

of Science. No claim to

originalU.S. Government

Works. Distributed

under a Creative

Commons Attribution

NonCommercial

License 4.0 (CC BY-NC).

Decoding crystallography from high-resolution electronimaging and diffraction datasets with deep learningJ. A. Aguiar1*, M. L. Gong1,2, R. R. Unocic3, T. Tasdizen2, B. D. Miller1

While machine learning has been making enormous strides in many technical areas, it is still massively underused intransmission electron microscopy. To address this, a convolutional neural network model was developed for reliableclassification of crystal structures from small numbers of electron images and diffraction patterns with no preferredorientation. Diffraction data containing 571,340 individual crystals divided among seven families, 32 genera, and230 space groups were used to train the network. Despite the highly imbalanced dataset, the network narrowsdown the space groups to the top two with over 70% confidence in the worst case and up to 95% in the commoncases. As examples, we benchmarked against alloys to two-dimensional materials to cross-validate our deep-learning model against high-resolution transmission electron images and diffraction patterns. We present thisresult both as a research tool and deep-learning application for diffraction analysis.

D
on M
arch 14, 2020http://advances.sciencem

ag.org/ow

nloaded from

INTRODUCTIONAugmented analysis—algorithm-powered, human-guided dataexploration—creates a powerful symbiosis between researchers andtheir analytical tools. The breadth of data collected simultaneouslyin the latest generation of scanning transmission electron microscopes(STEMs) presents opportunities for notable advancements in micros-copy, multimodal data analytics, and materials research (1). Inside aSTEM, materials are imaged for their structure, chemistry, and re-sponse to environmental stimuli. Arguably, some of the most usefulinformation gained from this materials characterization approach isthe technology to identify materials and their specific atomic arrange-ments as structure based, with atomic-scale imagery derived fromhigh-resolution imaging or diffraction-based techniques. With the ad-vent of advanced methods of ptychography, precession-based electrondiffraction, compressive imaging, and multidimensional imaging—including four-dimensional microscopy—researchers have developedan urgency to identify crystallographic structures and deviations effi-ciently in more massive datasets (2–6).

The breadth of simultaneously collected data in the latest generationof STEM also presents these same opportunities and more, includingaccelerated multidimensional data collection and real-time processing.With the native resolution of the microscopes, which varies with eachmicroscope from approximately 0.50 to over 2 Å (limited primarily byspherical aberration), these tools are naturally high-informationmachines. As experimental techniques, as well asmicroscopy hardware,evolve—including the recent additions of pixelated detectors—researchers need to address the proliferation of data (7–9). Recentadvancements in deep learning have made it possible to analyzehuman-intractable datasets and perform complex imaging tasks,including segmentation and classification (10–13). However, deeplearning and augmented analysis have not yet disrupted the micro-analysis and materials community as they have in the computer visioncommunity, despite open source implementations of machine learningalgorithms in packages such as TensorFlow and Caffe (14, 15).

In practical terms, learning aboutmaterial structure is not limited byresolution, but by the ability to process information in the context of

atomic arrangements regardless of resolution or technique. In the past,researchers have addressed the identification of structure from x-rayand neutron diffraction data using curve-fit models, Rietveld analysis,peak matching, exhaustive search, and first-hand material knowledge(16–19). Past studies from electron microscopy further reveal the com-plications of identifying phases and structures. Off-axis and obscure axisorientations complicate diffraction data complexity, potentially causingresearchers to overlook identifying diffracting peaks. Texturing issuesare addressed, in part, by precession electron diffraction, electronptychography, and multidimensional STEM imaging. Now, though,processing the abundance of information generated by these techniqueshas further limited our ability to use all forms of information (20, 21). Inthe past, George and Wang (22) reported a hybrid system to identifydiffraction patterns, but this system was not extended to include allmaterial structures and patterns. The opportunity lies in limiting thenumber of patterns and extending predictive models to all materialand crystallographic types. If we were to use modeling and predictivecapabilities to determine pending structure without a priori– or abinitio–based knowledge, we could significantly advance experimentalresearch, especially for those materials composed of multiple orienta-tions and phases. Examples include engineered composites and mul-tilayer systems. Without a predictive approach for translating rawstructural data from either high-resolution atomic-scale images or dif-fraction profiles, the ability to identify unknown data is restricted tocomputationally limited curve-fitting models and expertise.

Advances in computer vision and artificial intelligence present theopportunity for automation and augmentation of crystallographic andhigh-resolution imaging data (23). Deep-learningmodels have been ap-plied to many classification, segmentation, and compression challengesin the computer vision community (24–26). A limited number ofmachine or deep-learning models have reportedly proposed and dem-onstrated subimage sampling in image segmentation and inpainting(27, 28). Current event tracking and augmentation of microscope feeds,however, are limited primarily by postanalysis implementations, withthe main application being in situ microscopy. The barrier to theindependent crystallographic structural determination is translatingthe body of materials knowledge from current databases into acces-sible data workflows not limited to high-resolution electron micro-scopes and diffractometers (29).

In this report, we present an approach to expanding the use of deeplearning for crystal structure determination based on diffraction or

1 of 9

http://advances.sciencemag.org/


atomic-resolution imaging without a priori knowledge or ab initio–based modeling. Various machine learning models, such as NaïveBayes, decision forest, and support vector machines, were tested beforeconcluding that convolutional neural networks (CNNs) produce themodel with the highest accuracy. To determine crystallography fromthese data types, we have trained a CNN to perform diffraction-basedclassification without the use of any stored metadata. The CNN modelhas been trained on a dataset consisting of diffracted peak positionssimulated from over 538,000 materials with representatives from eachspace group. In summary, we assess the growing potential for crystal-lographic structure predictions using deep learning for high-throughputexperiments, augmenting our ability to readily identify materials andtheir atomic structures from as few as four Bragg peaks.

Dow

nlo

RESULTSDeep-learning model for evaluatingcrystallographic informationWe validated the neural network architecture and workflow based onhigh-resolution STEM imaging and electron diffraction from crystalline


strontium titanate (SrTiO3 or STO) islands on a face-centered cubicstructured magnesium oxide (MgO) substrate. Figure 1A is an atomicmass contrast STEM image of the overall sample, with the crystallineportion outlined. Using the high-resolution capabilities of an aberration-corrected STEMwith sub-Angstrom resolution, Fig. 1B is an atomicallyresolved high-angle annular dark-field image of the individual Sr, Ti,and O atoms, preferentially oriented along the [100] zone axesconfiguration illustrated in Fig. 1C by atomic species. Figure 1D is a fastFourier transformation (FFT) of the atomically resolved image takenalong the same orientation, along with this preferred crystallographicdirection. On the basis of the FFT, Fig. 1E is the two-dimensional azi-muthal integration of the pattern transformed into a one-dimensionalprofile. The pattern and profile provide the structural classification de-tails for classifying and predicting the structure using our deep-learningmodel approach.

Exploiting deep learning–based classification for crystallographicinformation builds and trains on public and established materialsdatabases, including the Open Crystallography Database, MaterialsProject Database, AmericanMineralogist Crystallographic Databases,and Inorganic Crystal Structure Database (30–35). Figure 1F is the

on March 14, 2020


aded from

Fig. 1. Neural network data architecture and workflow for crystal space group determination from experimental high-resolution atomic images and diffractionprofiles. (A) Anymaterial in an STEM, in this case crystalline STO islands distributed on a rock salt MgO substrate, can be simultaneously imaged with (B) high-resolution atomicmass contrast STEM imaging and (C) decoupled with a selective area. (D) FFT to reveal the material’s structural details. (E) On the basis of either electron diffraction or FFT of anatomic image, a two-dimensional azimuthal integration translates this information into a relevant one-dimensional diffraction intensity profile from which the relative peakpositions in reciprocal space can be indexed. arb. units, arbitrary units. (F) Seeding the prediction of crystallography is a hierarchical classification using a one-dimensionalconvolution neural network model replicated at (G), each layer from family to space group forming a nested architecture. On the basis of the derived peak positions in theazimuthal integration profile, the prediction on STO is reported in Table 2.

2 of 9



on March 14, 2020


Dow

nloaded from

one-dimensional CNN architecture used to train and evaluate a hierar-chical training dataset composed of 571,340 individual crystals dividedamong seven families, 32 genera, and 230 individual crystallographicspace groups. At each level of the hierarchy, we trained a neural networkto form a nested hierarchy for classification, as shown in Fig. 1G. EachCNNconsists of six convolutional blocks before three dense layers and aSoftmax layer for classification. Convolutional blocks are formed from aconvolutional layer, a max pooling layer, and an activation layer. Theconvolutional layers have a kernel size of two and start with 180 chan-nels, narrowing to 45 channels over the six blocks. Starting in the fourthblock, dropout is applied after the pooling layer. Dropout starts at 0.1and scales up to 0.2 and 0.3 in the fifth and sixth blocks, respectively.

On the basis of the input from the atomic-resolution STO STEMimage, the ranked order of predictions made from the FFT image–based profiles is as follows: 225 (Fm�3m), 221 (Pm�3m), 205 (Pa�3). Uponvalidation with the known crystal structure, we determined that thecrystal is structured as space group number 221, Pm�3m, subsequentlyoriented along the [100] zone axes containing the [200] and [110]family of crystallographic reflections. Using the data workflow andpipeline provides the generalized framework for classifying all knownmaterials and the flexibility to improve on the model in the future.

Training on diffraction data for hierarchicalstructural classificationInitially, there were approximately 650,000 individual structuresscreened for duplicates or other potential errors. A weighting schemawas applied for each class and assigned to address the occurrence ofoverly represented crystal types, noting substantial imbalances amongpopulated crystal families, groups, and space groups ranging from136,534 to less than 1000 (36–38). The aggregate accuracies and popu-lation statistics for each level of the hierarchy are reported in Table 1.We labeled crystal files by family, genera, and species and used thecorresponding label for different levels of the hierarchy to train andevaluate models. The table summarizes the population and accuracyoverall in seven crystal classes and 32-point symmetry groups. Each filesubsequently generated a distinct diffraction profile as a function of thecorresponding Bragg angle, using 571,340 crystal structures. We subse-quently generated interplanar d-spacings at each level of the hierarchyto train the model. The resolution in the individual binary signal, con-sisting of a normalized vector of intensity against the Bragg angle, wasset to 0.5°.

Figure 2A is the confusion matrices evaluated over 136,534 ran-domly chosen structures of 571,340 total structures at the family level,which includes examples of the best and worst classification schemes asshown in Fig. 2B.When compared at the genera level, cubic and ortho-rhombic confusion matrices illustrate the hierarchy and classificationscheme issues of imbalance in the data despite the weighting duringtraining. The example highlights the classification hierarchy analogousto nested network architecture capable of predicting structure at thefamily, genera, and space group levels. Predictions along the diagonalare true-positive predictions; a concentration of high percentages alongthe diagonal indicates an accurate model.

Optimization and modeling pipelining forcrystallographic analysisBecause of a lack of grand canonical examples or a human baseline,the deep-learning model was benchmarked by comparing it to othermachine-learning algorithms (39). Themodel architecture was tested atvarying depths, numbers of parameters, layer combinations, and


Table 1. Population statistics and training accuracy evaluated. Re-ported population for individual structures over family and genera con-stituting a database of 571,340 total structures. Note that N/A appears forthose genera with a single class.

Accuracy (%)
Population
Triclinic
91.04% 105,200
Pedial
N/A 16,740
Pinacoidal
N/A 88,460
Monoclinic
86.73% 217,156
Sphenoidal
86.50% 21,705
Domatic
74.95% 14,997
Prismatic
90.14% 180,454
Orthorhombic
75.75% 104,526
Rhombic-disphenoidal
77.18% 22,182
Rhombic-pyramidal
92.32% 19,722
Rhombic-dipyramidal
67.42% 62,622
Tetragonal
84.81% 40,770
Tetragonal-pyramidal
65.80% 1,081
Tetragonal-disphenoidal
76.46% 1,437
Tetragonal-dipyramidal
96.23% 5,112
Tetragonal-trapezohedral
82.99% 2,373
Ditetragonal-pyramidal
88.16% 1,450
Tetragonal-scalenohedral
84.22% 4,309
Ditetragonal-dipyramidal
81.14% 25,008
Trigonal
82.78% 31,252
Trigonal-pyramidal
81.48% 3,499
Rhombohedral
90.46% 6,017
Trigonal-trapezohedral
94.13% 2,321
Ditrigonal-pyramidal
82.90% 5,831
Ditrigonal-scalenohedral
89.43% 13,584
Hexagonal
86.07% 24,147
Hexagonal-pyramidal
88.17% 1,828
Trigonal-dipyramidal
90.14% 453
Hexagonal-dipyramidal
N/A 2,100
Hexagonal-trapezohedral
99.24% 930
Dihexagonal-pyramidal
92.01% 2,969
Ditrigonal-dipyramidal
95.36% 3,230
Dihexagonal-dipyramidal
93.64% 12,637
Cubic
95.47% 48,289
Tetartoidal
96.75% 1,842
Diploidala
90.56% 1,389
Gyroidal
93.68% 737
Hextetrahedral
97.14% 6,475
Hexoctahedral
87.29% 37,846
3 of 9



on March 14, 2020


Dow

nloaded from

dropout rates. The final model architecture selected consists of sixblocks of convolution. Max pooling and dropout were selected on thebasis of performance, the number of trainable parameters, and preser-vation of spatial information and then by tuning the last block of denselayers. We further determined three dense layers before classificationwas optimal for producing the highest accuracy.

During hyperparameter optimization, a batch size of 1000 waschosen in conjunction with weighting by the class occurrence to in-crease the prevalence of rarer classes in the gradient of each batch. Thenumber of peaks used to classify structure had a significant effect onthe prediction accuracy. Figure 3 compares the number of peaks pres-ent with the measured accuracies for each family based on the accom-panying confusion matrices. The confusion matrices are organizedacross seven crystal families at the genera level, which constitutesthe class hierarchy, followed by the number of peaks used.An identicaleffect is seen at the space group level as well. Comparisons to othermachine-learning methods, including decision forests, support vectormachines, and the Naïve Bayes model, are reported in table S1.

We further evaluated the deep-learningmodel for real-time analysison a single graphical processing unit (GPU) desktop machine to eval-uate the efficiency, sensitivity, and computing necessity for performingaugmented crystallographic determination in an accessible manner. Thesingle GPU machine used at runtime had a GTX 1050ti 3-GB graphicscard and a 3.2-GHz i7 quad-core processor. Running on this single GPUdesktop, the model can classify batches of 1000 profiles loaded insequence at a rate of 2600 to 3500 predictions/s. Conversely, when thesame profiles are loaded individually, the model classifies significantlymore slowly at a rate of 29 predictions/s. Classification speeds are thesame for predictions across families, genera, or space groups. Predictingspace groups for 48,289 observations consisting of a single family took


13.2 s, which is equivalent to 3525 predictions/s. Subsequently, the abil-ity to classify structure from diffraction in subsecond times allows forsignificant acceleration in the acquisition and prediction of at least twoorders of magnitude based on the current ability of experts to predictand augment the analysis without previous knowledge.

Diffraction analyses for high- to low-symmetry materialsHighlighting a generalized workflow over multiple crystal families, wereport multiple experimental validations. Researchers used both high-resolution imaging and selective area electron diffraction (SAED) tostudy cubic, hexagonal, tetragonal, and orthorhombic structures.Figure 4A is the experimental high-resolution image series of nano-crystalline cubic fluorite CeO2 to evaluate predictions based on cubiccrystal structures. Figure 4B is a low-magnification bright-field high-resolution electronmicroscope image and accompanying SAED pat-tern for a two-dimensional monolayer of graphene, supported on aholey silicon nitride support film. Figure 4C is a TEM SAED patternof the topological insulating material Bi1.15Sb 0.71Te0.85Se2.29 (BSTS).Figure 4D is a-phase uranium, which constitutes an orthorhombicstructure, space group 63, and international shorthand symbolCmcm,studied under four separate zone axis orientations as a series of SAEDpatterns input into the CNN model. Accompanying this series ofmaterials is the predictions reported in Table 2.

DISCUSSIONDeep learning for crystallographic prediction fromhigh-resolution imaging and diffraction dataA generalized workflow and accompanying network for crystallo-graphic materials such as STO at the nanometer scale in Fig. 1A

Fig. 2. Model evaluation from low- to high-symmetrymaterials. Given the 571,340 total structures, 136,534 randomly chosen structures were used to evaluate themodel atthe family level, finding a high concentration primarily along the 1:1 diagonal. At one level down to the genera level, we further compare the labeled best and worst confusionmatrices from cubic and orthorhombic crystal families, highlighting the highest and lowest level of accuracies trained into the nested neural network architecture. See theSupplementary Materials for accompanying confusion matrices over all crystal families and genera.

4 of 9



on March 14, 2020


Dow

nloaded from

provide the necessary capability to derive crystallographic structurefrom high-resolution images. Figure 1B is an atomic-resolution imagethat translates images into an accompanying crystallographic patternin Fig. 1C, where an azimuthal integration in Fig. 1D resolves the in-dividual interatomic d-spacings and accompanying Bragg angles forsubsequent crystal prediction and refinement. The input to the modelseeds a deep-learning model for nested prediction, using over 571,340crystals to provide a capability for deriving crystal structure. Top-twoaccuracy is used to discriminate between classes.

The deep-learningmodel was validated on the basis of experimentaldata acquired in various modalities using several different microscopes.On the basis of the input from atomic-resolution STO, the model pre-dicts space group 225 (Fm�3m) and 221 (Pm�3m) as the two most likelyspace groups. Through further analysis, we determined that the crystalis structured as space group number 221, Pm�3m, oriented along the[100] zone axes containing the [200] and [110] family of crystallograph-ic reflections.

Training on diffraction data for hierarchicalstructural classificationInitially, there were approximately 650,000 structure-based files thatneeded to be cleaned and simulated. The files were screened forformatting errors, missing essential information, and simulation errors.The files were used to simulate diffraction profiles while they were


checked against the crystal information file (CIF) file to ensure propersimulation. The profiles were further refined to a binary signal of peakpositions. The simulated signals contained several prominent peaks anddozens of lesser peaks that may not be present in all experimental set-tings and differed between electron, x-ray, and neutron experiments. Inthe case of electron microscopy, the intensity does not necessarily scaleagainst known structure factors and is strongly affected bymaterial tex-turing. For full-scan x-ray and neutron data in which the intensitiesscale against known structure factors, an applied threshold to removepeaks below the signal-to-noise ratio to seed the prediction with fewerpeaks leads to predicting crystal with a high degree of accuracy, wellabove 80%. The prominent diffraction peaks are the most reliableindicator of the structure. The previous crystallography analysis toolsfurther corroborate the model. On the basis of a binary representationof the data as a function of peak position, we trained the hierarchicalmodel on signals for each family and genera, removing peak intensityas a variable, which simplifies the representation.

Moving to the binary representation of peak positions eliminated theintensity of the peaks and allowed the models to be applied to severaldiffraction-basedmodalities. A simulated diffraction profile consisted oflattice spacings as a function of either Angstroms or integrated Bragg.Because of constraints on the number of learnable parameters, theBragg angle resolution was 0.5°. On the basis of a survey of peaksat 0.5°, it is a reasonable resolution for classification in the cases of

Fig. 3. Optimizing the CNN and pipeline for real time. The number of peaks used per family benchmarks the model’s sensitivity and accuracy. On the basis of theseconfusion matrices, a strict threshold to select four peaks is used in the current implementation. The result is the highest level of accuracy, predictive speed, andcompression of structural-based information from electron microscopy images and diffraction patterns.

5 of 9



on March 14, 2020


Dow

nloaded from

60- to 300-kV electron beams, typically operating voltages of modernelectron microscopes.

If space group classification can be learned from peak locationsalone, then aggregate signals for each family, genera, and species weresummed across all members and quantitatively compared. Theaggregate signals over crystal family are shared in fig. S7. Peak distribu-tions over genera or families at each level uniquely differentiate notablefeatures. At the family level of the hierarchy, significant overlaps in dis-tributions and similarities among genera caused predictions amongfamilies to be unreliable. Once the family of the crystal is determined,prediction accuracies rise into the 80 to 98% range.

Preprocessing the data yielded 571,340 simulated signals. The signalsare not uniformly distributed among all classes at any level of the hier-archy. There is a strong preference for higher-symmetry structureswithin the seven crystal families at the genera level, but there doesnot appear to be a preference at the space group level. The model wastrained on the materials contained within each database. The disparityin membership between classes introduces challenges into trainingdeep-learning models.

Datasets that are highly imbalanced are susceptible tomode collapse.In such a case, predicting themost common class yields a high accuracywithout discriminating betweenmaterials. To counteract the imbalancepresent in the crystallographic data, we imposed a weighting schema intraining. During training, the relative presence of classificationwas usedto compute a weight that would be applied during the loss calculation.Theweight was defined as the ratio of the total number of structures perspace group over all space groups. The sameweighting schemawas per-formed at the point group level.

In cases where the dataset is relatively balanced, the number ofexamples in each class is within the same order of magnitude; theweights are similar enough that they do not bias the model. When adataset is highly imbalanced, the number of examples in each classdiffers by more than an order of magnitude. This schema penalizes


the model for incorrectly predicting a rarer space group more harshlythan it rewards the model for correctly predicting a common spacegroup. Weighting the rare classes to be more important to the modelduring training had an ameliorating effect on the data imbalance butdid not eliminate it. Models trained without this weighting schemasuffered mode collapse and would not predict rare classes. To accountfor and further mitigate mode collapse from data imbalance, modelswere trained on top-one accuracy but evaluated on top-two accuracy.Misclassifications are predominantly to the common class, and top-twoaccuracy of the ranked predictions allows the model, in many cases, tocorrect for misclassifications due to imbalances in the data. The relativescore from the output of the Softmax layer determines the rank order.The confidence in the ranked prediction is based on themodel accuracyduring testing, not the relative score.

Optimizing a deep-learning framework forcrystallographic predictionInitial test models showed that attempting to classify directly to speciesproduced models with poor accuracy and compounding effects frommode collapse. Using a hierarchical model that assumes family, thenpredicts genera, and then species leads to higher accuracy at each step.Even with cascading error, the method is substantially more accurate.Comparing the confusion matrices from each family highlights a highlevel of accuracy. In Fig. 2A, there is a high degree of diagonalization inthe confusion matrices, indicating correct predictions.

Without a canonicalmodel or baseline accuracy,model benchmark-ing was performed by comparing results with other machine-learningalgorithms. CNNs outperformed other machine-learning methods, in-cluding decision forests, support vector machines, and the Naïve Bayesmodel. In certain situations, random forests appear to outperformCNNs. Upon delving into the random forest models, it is revealed thatrandom forestmodels are subject tomode collapse. Despite the fact thatmodels have a high accuracy above 80%, the model has not learned to

Fig. 4. Evaluation and validation over low- to high-symmetry materials with either high-resolution electron imaging or diffraction. The model has beenevaluated on (A) a cubic polycrystalline CeO2 using high-resolution imaging, (B) SAED of graphene at 60 kV, and (C) BSTS, a quantum-based topological insulator.(D) Rounding out the series is the orthorhombic a-phase uranium studied using selected area electron diffraction from four separate zone axes for the same materialinside a high-resolution FEI Titan STEM. The predictions are reported in Table 2.

6 of 9



on March 14, 2020


Dow

nloaded from

distinguish classes. It predicts the class that comprises 80% of the dataevery time. CNNs are susceptible tomode collapse as well, which occursmost prominently when classifying a crystal into a family.

Optimizing the deep-learning model required tuning varying ar-chitectural and training hyperparameters. Architectural parametersincluded depth, number of parameters, layer combinations, and drop-out rates. The final architecture consists of six blocks of convolution.Max pooling and dropout were selected on the basis of performance,many trainable parameters, and preservation of spatial information.The last block of dense layers was tuned, and three dense layers beforeclassification were optimally tuned, producing the highest accuracy.

During hyperparameter optimization, a batch size of 1000 waschosen in conjunction with weighting by the class occurrence to in-crease the prevalence of rarer classes in the gradient of each batch.The number of peaks used to classify the structure (see Fig. 3) wasthe hyperparameter thatmost significantly affected accuracy. The num-ber of peaks included is determined by a threshold of peak intensity ap-plied to the simulated patterns. Stricter thresholds, 80 to 90% intensityof themost prominent peak, produced signalswith fewer peaks. Relaxedthresholds below 50% produced signals with increasingly many peaks.Using a threshold stricter than 90% of max peak intensity almost uni-versally eliminates all but the maximum peak.

Whenoptimizing the structural parameters, we found that introduc-ing dropout in early layers of the network prevented it from learningfrom the sparse signals. Typically, dropout is implemented to preventoverfitting of data by ignoring portions of noisy signals. A heavily pro-cessed binary peak signal that only contains peak locations for three tosix peaks is used in training. The binary peak signal is generated fromthe azimuthal integration of an FFT or selected area diffraction pattern(SADP) as a rotationally invariant profile where individual peak loca-tions can be identified. A number of peak-finding techniques can beused; however, experience has led us to implement and tool a movingwindow type max voting peak finder that populates a binary signalsampled at less than 0.03 Å in real d-spacing. Such sparse vector repre-sentations of the data, including dropout early in themodel, could elim-inate the signal before propagating to learnable features. This resulted inworse models. Dropout was introduced gradually starting in the thirdconvolution block, starting at 0.10 and increasing to 0.30, in the last con-volutional layers to prevent overfitting. The six blocks of convolution,max pooling, and dropout condense the signal but maintain the spatialrelevance of the original data.


To supplement the data and provide amore robust training regimenfor the limited data, we opted to use cross-validation instead of split-ting the data into single training, testing, and validation sets. For cross-validation, the data were split into 10 folds; for each fold, we trained amodel on the other nine folds. The resulting models were comparedwith each other to test for overfitting and generalization. This processwas repeated for each level of the hierarchy. Validation was performedusing experimental data that the models had not seen in training orevaluation. The training and tuning of models were performed on ahigh-performance NVIDIA DGX-1 system. However, to ensure thatthemodel could be usable in a settingwhere high-performance comput-ing resources were not available, we performed speed benchmarking onan entry-level single GPU desktop.

Optimizing the deep-learning model foraugmented diffractionAlongside predictive accuracy, we considered performance on singleGPU machines during model tuning and design. Although trainedon an high-performance computing (HPC) machine, the model wasdesigned to be deployable on any readily available single GPUmachineor an entry-level cloud-based end point. Although the model is capableof classifying at a rate exceeding 3500 predictions/s for a large batch ofpreprocessed diffraction signals, this does not consider the time neces-sary to process diffraction patterns into the appropriate input form. Themodel is capable of handling large backlogs of data with this predictivespeed, but a more realistic workflow for newly generated data would berunning small batches or sequentially predicting for each observation.When analyzing each diffraction image through the full pipeline, in-cluding the azimuthal integration and peak-finding algorithms, thereis a significant slowdown in predictive speed, as is expected. This isthe case in a current workflow from raw to the processed peak position.At 22 predictions/s, it is possible to achieve real-time analysis of a livecamera feed.

Evaluating the predictive nature of deep learningfor materialsAfter training and tuning the models on a synthetic dataset, it was nec-essary to validate the models on experimental data. We selected severalwell-knownmaterials with known crystallographic structures represent-ative of ongoing materials research programs. The sparse sampling ofmaterials allows us to validate the processing pipeline and hierarchical

Table 2. Crystallographic prediction of experimental images and diffraction patterns. Comparing from high- to lower-symmetry materials, this shows theprediction for each material.

Material
Expectation First prediction Second prediction Third prediction
CeO2

Cubic Fm�3m
225 219 221
No. 225
87.1% 5.6% 3.6%
C graphene
Hexagonal P63/mmc 194 173 191
No 194
90.1% 2.5% 0.2%
BSTS
Trigonal R�3m 166 163 148
No. 166
26.2% 25.7% 3.48%
a-Phase U
Orthorhombic Cmcm 74 19 63
No. 63
34.7% 33.7% 15.9%
7 of 9



on http://advances.sciencem

ag.org/D

ownloaded from

model. These materials range from cubic to orthorhombic, whereineach case is shown in Fig. 4. Thematerials are representatives of a crystalfamily.We started with higher-symmetry cubic crystals in Fig. 4A fromlow to high resolution for nanocrystalline CeO2. On the basis of eachaccompanying FFT, in Table 2, we report the top three predictions witha relative ranking score of each prediction included in parenthesis. Forfluorite-based CeO2, space group 225, we predicted Fm�3m within thetop three for each FFT in Table 2.

Beyond cubic structures, graphene constitutes a hexagonal lattice ofindividual carbon atoms. Figure 4B is a SAED pattern taken along theprimary c axis, where the deep-learning model predicts the structure asspace group 194, P63/mmc in the top three in Table 2. Predictive capa-bilities were explored in Fig. 4C for a quantum topological insulatingmaterial, BSTS, which is arranged in the trigonal structure with spacegroup 53 (R�3m). Using the same model subsequently for lowestsymmetry, the deep-learning model predicts the structure as spacegroup 166, R�3m, in the top three in Table 2 for all four zone axes re-ported in Fig. 4D for a-phase uranium. Studied with SAED along theprimary c axis, the deep-learning model predicts the structure as spacegroup 63, Cmcm, in the top three reported in Table 2.

For all of these predictions, themodel displays a level of sensitivity tothe number of peaks used for classification. In several cases, more thanfour peaks were detected. Usingmore than four peaks can present someambiguity as to which peaks should be used for classification without apriori knowledge. In these cases, we can use simple heuristics to narrowdown the selection of peaks. In the case of electron diffraction, we haveignored peak below the resolution limit 0.5 Å and above 8 Å are diffrac-tion limited. Despite this range, this is a robust model to performgeneralized crystallographic classification. For example, in the case ofBSTS, two different, valid sets of peaks were fed into the model, gener-ating two different sets of predictions. The correct space group of 166,R�3m, was in the top two results for each set, but the third predictionchanged. Future explorations of these different possible detected peakcombinations may lead to further improvements of the model and re-moval of ambiguities in the predictions.

March 14, 2020

CONCLUSIONOur results andmodel report on how to reliably extract crystallographicstructural information in diffraction using a deep-learning nestedframework.We have demonstrated howCNNs can adequately accountfor imaging and diffraction data to effectively extend the limits ofhuman-centric analysis. As a result, even peaks in diffracted low-signalto noisy images canpotentially be extracted. Thiswould confirmpreviousexpectations that a combination of machine- and image-processingroutines increases the applicability of electron microscopes anddiffraction-based techniques for materials research. On the basis ofthe results, we report on the classification of crystal structures from aslow as four distinct Bragg reflections contained in either an atomic-resolution image or diffraction pattern. The simplicity and efficiencyof themethod, capable of predicting crystallographic structure, reduceartifacts and robustly address the need for efficient cross-validation,surpassing limitations in crystallography of unknown materials.

MATERIALS AND METHODSHigh-resolution TEM and diffractionMicroscopy was collected and assembled for training using Angstrom-to nanometer-sized probes to collect high-resolution images and dif-


fraction patterns from an FEI Titan, JEOL ARM, and JEOL 2800, eachoperated at 60 or 200 kV and capable 0.8-Å resolution. This includedlow-, medium-, and high-angle annular dark-field STEM ranging from1024 by 1024 to 2048 by 2048 pixel images with sub–100-ms dwell timesto examine samples.

For electron-transparent TEM, thin samples of nanocrystallineCeO2 and a-phase uranium were prepared using an FEI Helios DualBeam focus ion beam (FIB) instrument. These samples were coatedwith a layer of C to improve sample conductivity andminimize sampledrift inside the FIB. Inside the FIB, a gradient of fine- to coarse-grainedion beamplatinum layers were deposited over of an area 15 mmby 3 mmwith a thickness of 3 mm. Thin foil lift-out proceeded over this rectan-gular area with final lamellae measured as 13 mmby 5 mmby 100 nm intotal thickness. The final lamellae were lifted and mounted to a molyb-denum Omniprobe TEM grid for examination. A final cleaning wasperformed using a 5-kV gallium beam and a beam current of 12 pAto remove material deposits during FIB preparation to reduce millingdamage.Nanoparticle samples ofCe8Gd2O19were crushed, filtered, anddispersed onto a TEM holey carbon grid for examination. Researchersminimized experimental imaging and diffraction artifacts. The cameraswere set up with quadrant-based gain normalization, and the noisethreshold was characterized for each camera. The cameras were keptat their operating temperatures for long periods of time before carryingout the experiments to achieve thermal equilibrium, and long waitperiods were enforced between acquisitions to confirm that the darkcounts were as constant as possible over the time required for a fullexperimental acquisition. In this fashion, only unstructured whitenoise was observed at short (<20ms) and longer (>1 s) time durationsafter dark reference removal. After this initial setup, subsequentimages and diffraction patterns were collected without adjusting anybeam or camera parameters.

Collected two-dimensional diffraction patterns and fast Fourier–transformed diffraction patterns were individually azimuthallyintegrated and converted into a one-dimensional profile. A small countoffset between 0 and 1000 counts was applied randomly to the camerato minimize systematic noise. Individual members of the diffraction se-ries were then aligned, and the background profile was subtracted.Provided the acquisition time for individual images is sufficiently small,systematic or correlated noise (e.g., high voltage instabilities, cameranoise, and vibrations) is effectively suppressed.

SUPPLEMENTARY MATERIALSSupplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/5/10/eaaw1949/DC1Fig. S1. Confusion matrix for monoclinic genera and space group.Fig. S2. Confusion matrix for orthorhombic genera and space group.Fig. S3. Confusion matrix for tetragonal family and genera 15.Fig. S4. Confusion matrix for trigonal family and genera 20.Fig. S5. Confusion matrix for hexagonal family and genus 25.Fig. S6. Confusion matrix for cubic family and genera 31.Fig. S7. Aggregate signals for triclinic, monoclinic, orthorhombic, and cubic families plottedagainst two theta values.Table S1. Comparing models for crystallographic determination.

REFERENCES AND NOTES1. S. V. Kalinin, B. G. Sumpter, R. K. Archibald, Big–deep–smart data in imaging for guiding

materials design. Nat. Mater. 14, 973–980 (2015).2. H. Yang, I. MacLaren, L. Jones, G. T. Martinez, M. Simson, M. Huth, H. Ryll, H. Soltau,

R. Sagawa, Y. Kondo, C. Ophus, P. Ercius, L. Jin, A. Kovács, P. D. Nellist, Electron

8 of 9

http://advances.sciencemag.org/cgi/content/full/5/10/eaaw1949/DC1

http://advances.sciencemag.org/cgi/content/full/5/10/eaaw1949/DC1



on March 14, 2020


Dow

nloaded from

ptychographic phase imaging of light elements in crystalline materials using Wignerdistribution deconvolution. Ultramicroscopy 180, 173–179 (2017).

3. A. Belianinov, R. Vasudevan, E. Strelcov, C. Steed, S. M. Yang, A. Tselev, S. Jesse,M. Biegalski, G. Shipman, C. Symons, A. Borisevich, R. Archibald, S. Kalinin, Big data anddeep data in scanning and electron microscopies: Deriving functionality frommultidimensional data sets. Adv. Struct. Chem. Imaging 1, 6 (2015).

4. S. Gao, P. Wang, F. Zhang, G. T. Martinez, P. D. Nellist, X. Pan, A. I. Kirkland, Electronptychographic microscopy for three-dimensional imaging. Nat. Commun. 8, 163 (2017).

5. M. J. Humphry, B. Kraus, A. C. Hurst, A. M. Maiden, J. M. Rodenburg, Ptychographicelectron microscopy using high-angle dark-field scattering for sub-nanometre resolutionimaging. Nat. Commun. 3, 730 (2012).

6. H. Yang, R. N. Rutte, L. Jones, M. Simson, R. Sagawa, H. Ryll, M. Huth, T. J. Pennycook,M. L. H. Green, H. Soltau, Y. Kondo, B. G. Davis, P. D. Nellist, Simultaneous atomic-resolution electron ptychography and Z-contrast imaging of light and heavy elements incomplex nanostructures. Nat. Commun. 7, 12532 (2016).

7. P. Denes, J.-M. Bussat, Z. Lee, V. Radmillovic, Active pixel sensors for electron microscopy.Nucl. Instrum. Methods Phys. Res., Sect. A 579, 891–894 (2007).

8. M. Battaglia, D. Contarato, P. Denes, P. Giubilato, Cluster imaging with a direct detectionCMOS pixel sensor in transmission electron microscopy. Nucl. Instrum. Methods Phys.Res., Sect. A 608, 363–365 (2009).

9. T. A. Caswell, P. Ercius, M. W. Tate, A. Ercan, S. M. Gruner, D. A. Muller, A high-speed areadetector for novel imaging techniques in a scanning transmission electron microscope.Ultramicroscopy 109, 304–311 (2009).

10. M. Belkin, P. Niyogi, Laplacian eigenmaps for dimensionality reduction and datarepresentation. Neural Comput. 15, 1373–1396 (2003).

11. L. Parra, K.-R. Mueller, C. Spence, A. Ziehe, P. Sajda, Unmixing hyperspectral data.Adv. Neural Inf. Process. Syst. 12, 942–948 (2000).

12. C.-T. Chu, S. K. Kim, Y.-A. Lin, Y. Y. Yu, G. Bradski, A. Y. Ng, K. Olukotun, Map-reduce formachine learning on multicore. Adv. Neural Inf. Process. Syst. 19, 281–288 (2006).

13. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille, DeepLab: Semantic imagesegmentation with deep convolutional nets, atrous convolution, and fully connectedCRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2018).

14. M. Abadi, Ashish Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis,J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray,C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke,V. Vasudevan, F. Viegas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng,TensorFlow: Large-scale machine learning on heterogeneous distributed systems.arXiv:1603.04467 [cs.DC] (14 March 2016).

15. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell,in Proceedings of the 22nd ACM International Conference on Multimedia (ACM, 2014),pp. 675–678.

16. D. M. Moore, R. C. Reynolds, X-ray Diffraction and the Identification and Analysis of ClayMinerals (Oxford Univ. Press Oxford, 1989), vol. 322.

17. R. A. Young, The Rietveld Method (International Union of Crystallography, 1993), vol. 5.18. B. D. Cullity, S. R. Stock, Elements of X-ray Diffraction (Pearson Education, 2014).19. G. M. Sheldrick, SHELXT–integrated space-group and crystal-structure determination.

Acta Crystallogr. A Found. Adv. 71, 3–8 (2015).20. J. Portillo, E. F. Rauch, S. Nicolopoulos, M. Gemmi, D. Bultreys, Precession electron

diffraction assisted orientation mapping in the transmission electron microscope.Materials Science Forum 644, 1–7 (2010).

21. E. F. Rauch, J. Portillo, S. Nicolopoulos, D. Bultreys, S. Rouvimov, P. Moeck, Automatednanocrystal orientation and phase mapping in the transmission electron microscope on thebasis of precession electron diffraction. Z. Kristallogr. Cryst. Mater. 225, 103–109 (2010).

22. N. George, S.-g. Wang, Neural networks applied to diffraction-pattern sampling. Appl. Opt.33, 3127–3134 (1994).

23. P. M. Voyles, Informatics and data science in materials microscopy. Curr. Opin. Solid StateMater. Sci. 21, 141–158 (2017).

24. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. arXiv:1512.03385 [cs.CV] (10 December 2015).

25. G. Huang, Z. Liu, K. Q. Weinberger, Densely connected convolutional networks, in 2017 IEEEConference on Computer Vision and Pattern Recognition (CVPR, 2017), pp. 2261–2269.

26. E. Shelhamer, J. Long, T. Darrell, Fully convolutional networks for semantic segmentation.IEEE Trans. Pattern Anal. Mach. Intell. 39, 640–651 (2017).

27. M. Yu, A. B. Yankovich, A. Kaczmarowski, D. Morgan, P. M. Voyles, Integratedcomputational and experimental structure refinement for nanoparticles. ACS Nano 10,4031–4038 (2016).


28. A. J. Logsdail, Z. Y. Li, R. L. Johnston, Development and optimization of a novel geneticalgorithm for identifying nanoclusters from scanning transmission electron microscopyimages. J. Comput. Chem. 33, 391–400 (2012).

29. Z. Saghi, M. Benning, R. Leary, M. Macias-Montero, A. Borras, P. A. Midgley, Reduced-doseand high-speed acquisition strategies for multi-dimensional electron microscopy.Adv. Struct. Chem. Imag. 1, 7 (2015).

30. S. Gražulis, A. Daškevič, A. Merkys, D. Chateigner, L. Lutterotti, M. Quirós,N. R. Serebryanaya, P. Moeck, R. T. Downs, A. Le Bail, Crystallography Open Database(COD): An open-access collection of crystal structures and platform for world-widecollaboration. Nucleic Acids Res. 40, D420–D427 (2012).

31. S. Gražulis, D. Chateigner, R. T. Downs, A. F. T. Yokochi, M. Quirós, L. Lutterotti,E. Manakova, J. Butkus, P. Moeck, A. Le Bail, Crystallography open database – anopen-access collection of crystal structures. J. Appl. Crystallogr. 42, 726–729 (2009).

32. R. T. Downs, M. Hall-Wallace, The American mineralogist crystal structure database.Am. Mineral. 88, 247–250 (2003).

33. S. Gražulis, A. Merkys, A. Vaitkus, M. Okulič-Kazarinas, Computing stoichiometricmolecular composition from crystal structures. J. Appl. Crystallogr. 48, 85–91 (2015).

34. A. Merkys, A. Vaitkus, J. Butkus, M. Okulič-Kazarinas, V. Kairys, S. Gražulis, COD::CIF::Parser:An error-correcting CIF parser for the Perl language. J. Appl. Crystallogr. 49, 292–301(2016).

35. A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter,D. Skinner, G. Ceder, K. A. Persson, Commentary: The Materials Project: A materialsgenome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).

36. S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier,K. A. Persson, G. Ceder, Python Materials Genomics (pymatgen): A robust, open-sourcepython library for materials analysis. Comput. Mater. Sci. 68, 314–319 (2013).

37. A. H. Larsen, J. J. Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Dułak, J. Friis,M. N. Groves, B. Hammer, C. Hargus, E. D. Hermes, P. C. Jennings, P. B. Jensen, J. Kermode,J. R. Kitchin, E. L. Kolsbjerg, J. Kubal, K. Kaasbjerg, S. Lysgaard, J. B. Maronsson, T. Maxson,T. Olsen, L. Pastewka, A. Peterson, C. Rostgaard, J. Schiøtz, O. Schütt, M. Strange,K. S. Thygesen, T. Vegge, L. Vilhelmsen, M. Walter, Z. Zeng, K. W. Jacobsen, The atomicsimulation environment—A Python library for working with atoms. J. Phys. Condens. Matter29, 273002 (2017).

38. P. Juhás, C. Farrow, X. Yang, K. Knox, S. Billinge, Complex modeling: A strategy andsoftware program for combining multiple information sources to solve ill posed structureand nanostructure inverse problems. Acta Crystallogr. A Found Adv. 71, 562–568 (2015).

39. A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet classification with deep convolutionalneural networks. Commun. ACM 60, 84–90 (2017).

Acknowledgments: We thank I. Harvey for the many useful discussions and contributions tothis work. M. Patel and H. Yoon are acknowledged for providing samples and data for analysis.A portion of this research was conducted at the Center for Nanophase Materials Sciences,which is a DOE Office of Science User Facility (R.R.U.). We also acknowledge D. Masiel, B. Reed,K. Jungjohann, S. Misra, J. Gu, and R. Mariani for the helpful discussions. Funding: Thiswork was supported through the INL Laboratory Directed Research and Development (LDRD)Program under DOE Idaho Operations Office Contract DE-AC07-05ID145142. Authorcontributions: J.A.A., R.R.U., and B.D.M. performed the microscopy and measurements.T.T. and M.L.G. were involved in developing the deep-learning model for diffraction analysis.J.A.A. and M.L.G. processed the experimental data, performed the analysis, drafted themanuscript, and designed the figures. R.R.U. and B.D.M. shared the samples and characterizedthem with high-resolution electron microscopy. T.T. aided in interpreting the results andthe machine-learning model and worked on the manuscript. All authors discussed the resultsand commented on the manuscript. Competing interests: The authors declare that theyhave no competing interests. Data and materials availability: All data needed to evaluatethe conclusions in the paper are present in the paper and/or the Supplementary Materials.Additional data related to this paper may be requested from the authors. The software is opensource. It can be downloaded at https://github.com/SCIInstitute/DiffractionClassification.git.

Submitted 27 November 2018Accepted 1 October 2019Published 30 October 201910.1126/sciadv.aaw1949

Citation: J. A. Aguiar, M. L. Gong, R. R. Unocic, T. Tasdizen, B. D. Miller, Decoding crystallographyfrom high-resolution electron imaging and diffraction datasets with deep learning. Sci. Adv. 5,eaaw1949 (2019).

9 of 9

https://arxiv.org/abs/1603.04467



https://github.com/SCIInstitute/DiffractionClassification.git


deep learningDecoding crystallography from high-resolution electron imaging and diffraction datasets with

J. A. Aguiar, M. L. Gong, R. R. Unocic, T. Tasdizen and B. D. Miller

DOI: 10.1126/sciadv.aaw1949 (10), eaaw1949.5Sci Adv

ARTICLE TOOLS http://advances.sciencemag.org/content/5/10/eaaw1949

MATERIALSSUPPLEMENTARY http://advances.sciencemag.org/content/suppl/2019/10/25/5.10.eaaw1949.DC1

REFERENCES

http://advances.sciencemag.org/content/5/10/eaaw1949#BIBLThis article cites 32 articles, 1 of which you can access for free

PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions

Terms of ServiceUse of this article is subject to the

is a registered trademark of AAAS.Science AdvancesYork Avenue NW, Washington, DC 20005. The title (ISSN 2375-2548) is published by the American Association for the Advancement of Science, 1200 NewScience Advances

License 4.0 (CC BY-NC).Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial Copyright © 2019 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of

on March 14, 2020


Dow

nloaded from

http://advances.sciencemag.org/content/5/10/eaaw1949

http://advances.sciencemag.org/content/suppl/2019/10/25/5.10.eaaw1949.DC1

http://advances.sciencemag.org/content/5/10/eaaw1949#BIBL

http://www.sciencemag.org/help/reprints-and-permissions

http://www.sciencemag.org/about/terms-service


Decoding crystallography from high-resolution electron ... · was applied for each class and...

Documents

Transcript of Decoding crystallography from high-resolution electron ... · was applied for each class and...