Advances and challenges in cryo ptychography at theAdvanced Photon Source

J. Deng⇤, D. J. Vine†, S. Chen†, Y. S. G. Nashed⇤⇤, Q. Jin‡, T. Peterka⇤⇤, S. Vogt†

and C. Jacobsen†,§

⇤Applied Physics, Northwestern University, Evanston IL 60208, USA†X-ray Science Division, Advanced Photon Source, Argonne National Laboratory, Argonne, IL 60439, USA⇤⇤Mathematics and Computing Science Division, Argonne National Laboratory, Argonne, IL 60439, USA

‡Department of Physics & Astronomy, Northwestern University, Evanston, IL 60208, USA§Department of Physics & Astronomy and Chemistry of Life Processes Institute, Northwestern University,

Evanston, IL 60208, USA

Abstract.Ptychography has emerged as a nondestructive tool to quantitatively study extended samples at a high spatial resolution.

In this manuscript, we report on recent developments from our team. We have combined cryo ptychography and fluorescencemicroscopy to provide simultaneous views of ultrastructure and elemental composition, we have developed multi-GPU parallelcomputation to speed up ptychographic reconstructions, and we have implemented fly-scan ptychography to allow for fasterdata acquisition. We conclude with a discussion of future challenges in high-resolution 3D ptychography.Keywords: ptychography, cryogenic sample, parallel computation, fly scan, 3D ptychographyPACS: 07.85.Qe, 42.30.Rx, 61.05.cp, 87.59.-e

INTRODUCTION

Ptychography has been developed as a powerful coherent diffraction imaging (CDI) technique to reveal the details ofextended objects at a high spatial resolution [1, 2]. In ptychography, a confined illumination probe is scanned acrossan extended object with some overlap between neighboring illuminated points, while diffraction patterns are recordedfrom each scan point. From these diffraction data, one can use a phase retrieval algorithm to reconstruct the object’scomplex transmission function with a spatial resolution not limited by the optics but by the highest spatial frequenciesmeasured in the diffraction patterns. In addition, the probe function can be reconstructed, which allows ptychographyto be used for the characterization of focused beams [3]. Due to the short wavelength and high penetration of x-rays,x-ray ptychography has proven to be especially promising over a broad range of applications: it has been used to imagesubcellular structures of biological samples [4, 5], strains and dislocations of crystals [6, 7], magnetic domains in thinfilms [8], and chemical components of samples using x-ray absorption edges [9, 10]. When combined with angularprojections, ptychographic tomography can provide 3D quantitative measurements of electron density [11, 12].

Although tremendous progress in ptychography has been made in recent years, there have still been limitationsdue to coherent flux, radiation damage, and the speed of data acquisition and reconstruction. We present here someadvances in ptychography we have achieved at the Advanced Photon Source (APS) to address these challenges. Weconclude with a discusison of the issues involved in high-resolution 3D ptychography.

SIMULTANEOUS CRYO PTYCHOGRAPHY AND FLUORESCENCE IMAGING

X-ray fluorescence microscopy (XFM) offers high sensitivity for quantitative mapping of elements in biologicalsamples. However, a majority of biological structures are not easily visualized in XFM since XFM is relatively blind tothe light elements (such as H, C, N, O) which are the main constituents of biological materials. Ptychography deliversquantitative phase contrast and high spatial resolution, providing a good solution to visualize biological ultrastructure.While the spatial resolution of ptychography can in theory reach the wavelength limit, radiation damage to biologicalsamples affects the obtainable resolution. A good solution towards this problem is to work with frozen-hydratedbiological specimens under cryogenic conditions. Cryogenic samples are better able to withstand beam-induced

degradation and can provide high-fidelity structural and ionic elemental preservation, as well mitigate the effectsof radiation damage. Therefore, we combined the above techniques (cryogenic conditions, fluorescence microscopy,and ptychography) together on the Bionanoprobe [13], a hard x-ray fluorescence nanorobe with cryogenic capabilities,to image frozen-hydrated biological samples. In this combination experiment, the fluorescence spectra and far-fielddiffraction patterns were recorded simultaneously from a frozen-hydrated Chlamydomonas alga scanned by a 5.2 keVx-ray focused beam with a spot size of ⇠ 85 nm [14]. Figure 1(a) shows P, S, K, and Ca fluorescence images, where thepresence of K within the cell suggests good preservation of membrane integrity in the cryogenic sample preparation.The ptychographic image (Fig. 1(b)) reveals details of the organelles and membranes not seen in the fluorescence maps.With the combination of ptychographs and elemental maps, identification of the cellular components and quantificationof elemental concentrations can become easier.

2 μm

P

SK

Ca

(a)

1 μm

0 0.76(rad)

(b)

1 μm

(c)

GPU1 GPU2

GPU3 GPU4

GPU5 GPU6

FIGURE 1. Simultaneous x-ray fluorescence and ptychographic imaging of a frozen-hydrated Chlamydomonas reinhardtii algaobtained from a 167⇥151 point scan data [14]. (a) X-ray fluorescence maps reveal the distribution of several elements in thealga. (b) The ptychographic reconstruction shows the biological ultrastructure at sub-30 nm resolution. The ptychographic imagereveals features not seen in the elemental maps and adds rich contextual information. (c) Multiple-GPU parallel reconstruction. The167⇥151 diffraction patterns were partitioned across several GPUs to reduce processing time from hours to minutes, after whichthe reconstruction results from each GPU are stitched together to form one image as shown in (b).

RAPID PTYCHOGRAPHY RECONSTRUCTION WITH MULTI-GPUS PARALLELCOMPUTATION

Smaller nanofocused beam sizes and larger scan areas are leading to a rapid increase in the size of ptychographydatasets for reconstruction. This is especially obvious in the combined ptychography-fluorescence approach shownabove: the focused beam needs to be as small as possible for high-resolution fluorescence imaging, so the small beamsize combined with the ptychographic overlap requirements leads to a large volume of datasets. The computationaland memory requirements will become even more challenging when ptychography is extended to 3D imaging, sinceptychographic scans will need to be acquired at many projection angles. Therefore, a fast and real-time ptychographyreconstruction implementation is highly desirable. We have developed a rapid reconstruction code with multi-GPUimplementation for real-time ptychographic phase retrieval [15]. Figure 1(c) shows an example of partitioning the167⇥151 diffraction patterns (⇠ 6.2 GB) across 6 GPUs for independent reconstructions. The reconstruction resultsfrom each GPU were then stitched together as one (Fig. 1(b)) by a post-procedure of registration and phase normaliza-tion. Moreover, a function of information sharing on the neighboring boundaries instead of across all reconstructionarea [16] is also provided in this developed code, which allows GPUs to communicate their results to each other duringthe reconstruction, yielding a result as if the reconstruction is performed using one GPU. The image reconstruction canbe sped up by more than a factor of 100 by employing GPUs on one compute note, and can be faster when distributingthe task over more than 100 nodes in the cluster. This parallel reconstruction method allows one to do real-time dataanalysis, which is extremely helpful to guide the running ptychography experiment.

FLY-SCAN PTYCHOGRAPHY FOR FAST DATA COLLECTION

X-ray ptychography was originally developed using a step-scan mode, in which the scanning microscope worked ina move-settle-acquire sequence for data acquisition at each point. The motor motion and settle time contributes anoverhead time on scans, slowing down data acquisition. With diffraction limited storage rings expected to providehundredfold gains in coherent flux [17], exposure times will decrease accordingly so the percentage of scan overheadcould be larger than 90%. To address the scan overhead problem, a continuous motion approach has been proposed[18] and demonstrated [19, 20, 21] in ptychography. In fly scan ptychography, the probe moves relative to the objectcontinuously within a scan line, while the data is acquired by detectors with a very small time interval of detector deadtime. This continuous scan mode has already been well used in incoherent imaging systems, such as X-ray fluorescencemicroscopy [22]. However, diffraction patterns acquired from a continuously moving sample become blurry withdegraded speckle visibility, bringing difficulties for ptychographic reconstruction. A recent important study [23] hasshown that mixed-state issues can be well addressed using multiple object and/or probe modes in reconstruction. Sincefly-scan diffraction patterns can be described as an incoherent sum of diffractions from discrete positions, the movingobject can be recovered by using multiple probe modes in the reconstruction. We have explored the limits of probeoverlap in fly-scan ptychography with multiple probe mode reconstruction [20].

The speedup that fly-scan ptychography can provide will become especially important when ptychography isextended to do 3D imaging of the sample [11]. Figure 2 shows a demonstration of fly-scan ptychographic tomographyon a gold test sample with a thickness of 200 nm; the resolution of a single projection image (Fig. 2(a)) is about 18 nmas determined by line-cut method. The whole 3D dataset of 31 projections with 1200 scan points each was acquiredwithin 1.5 hours, compared to the 10 hours that would have been required for step-scan ptychography. The phasecontrast images from the 2D projections were then combined using maximum likelihood expectation maximization(MLEM) reconstruction algorithm to yield a tomographic three-dimensional view of the sample (see Fig. 2(b)).

The ability to collect both ptychographic and fluorescence data in fly-scan mode enables fast multi-modal imaging.Fast data acquisition provided by the fly-scan ptychography combined with rapid parallel reconstruction allows forhigh-throughput ptychography, which will become increasingly important as brighter x-ray sources are developed.

500 nm

x

yz

(a) (b)

FIGURE 2. Fly-scan 3D ptychography on a gold test sample with a thickness of 200 nm. 31 projections of 40⇥30 scan pointseach were recorded in fly-scan mode with a 5.2 keV focused beam of ⇠ 100 nm, spanning a range of tilt angles from �60� to60� with 4� angular spacing; the acquisition time per projection (covering 2⇥3 µm) is about 3 min with an exposure time of 110ms per point, yielding the total collection time of ⇠ 1.5 hours for the whole 3D dataset. (a) Projection image reconstructed fromptychographic data at 0� using 5 probe modes. (b) 3D volume rendering of the tomographic reconstruction.

3D PTYCHOGRAPHY AND DEPTH OF FIELD LIMITS

3D ptychographic imaging has been demonstrated using reconstructed phase contrast projection images in standardtomographic reconstruction [11]. What ptychography offers in this approach is a quantitative phase contrast image ofextended objects, at a resolution beyond that of the probe-forming lens or pinhole. However, as spatial resolution of 3Dptychography continues to improve by, for example, the use of samples at cryogenic conditions (to mitigate radiationdamage) and faster fly-scan approaches (to obtain sufficient angular sampling), the validity of treating ptychographicmeasurements as pure projection images becomes challenged.

In the case of a lens-based imaging system, the transverse resolution cutoff dt of a coherent image is limited by thenumerical aperture q of the lens by dt = 0.5l/q , where l is the wavelength. The depth of field (DOF) goes like [24]DOF= l/q 2 (a factor of 2 beyond what is given by Chapman et al. [25]), giving a relationship of

DOF = 4d 2

t

l. (1)

A necessary condition for the projection approximation in ptychography to be valid is when the sample thickness isless than the depth of field; in other words, the depth of field limits the maximum thickness of the sample for pureprojection images at a desired resolution. Figure 3 plots the depth of field with respect to the transverse resolution atseveral x-ray energies. Each DOF line limits the maximum sample thickness for pure projection imaging at the desiredresolution at that specific photon energy. As the spatial resolution is improved to smaller values, the correspondingDOF (or maximum sample thickness) decreases as the square of these improvements. This brings challenges tonanoscale 3D ptychography of samples with a size of microns to tens of microns, which is precisely the thicknessrange where x-ray microscopy offers significant advantages over electron microscopy. 3D pychographic tomographyhas been pioneered at the cSAXS beamline operating at 6.2 keV at the Swiss Light Source; we show as data points inFig. 3 their achieved transverse resolution for objects with various thicknesses, along with our first demonstration of3D fly-scan ptychography (Fig. 2). All of these experiments have been within the DOF thickness limit of Eq. 1, withthe experiment by Holler et al. being closest to the limit (for details see Supplementary Information of [12]).

Transverse resolution (nm)

Dep

th o

f fie

ld /

obje

ct th

ickn

ess

(μm

)

1 10 100 5000.01

0.1

1

10

100

12.5 k

eV

0.1 ke

V0.5 ke

V2.5 ke

V

DOF limit a

t 6.2 k

eV

5.2 ke

VFig.2

Holler 14 Silva 15

Wilke 12

Holler 12

Trtik 13

Dierolf 10 Diaz 12

Diaz 14

Zanette 15

6.2 ke

V

FIGURE 3. Depth of field (DOF; Eq. 1) as a function of the achieved transverse resolution at a series of photon energies. As thetransverse resolution improves, the depth of field decreases with the square of these improvements, putting a limit on the samplethickness for straightforward interpretation. The red curve is the DOF limit at 6.2 keV, the energy of ptychographic tomographydemonstrations (red dots; see [11, 26, 4, 27, 12, 28, 29, 30, 31]) at the cSAXS beamline at the Swiss Light Source routinely conductthe 3D ptychography. The blue dot represents our fly-scan 3D ptychography study (Fig. 2) at 5.2 keV.

One solution to overcome the above challenges in high-resolution 3D ptychography is using hard x-rays with higherenergy. Figure 3 shows that the depth of field increases with increases in photon energy for a fixed resolution. Hardx-rays have larger depth of field and better beam penetration as well, so that they are more suitable to study large

samples in 3D compared to soft x-rays, though one also has less contrast for features of a given thickness. Anotherapproach is to extend ptychography to deal with continuous 3D objects. A multislice reconstruction approach has beenused in ptychography to image separated depth planes [32, 33]. Is there an extension of this approach that can be usedfor ptychographic tomography of continuous 3D objects that extend beyond the depth of focus limit of Eq. 1? This isan important question for the future.

ACKNOWLEDGMENTS

We thank NIH National Institute of General Medical Sciences for partial support of this work under Grant1R01GM104530. The Bionanoprobe is funded by NIH/National Center for Research Resources High End Instru-mentation Grant 1S10RR029272-01 as part of the American Recovery and Reinvestment Act. Use of the AdvancedPhoton Source, an Office of Science User Facility operated for the US Department of Energy (DOE) Office of Scienceby Argonne National Laboratory, was supported by the US DOE under Contract DE-AC02-06CH11357.

REFERENCES

1. J. Rodenburg, A. Hurst, A. Cullis, B. Dobson, F. Pfeiffer, O. Bunk, C. David, K. Jefimovs, and I. Johnson, Physical ReviewLetters 98, 034801 (2007).

2. P. Thibault, M. Dierolf, A. Menzel, O. Bunk, C. David, and F. Pfeiffer, Science 321, 379–382 (2008).3. A. Schropp, P. Boye, J. M. Feldkamp, R. Hoppe, J. Patommel, D. Samberg, S. Stephan, K. Giewekemeyer, R. N. Wilke,

T. Salditt, J. Gulden, A. P. Mancuso, I. A. Vartanyants, E. Weckert, S. Schöder, M. Burghammer, and C. G. Schroer, AppliedPhysics Letters 96, 091102 (2010).

4. R. N. Wilke, M. Priebe, M. Bartels, K. Giewekemeyer, A. Diaz, P. Karvinen, and T. Salditt, Optics Express 20, 19232–19254(2012).

5. D. J. Vine, D. Pelliccia, C. Holzner, S. Baines, A. Berry, I. McNulty, S. Vogt, A. G. Peele, and K. Nugent, Optics Express 20,18287–18296 (2012).

6. P. Godard, G. Carbone, M. Allain, F. Mastropietro, G. Chen, L. Capello, A. Diaz, T. H. Metzger, J. Stangl, and V. Chamard,Nature Communications 2, 568–6 (2011).

7. Y. Takahashi, A. Suzuki, S. Furutaku, K. Yamauchi, Y. Kohmura, and T. Ishikawa, Physical Review B 87, 121201(R) (2013).8. A. Tripathi, J. Mohanty, S. H. Dietze, O. G. Shpyrko, E. Shipton, E. E. Fullerton, S. S. Kim, and I. McNulty, Proceedings of

the National Academy of Sciences 108, 13393–13398 (2011).9. A. M. Maiden, G. R. Morrison, B. Kaulich, A. Gianoncelli, and J. M. Rodenburg, Nature Communications 4, 1669 (2013).10. D. A. Shapiro, Y.-S. Yu, T. T. andJordi Cabana, R. Celestre, W. Chao, K. Kaznatcheev, A. L. D. Kilcoyne, F. M. andStefano

Marchesini, Y. S. Meng, T. Warwick, L. L. Yang, and H. A. Padmore, Nature Photonics 8, 765–769 (2014).11. M. Dierolf, A. Menzel, P. Thibault, P. Schneider, C. Kewish, R. Wepf, O. Bunk, and F. Pfeiffer, Nature 467, 436–439 (2010).12. M. Holler, A. Diaz, M. Guizar-Sicairos, P. Karvinen, E. Färm, E. Härkönen, M. Ritala, A. Menzel, J. Raabe, and O. Bunk,

Scientific Reports 4, 3857 (2014).13. S. Chen, J. Deng, Y. Yuan, C. Flachenecker, R. Mak, B. Hornberger, Q. Jin, D. Shu, B. Lai, J. Maser, C. Roehrig, T. Paunesku,

S. Gleber, D. Vine, L. Finney, J. Von Osinski, M. Bolbat, I. Spink, Z. Chen, J. Steele, D. Trapp, J. Irwin, M. Feser, E. Snyder,K. Brister, C. Jacobsen, G. Woloschak, and S. Vogt, Journal of Synchrotron Radiation 21, 66–75 (2014).

14. J. Deng, D. J. Vine, S. Chen, Y. S. G. Nashed, Q. Jin, N. W. Phillips, T. Peterka, R. Ross, S. Vogt, and C. J. Jacobsen,Proceedings of the National Academy of Sciences 112, 2314–2319 (2015).

15. Y. S. Nashed, D. J. Vine, T. Peterka, J. Deng, R. Ross, and C. Jacobsen, Optics Express 22, 223015 (2014).16. M. Guizar-Sicairos, I. Johnson, A. Diaz, M. Holler, P. Karvinen, H. C. Stadler, R. Dinapoli, O. Bunk, and A. Menze, Optics

Express 22, 14859–14870 (2014).17. M. Eriksson, J. F. van der Veen, and C. Quitmann, Journal of Synchrotron Radiation 21, 837–842 (2014).18. J. N. Clark, X. Huang, R. J. Harder, and I. K. Robinson, Optics Letters 39, 6066–6069 (2014).19. P. M. Pelz, M. Guizar-Sicairos, P. Thibault, I. Johnson, M. Holler, and A. Menzel, Applied Physics Letters 105, 251101

(2014).20. J. Deng, Y. S. G. Nashed, S. Chen, N. W. Phillips, T. Peterka, R. Ross, S. Vogt, C. Jacobsen, and D. J. Vine, Optics Express

23, 5438–5451 (2015).21. X. Huang, K. Lauer, J. N. Clark, W. Xu, E. Nazaretski, R. Harder, I. K. Robinson, and Y. S. Chu, Scientific Reports 5, 9074

(2015).22. E. Lombi, M. D. Jonge, E. Donner, C. G. Ryan, and D. Paterson, Analytical and Bioanalytical Chemistry 400, 1637–1644

(2011).23. P. Thibault, and A. Menzel, Nature 494, 68–74 (2013).24. Y. Wang, C. Jacobsen, J. Maser, and A. Osanna, Journal of Microscopy 197, 80–93 (2000).25. H. Chapman, A. Barty, S. Marchesini, A. Noy, S. P. Hau-Riege, C. Cui, M. Howells, R. Rosen, H. He, J. Spence, U. Weierstall,

T. Beetz, C. Jacobsen, and D. Shapiro, Journal of the Optical Society of America A 23, 1179–1200 (2006).

26. A. Diaz, P. Trtik, M. Guizar-Sicairos, A. Menzel, P. Thibault, and O. Bunk, Physical Review B 85, 020104 (2012).27. M. Holler, J. Raabe, A. Diaz, M. Guizar-Sicairos, C. Quitmann, A. Menzel, and O. Bunk, Review of Scientific Instruments 83,

073703 (2013).28. P. Trtik, A. Diaz, M. Guizar-Sicairos, A. Menzel, and O. Bunk, Cement & Concrete Composites 36, 71–77 (2013).29. A. Diaz, M. Guizar-Sicairos, A. Poeppel, A. Menzel, and O. Bunk, Carbon 67, 98–103 (2014).30. J. C. da Silva, K. Mader, M. Holler, D. Haberthür, A. Diaz, M. Guizar-Sicairos, W. Cheng, Y. Shu, J. Raabe, A. Menzel, and

J. A. van Bokhoven, ChemCatChem Communications 7, 413–416 (2015).31. I. Zanette, B. Enders, M. Dierolf, P. Thibault, R. Gradl, A. Diaz, M. Guizar-Sicairos, A. Menzel, F. Pfeiffer, and P. Zaslansky,

Scientific Reports 5, 9210 (2015).32. A. M. Maiden, M. J. Humphry, and J. M. Rodenburg, Journal of the Optical Society of America A 29, 1606–1614 (2012).33. A. Suzuki, S. Furutaku, K. Shimomura, K. Yamauchi, Y. Kohmura, T. Ishikawa, and Y. Takahashi, Physical Review Letters

112, 053903 (2014).