Superresolution in Fluorescence and Diffraction Microscopies with

Multiple Illuminations

- Jules Girard -

2 December 2011

1/27

Introduction : Imaging with optics and

resolution

Probing function

Parameter of interest

× 𝑓 𝑖𝑙𝑙𝑀= 𝑓 𝑜𝑏𝑗 ∗𝑝𝑠𝑓()

Dete

ctor

Imaging device

×~𝑝𝑠𝑓

(~𝑓 𝑜𝑏𝑗 ∗~𝑓 𝑖𝑙𝑙)( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙) ~

𝑀 𝑀Low-pass filter

~𝑀=(~𝑓 𝑜𝑏𝑗∗~𝑓 𝑖𝑙𝑙)×~𝑝𝑠𝑓

FT

𝑀=( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙)∗𝑝𝑠𝑓 ~𝑀=(~𝑓 𝑜𝑏𝑗∗~𝑓 𝑖𝑙𝑙)×~𝑝𝑠𝑓

ky

kx

~𝑓 𝑜𝑏𝑗 (~𝑓 𝑜𝑏𝑗 ∗

~𝑓 𝑖𝑙𝑙)

→

Introduction : Extend resolution with

illumination

More generally :

=

~𝑓 𝑖𝑙𝑙

∗ ky

kx

ky

kx

×~𝑝𝑠𝑓

W. Lukosz and M. Marchand, Optica Acta 10, 241-255 (1963).W. Lukosz, JOSA 56, 1463 (1966).

By using multiple and inhomogeneous illuminations, we can shift high frequency parts of the object spatial spectrum into the passband defined

by the psf

2/27

Introduction : Reconstruct a super-resolution

image𝑀 𝑖= ( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙 ,𝑖 )∗𝑝𝑠𝑓 ~𝑀 𝑖=(~𝑓 𝑜𝑏𝑗∗~𝑓 𝑖𝑙𝑙 , 𝑖)×~𝑝𝑠𝑓

Inversion → numerical data processing

2 cases

is known is unknown

Non-linear inversion Find both and with the use of constraints

3/27

(𝑖=1. .𝑁 )

« Direct » inversionwith analytical approach

Presentation Outline

I. Optical Diffraction Tomography

II. Structured Illumination Fluorescence Microscopy

4/27

𝑀=( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙)∗𝑝𝑠𝑓 ~𝑀=(~𝑓 𝑜𝑏𝑗∗~𝑓 𝑖𝑙𝑙)×~𝑝𝑠𝑓

=

I. Optical Diffraction Tomography

Objective

Fourier Space

𝜽 𝒊𝒏𝒄

𝐸𝑡𝑜𝑡 ( 𝑥 , 𝑦 ,𝑧 )=𝐸𝑟𝑒𝑓 ( 𝑥 , 𝑦 ,𝑧 )+𝐸𝑑 ( 𝑥 , 𝑦 ,𝑧 )

II. Optical Diffraction Tomography

≠ illuminations → ≠ → access to ≠ parts of

~𝐸𝑑(�⃗�)=

~𝑀= (~𝑓 𝑜𝑏𝑗 ( �⃗� )∗~𝑓 𝑖𝑙𝑙 ( �⃗� ))×~𝑝𝑠𝑓 ( �⃗�)

We measure :

~Δ 𝜀(Sample dielectric

permittivity contrast)

~Etot(Total internal electric field)

= 0 for lateral

frequencies >

Reconstruct : quantitative microscopy of unstained sample

�⃗�

5/27

𝑥

𝑧

E Wolf, Optics Communications 1, 153-156 (1969).

V Lauer, Journal of Microscopy 205, 165-76 (2002).

Laser (λ=633n

m)

CC

D

Phase modulator

(G. Maire, F. Drsek, H.Giovannini)

Sample

Experiment

Calibration and normalization

→

Inversion : ) →

II. Optical Diffraction Tomography

Illumination with « plane waves » under ≠ incidences

Measure complex values of

6/27

Low : Born Approximation

is diffraction limited → Abbe limit

High : Multiple Scattering Regime

depends on object and illumination

is not diffraction limited → resolution improvement ? ( ?)

1. 2.

~Δ 𝜀

~Etot

=

𝒌𝟎𝑵𝑨 𝒌𝟎𝑵𝑨𝟐𝒌𝟎 𝑵𝑨𝒅=𝝀 /(𝟐𝑵𝑨)

II. Optical Diffraction Tomography

7/27

air

glass

50 nm

50 nm

25 nm

zx

50°

simulations

• λ = 633 nm

• Abbe limit with NA = 1.5

→ 211 nm

= 10-2

Low

|𝐸 𝑡𝑜𝑡| = 28.8

High (Ge)

<

!

|𝐸 𝑡𝑜𝑡|

II. Optical Diffraction Tomography

8/27

11

air

glass

50 nm

50 nm

25 nm

zx Germanium rods

TIRF configuration (10

angles)

NA = 1.3

→ Abbe limit : 245 nm

Experimental validation

(A. Talneau – LPN)

II. Optical Diffraction Tomography

0

Z (

µm

)

0,5

0

0,5

Z (

µm

)

9/27

II. Optical Diffraction Tomography

We achieved quantitative reconstruction of the

permittivity map of unstained sample even with a

multiple scattering regime

Multiple scattering : drawback way to improve the

resolution of ODT far beyond diffraction limit

Conclusion

10/27

II. Structured Illumination in

Fluorescence microscopy on 2D

samples

(2D)

III. Structured Illumination Microscopy in

Fluorescence

Objective Tube Lense

CC

D

𝑀=( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙)∗𝑝𝑠𝑓

(field intensity)(fluorescence density)

(2D and 1D)

−2𝑘0 𝑁𝐴+2𝑘0 𝑁𝐴 𝑘𝑥

0,5

1

0𝑘𝑥

𝑘𝑦

11/27

ky

kx

~𝑓 𝑜𝑏𝑗

∗~𝑓 𝑖𝑙𝑙

Use periodic pattern →

III. Structured Illumination Microscopy in

Fluorescence𝑀=( 𝑓 𝑜𝑏𝑗× 𝑓 𝑖𝑙𝑙)∗𝑝𝑠𝑓

𝐼𝜌

~𝑀

¿�⃗�

~𝑀

R. Heintzmann and C. Cremer, SPIE, pp. 185-196. (1998)

Mats G L Gustafsson, Journal of Microscopy 198, 82-7

(2000).

Requirements for illumination pattern :• Accurate translation → needed for discrimination of the

3 copies • High contrast → higher SNR (no dim for shifted copies

of ) 12/27

Use of non-linearities : →

(R. Heintzmann et al., JOSA A, 19, 2002 & M G L Gustafsson, PNAS, 102, 2005)

High index substrate

→ limited n and/or absorption

Nanostructured devices with plasmonics

→ field bound to the structure + difficulties to cover a large area

III. Structured Illumination Microscopy in

FluorescenceLimit : Illumination pattern is diffraction limited :

= : twice better than classical WF

How can we reach higher frequencies ?

Get below diffraction limit (surface imaging)

13/27

Glass coverslip@ 633nm

a-Si layer @ 633nm

 

𝒙

z=0 

   

  

   

 𝒚

  𝑘 𝑦 

 

 

𝑘𝑥

 

  

 

Grating assisted Structured Illumination Microscopy

Dielectric resonant grating ≈ 2D waveguide + 2D sub-λ grating

�⃗�𝒊𝒏𝒄

�⃗�𝒎𝒐𝒅𝒆

Hexagonal geometry : 6 equivalent orientations → near isotropic

resolution

III. Structured Illumination Microscopy in Fluorescence

𝑧

Design optimization → numerical simulations14/27

Gratings fabrication process

 

2. Grating patterning(e-beam + RIE)

1. aSi deposition

(PECVD)

3. Planarization(A. Cattoni)

(J. Girard, A. Talneau, A. Cattoni LPN – CNRS)

A. Cattoni, A. Talneau, A-M Haghiri-Gosnet, J. Girard, A. Sentenac (oral presentation, MNE 2011)15/2

7

III. Structured Illumination Microscopy in Fluorescence

III. Structured Illumination Microscopy in

FluorescenceExcitation modes of the grating substrate

�⃗�𝒊𝒏𝒄

�⃗�𝒎𝒐𝒅𝒆

�⃗�𝒊𝒏𝒄

�⃗�𝒎𝒐𝒅𝒆

1 beam excitation 2 beams excitation

|𝐾 −1,0|≈1.3×(2𝑘𝑂 𝑁𝐴)

|𝐾 −1,0+2 �⃗�𝑖𝑛𝑐∥|<2𝑘𝑂 𝑁𝐴

|2 [ �⃗�− 1,0+�⃗�𝑖𝑛𝑐 ∥ ]|≈1.6×2𝑘𝑂 𝑁𝐴

rightleft

17/27

III. Structured Illumination Microscopy in

Fluorescence Control of orientation, phase

and incidence angle on the substrate (65°)

Dich

roïc

Mirr

or

Obje

ctive

(O

il, N

A 1.

49)

Experimental setup

16/27

III. Structured Illumination Microscopy in

FluorescenceGrating characterization : SNOM measurements

Stretched fiber

65°

z=

1 beam excitation

�⃗�𝒊𝒏𝒄

�⃗�𝒎𝒐𝒅𝒆

�⃗�𝒊𝒏𝒄

�⃗�𝒎𝒐𝒅𝒆

(Geoffroy Scherrer, ICB, Dijon)

High Frequency Pattern from the Grating

18/27

Grid Shifting

Theoretical simulation

simulation

III. Structured Illumination Microscopy in

FluorescenceGrating characterization : Far field fluorescence measurements

2 beams excitation : Low frequency component of the intensity pattern

WF Fluorescence observation with ~homogeneous layer of fluorescent beads

19/27

III. Structured Illumination Microscopy in

Fluorescence Our manufactured gratings can produce a grid of light with

180 nm period (λ/3.5) (down to 147 nm, λ/4.3 with alternative design)

a high contrast

The possibility to shift its position

According to , a final resolution

of up to 87 nm could be reached at λ =633 nm!

However we need to know the illumination pattern for inversion procedure

=

20/27

24

“Blind” SIM Inversion

M 1= ( 𝐼1×𝜌 )∗ 𝑃𝑆𝐹M 2= ( 𝐼 2×𝜌 )∗𝑃𝑆𝐹

M n=( 𝐼𝑛×𝜌 )∗ 𝑃𝑆𝐹…

equations

unknowns :

1𝑁 ∑

𝑖=1

𝑁

𝐼𝑛=𝐼 0

+1

F ( 𝜌 , 𝐼 1 ,…, I n )=∑𝑖=1

𝑁

|𝑀 𝑖− [ ( 𝐼𝑖×𝜌 )∗ 𝑃𝑆𝐹 ]|2(Emeric Mudry & Kamal Belkebir)

Iterative optimization of estimates of and

through minimization of a cost function :

21/27

III. Structured Illumination Microscopy in Fluorescence

Observation of fluorescent beads (Ø 90nm) immersed in glycerin with

classical SIM

Experimental validation

 

 

WF image Our Result

Optimized « analytical » algorithm

Inversion by Pr. R. Heintzmann

Deconvolution of the WF image

22/27

III. Structured Illumination Microscopy in Fluorescence

|~𝐼|

𝐼

Simulation Measurement

Speckle illumination

1𝑁 ∑

𝑖=1

𝑁

𝐼𝑛 (𝑥 , 𝑦 )𝑁→∞→

𝐼 0

1. Contains every accessible frequencies

2. Known average illumination

3. Experiment far simpler than standard SIM

Speckle pattern is a perfect candidate for SIM with our ‘blind’ inversion

algorithm

23/27

III. Structured Illumination Microscopy in Fluorescence

object WF image

One measured image

N ≈ 80

Speckle illumination : simulations

Photon budget : average of 130

photon/pixel/imageReconstructed

=

=

Deconvolution

Deconvolution

=

speckles

speckles

24/27

III. Structured Illumination Microscopy in Fluorescence

Rabbit Jejunum slices (150nm thick) (Cendrine Nicoletti, ISM, Marseille)

TEM image of a similar sample WF image

Reconstructed image from 100 speckle illuminations

Deconvolution of WF image

Speckle illumination : experimental results

25/27

III. Structured Illumination Microscopy in Fluorescence

General Perspectives

I.Optical Diffraction Tomography :

Extend to 3D samples

Use other configuration (grating substrate, mirror substrate…)

II. Structured Illumination in Fluorescence Microscopy

1. Grating-assisted SIM :

Make super-resolved images of real samples : use a priori

information for inversion procedure

2. Speckle illumination :

Extend to 3D samples

27/27

SIM with unknown illumination patterns

Extension of SIM to the use of random speckle patterns

Not effective yet for grating-assisted SIM (inhomogeneous

average illumination)

26/27

III. Structured Illumination Microscopy in Fluorescence

Conclusion

Thanks…

Geoffroy Scherrer Anne Talneau Andrea Cattoni

The whole MOSAIC team for advices, seminars, discussions, equipment, facilities…

Eric Le Moal Guillaume Maire Emeric Mudry Kamal Belkebir Anne Sentenac

Thank you for your attention

33

II. Optical Diffraction Tomography

air

Si

300 nm

z

x

100 nm

110

NA = 0.7 (used up to 0.53 only for illumination)

→ Abbe limit : 500 nm (450nm for full NA)

AFM profile

Reconstructed map Reconstructed profileReconstructed profile

(linear inversion)

34

II. Optical Diffraction Tomography

Multiple scattering and resolution

= 28.8

(Germanium) (a) = 2 (b) = 7 (c) = 14

Modulation of for the object 2 :

Simulation of () =()

for a plane wave illumination (incidence 50°)

= 10-2

100nm

25nm

35

III. Structured Illumination Microscopy in Fluorescence

Grating assisted SIM : getting some images

Problem with inversion : Intensity pattern is not perfectly known

Speckle algorithm is not able to retrieve frequencies >

Add of a priori information (rough orientation and frequencies)