original presentation

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy

Imaging Technology Group Visualization, Media and Imaging Laboratory

Microscopy Suitewww.itg.uiuc.edu

Don Appleman, Visiting Research Programmer, Imaging Technology Group

April 1, 2003

Constructing a Research Grade Interactive Remote Microscopy

Application:

Goals, Challenges, and Results

ITG Forum

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



ITG Mission

Mission of the Imaging Technology Group

• Provide state-of-the-art imaging facilities for Institute researchers.

• Develop advanced imaging technologies with emphasis on projects in remote and automated instrument control.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Remote Instrumentation

Remote access to imaging instrumentation

Why?

• Unique and/or expensive instrumentation • Distributed expertise • Outreach; Education; Training of professional

researchers and engineers

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



What is Bugscope?

What is Bugscope?• Web-based access to an

Environmental Scanning Electron Microscope (ESEM) for K-12 classrooms to image “bugs”.

• Schools in 33 states have used the ESEM over the Internet in more than 120 sessions using the current software.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Current Technology

Initial Implementation

• Separate Controller & Observer software• Chat Window for conversing with microscopist• Java applets or CGI interface to microscope server software• Still images only

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Current Technology

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Current Technology

Drawbacks

• Still images only• Time to acquire image is long

This results in the following problems …

• Difficulty navigating across the sample• Difficulty focusing• Loss of concentration on the task at hand

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Current Technology

Current Project

• The current project is a remote instrumentation project funded by NASA.

• NASA is interested in the remote operation of complex scientific equipment over long distances in real time.

• The project goal is to enhance the Bugscope software, known as the Interactive Remote Microscopy Application (IRMA), to the point where it qualifies as a research-grade interface.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Future Technology

Approach — Improved Input & Output• Input — Sufficient control of the microscope

+• Output — True-to-life viewing experience

=

Goals• Remote and collaborative research among

isolated researchers• Training new users to operate the ESEM• Teaching classes ranging from nanoscience to

biology

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Future Technology

Proposal• Add streaming video to Bugscope

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Future Technology

Proposal• Add streaming video to Bugscope

Challenges• Bandwidth requirements• Image quality• Image latency• Image size

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

For a given bandwidth capacity, pick any two

• Image qualityFidelity of the reconstructed image to the original

source

• Image latencyDelay in real-time between acquisition of the image and

display of the image to the remote user

• Image sizeThe size of the image displayed to the remote user

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Videocast options

• Broadcast– Video stream sent to everyone on a

limited network.– Server bandwidth equivalent to a single

stream.– Creates lots of network traffic.– Not compatible with Internet delivery to

arbitrary users.

• Unicast– Dedicated video stream sent to a single,

known client.– Serves a single client.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Videocast options, cont’d

• Multi-unicast– Multiple dedicated video streams sent to multiple clients.– Server bandwidth multiplied by number of simultaneous users.– Client bandwidth multiplied by number of simultaneous users

on given Internet feed (classroom full of students).

• Multicast– Video stream sent to abstract address, replicated by routers.– Server bandwidth equivalent to a single stream.– Client bandwidth equivalent to a single stream for Internet

feed.– Requires properly configured, compatible routers.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Videocast solution

Hybrid Video Server• Initiate multicast• Listen for client connections• Inform clients of the active

multicast address• If the client reports that it cannot

see the multicast– Start a dedicated unicast to client

address– Inform client of unicast address:port– Monitor client status

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Project-specific Advantages• No audio• Monochromatic source

The solution• H263 video-conferencing video

format• Video server written in Java

(using Java Media Framework)• Java applet video client (adjunct

to extant software)

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Project-specific Results, low bandwidth

• Video stream bandwidth ~100Kbps

• Image quality acceptable (some quality lost, still image needed for actual study); good enough for fine focus control using current interface

• Image latency less than or equal to 500 ms

• Image size 352x288

• Video frame rate ~15 frames per second (variable bit rate)

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Streaming Video

Project-specific Results, high bandwidth

• Video stream bandwidth ~3Mbps

• Image quality very good (JPEG quality 0.8, still image usually not needed for actual study)

• Image latency less than or equal to 500 ms

• Image size 640x480

• Video frame rate ~15 frames per second (variable bit rate)

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Beyond Video

A new, more complete and more natural user interface.

Expose the majority of the controls accessible when using the ESEM directly.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Beyond Video

New controls• Beam on/off, kV, and spot size• Scan mode (Slow scan 1, 2, 3, or TV

mode)• Filters (Live, Integrate 1, Average 4)• Detectors (Secondary electron,

Backscatter electron, CCD camera)

Improved controls• New Navigation, Focus, and

Measurement Tools

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Beyond Video

Not all controls can be included

• Cannot change specimens

• Cannot activate vacuum

• Cannot operate manual tilt mechanism

An operator is required to prepare and mount the specimen, and to activate the vacuum.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Navigation Tool

ESEM Navigation

Click anywhere relative to the green overlay circle, and the stage will move in that direction.

In “TV” mode, click & hold moves smoothly in given direction.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Navigation Tool

Bugscope Navigation

• Control+click on Observer to center image on selected location

• Controller stage control

• Controller presets

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Navigation Tool

Proposed Navigation

• Click & hold over video window to move as if in “TV” mode on ESEM

• Keyboard or other control to move by ½ screen in any direction

• Control+click on video display to center image on selected location

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Focus Tool

ESEM Focus

Click & drag right mouse button left/right to increase/decrease focus distance.

Control+click & drag right mouse button left/right or up/down to adjust stigmator x,y.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Focus Tool

Bugscope Focus

• Coarse control +/- to adjust focus by 1mm

• Fine control to adjust by 0.1mm to 0.9mm

• ESEM Auto Focus (slow, can be unreliable)

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Focus Tool

Proposed Focus (same as ESEM focus)

• Using video window, click & drag right mouse button left/right to increase/decrease focus distance.

• Using video window, control+click & drag right mouse button left/right or up/down to adjust stigmator x,y.

• May also use trio of images, side by side, each showing a different focal depth that the user can select.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



User Base

Varying user base

• Educational Outreach (Bugscope)– Bandwidth limited (usually)– Little or no experience

operating a real ESEM– Not fully accountable– Chat with professional required

• Professional Researchers– Lots of bandwidth (usually)– May have experience operating

a real ESEM– Higher level of accountability– Professional assistance

unneeded

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



User Base

Varying user base, solution

Configurable control set• For educational outreach, eliminate controls

that are confusing, hard to use, or which might damage the equipment.

Configurable video server• Configure to use available bandwidth

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



Conclusions

Conclusions:

• With the addition of streaming video to Bugscope, the issue of realism for the output of the IRMA software has been addressed.

• Once we complete the enhancements to the input interface of the Controller, IRMA will be a true, research-ready option for usage of the ESEM.

• With the requirement that an operator be present to prepare and mount the specimens, and to operate the vacuum controls, IRMA will remain limited.

• As a proof-of-concept of remote instrumentation capabilities, the completed project is a success.

Beckm

an

Institu

te fo

r Ad

van

ced

Scie

nce a

nd

Tech

nolo

gy



References

http://bugscope.beckman.uiuc.edu

The Bugscope web site

Technical reports

http://bugscope.beckman.uiuc.edu/publications/index.htm

original presentation

Documents

Transcript of original presentation