Trends on real time control for adaptive optics

EuropeanSouthernObservatory


© ESO© ESOPage 1 Enrico FedrigoEnrico Fedrigo

TRENDS ON REAL TIME CONTROL FOR ADAPTIVE OPTICS

TRENDS ON REAL TIME CONTROL FOR ADAPTIVE OPTICS

Enrico FedrigoEnrico Fedrigo




Source of inspirationSource of inspiration

Where to detect trends?Where to detect trends?The Real Time Control WorkshopThe Real Time Control Workshop

Garching December 4Garching December 4thth, 5, 5thth , 2012 , 2012

ESO Messenger 151, March 2013, pages 55-57ESO Messenger 151, March 2013, pages 55-57

ESO ELT Telescope RTCESO ELT Telescope RTCAdvanced prototype developedAdvanced prototype developed

ESO ELT Instrument RTC Development PlanESO ELT Instrument RTC Development PlanPlan under development based on Phase-A Instrument requirementsPlan under development based on Phase-A Instrument requirements

My own experienceMy own experience

Firenze, 26-31 May 2013; AO4ELT3Firenze, 26-31 May 2013; AO4ELT3




The Real Time Control WorkshopThe Real Time Control Workshop

22ndnd meeting, first in Durham 13 meeting, first in Durham 13thth, 14, 14thth April 2011 April 2011 66 registered participants, 20% commercial66 registered participants, 20% commercial 28 talks, 1 special invited talk, 7 sessions, 1 panel 28 talks, 1 special invited talk, 7 sessions, 1 panel

discussion, 2 free-form open discussionsdiscussion, 2 free-form open discussions 2 major topics: technology and algorithms2 major topics: technology and algorithms All talks here: All talks here:

http://www.eso.org/sci/meetings/2012/RTCWorkshop.htmlhttp://www.eso.org/sci/meetings/2012/RTCWorkshop.html





RTC Workshop: TechnologyRTC Workshop: Technology

DSPDSP


DeterministicMassively parallelDifficult to program (now)Expensive

High degree of parallelismSimpler to program but need internal

knowledgeInternally deterministicLacks I/O (but there is GPUdirect)Non standard, subject to vendor lock-inRelatively cheapVery fast evolution

Non deterministicMulti-core/many-coreEasiest to program, most difficult to optimizeCheapVery fast evolution




FPGAFPGA

TMT conceptTMT concept- Based on a commercial card with 8xVirtex6 from Based on a commercial card with 8xVirtex6 from

NutaqNutaq- Not the best match for MVMNot the best match for MVM- PowerfulPowerful





FPGAFPGA

Microgate productMicrogate product- Tailored to a specific product, adaptive mirrors, Tailored to a specific product, adaptive mirrors,

where COTS might not always be the best choicewhere COTS might not always be the best choice- Delivers the required performanceDelivers the required performance- Custom product: obsolescence managed in-Custom product: obsolescence managed in-

househouse





FPGAFPGA

ESO SPARTA productESO SPARTA product- Uses FPGA to manage communication and to Uses FPGA to manage communication and to

compress the input stream (WPU)compress the input stream (WPU)- Design of 2005, getting obsoleteDesign of 2005, getting obsolete- Still 80us is respectableStill 80us is respectable- It deliversIt delivers





CPUCPU

Durham RTC system DARCDurham RTC system DARC CPU-based with support for GPUCPU-based with support for GPU Good to test algorithmsGood to test algorithms Flexible, expandable Flexible, expandable Tested on skyTested on sky Interfaces to simulatorInterfaces to simulator


See talkSee talkFridayFriday




CPUCPU

Kiepenheuer Institute AO systemKiepenheuer Institute AO system Stock Linux with few tweaks to improve real-timeStock Linux with few tweaks to improve real-time Correlation Shack-HartmannCorrelation Shack-Hartmann FlexibleFlexible Tested on skyTested on sky Moving to FPGA?Moving to FPGA?





CPUCPU

ESO’s SPARTA all-CPUESO’s SPARTA all-CPU For VxWorks (partially available on Linux)For VxWorks (partially available on Linux) Runs on IntelRuns on Intel Can be turned to FPGACan be turned to FPGA Same supervisorSame supervisor

save investmentssave investments





ELT: the Telescope RTCELT: the Telescope RTC


Biggest CPU-based system so farBiggest CPU-based system so far Based on tweaked BSDBased on tweaked BSD Designed for a specific application Designed for a specific application




The Intel PhiThe Intel Phi

Recent product from IntelRecent product from Intel Dedicated to HPCDedicated to HPC Modest speed-up promised, Modest speed-up promised, Still you can put 8 of them in one machineStill you can put 8 of them in one machine First tests disappointingFirst tests disappointing Interesting for portabilityInteresting for portability Roadmap to be verifiedRoadmap to be verified





High performance on CPUsHigh performance on CPUs


Matteo Frigo, creator of FFTW and CilkMatteo Frigo, creator of FFTW and Cilk




GPUGPU


TNO proposal for an array of GPUsTNO proposal for an array of GPUs- MVM (cuBLAS or Fujimoto)MVM (cuBLAS or Fujimoto)- 4xGPU good for ELT SCAO4xGPU good for ELT SCAO- Uses external API or librariesUses external API or libraries




GPUGPU


TMT conceptTMT concept- MVM, 2xGPU per WFSMVM, 2xGPU per WFS- More GPUs on cluster systemMore GPUs on cluster system- Slow update of the control matrixSlow update of the control matrix

See talkSee talkWangWangFridayFriday




GPUGPU

LESIA projectLESIA project- Real time and Real time and simulatorsimulator on GPU on GPU- Addresses the latency problem with GPUdirectAddresses the latency problem with GPUdirect- Relies on Nvidia and CUDARelies on Nvidia and CUDA


See talkSee talkGratadourGratadour

todaytoday




Technological trendsTechnological trends


GPUs are the hottest technological componentGPUs are the hottest technological component The latency problem is being addressed by Nvidia with GPUdirectThe latency problem is being addressed by Nvidia with GPUdirect The roadmap is robust and proceeding at a fast paceThe roadmap is robust and proceeding at a fast pace Prototypes show ELT GLAO/SCAO can be targetedPrototypes show ELT GLAO/SCAO can be targeted

CPU: going the parallel wayCPU: going the parallel way Not always easy to manageNot always easy to manage

FPGA still important for high performance and special tasksFPGA still important for high performance and special tasks Communication layer, stream processingCommunication layer, stream processing Can implement a complete very high performance (==low latency) Can implement a complete very high performance (==low latency)

system; concepts can target EPICS full MVMsystem; concepts can target EPICS full MVM High level tools appearing (OpenCL, C-to-VHDL)High level tools appearing (OpenCL, C-to-VHDL)

Real Challenge: write a portable software that can benefit Real Challenge: write a portable software that can benefit from the advantages of each platformfrom the advantages of each platform




Transistor densityTransistor density

Transistors, Transistors, frequency, power, frequency, power, performance, and performance, and cores over time cores over time (1985-2010).(1985-2010).

Computer density: still growing Computer density: still growing Clock speed: haltedClock speed: halted

Performance: growing, slowerPerformance: growing, slowerPower density: haltedPower density: halted

Cre

dits

: C

omm

itte

e on

Sus

tain

ing

Gro

wth

in

Com

put

ing

Per

form

ance





ParallelismParallelism

Parallelism and distributed computing is needed. The Parallelism and distributed computing is needed. The 5 challenges:5 challenges:

Extract parallelism from algorithm, find independent Extract parallelism from algorithm, find independent execution branchesexecution branchesAmdahl law:Amdahl law:

LocalityLocality

CommunicationCommunication

SynchronisationSynchronisation

Load balancingLoad balancing





Amdahl lawAmdahl law

P=99%. N=256: How much ‘speedup’?

0 5 10 15 20 25 30 35 40 45 501

2

3

4

5

6

7

8

9

P=50%

P=60%

P=70%P=80%

P=90%

0 100 200 300 400 500 600 700 800 900 10000

10

20

30

40

50

60

70

80

90

100

P=50%

P=60%

P=70%P=80%

P=90%

P=99%72





Usage trendsUsage trends


GPUGPU Most groups stay away from the internals of the GPU and use them Most groups stay away from the internals of the GPU and use them

through standard libraries (BLAS)through standard libraries (BLAS) Some get into the technology for a further optimisation stepSome get into the technology for a further optimisation step

FPGAFPGA Still perceived as “difficult” and expensiveStill perceived as “difficult” and expensive Groups looking into ways to simplify the developmentGroups looking into ways to simplify the development

CPUCPU They have always been there, now increasing scopeThey have always been there, now increasing scope MACAO and SPARTA Light for small/mid size systemsMACAO and SPARTA Light for small/mid size systems DARC/KAOS for small/mid size DARC/KAOS for small/mid size FORCE prototype for ELT entry level (GLAO)FORCE prototype for ELT entry level (GLAO)




The real issueThe real issue

Development costsDevelopment costs





RTC Workshop: AlgorithmsRTC Workshop: Algorithms


Algorithm System Speedup

Cure (CuReD, Cure w/ preprocessing)Misalignment estimationTested in the real world

SCAO/XAO 100-1000

CLIF for Pyramid XAO 200

Kaczmarz MCAO/LTAO/MOAO

10-200

Wavelets with PCG MCAO 5-100

Frim (PCG)Heavily used to accelerate simulations

SCAO/LTAO/MCAO

30-50

SABRE XAO 10-20




Smart algorithms performanceSmart algorithms performance


CuReD performance

See talkSee talkShatokinaShatokina

FridayFriday






Kazcmarz performance

See talkSee talkRamlauRamlautodaytoday






SABRE overview




Smart arrangementSmart arrangement Split an on-line partSplit an on-line part

and an off-line partand an off-line part Applicable to any iterative algorithmApplicable to any iterative algorithm

Frim accelerationFrim acceleration


P

i

Q

jji jnyainxbnb

nba

nxba

ny

1 1

00

0

][][][

][1

][1

][

On-lineOn-line

This is how 80µs latency is achievedThis is how 80µs latency is achieved

Off-lineOff-line

SPARTA does it on the IIR controller:SPARTA does it on the IIR controller:

P

i

Q

jji jnyainxb

any

0 10

][][1

][

See posterSee posterBechetBechet




The latency (and jitter) issueThe latency (and jitter) issue

How crucial is the latency and the jitter?How crucial is the latency and the jitter?

Specifications on latency and jitter must be carefully checked Specifications on latency and jitter must be carefully checked against Top Level Requirements to avoid over-specifying against Top Level Requirements to avoid over-specifying the real time controllerthe real time controller





Trends on algorithmsTrends on algorithms

Smart algorithms are readySmart algorithms are ready Some tested on skySome tested on sky Need to characterize them, mapping to different platformsNeed to characterize them, mapping to different platforms

Biggest question: do we need them?Biggest question: do we need them? Brute-force MVM on optimised hardware can be used to Brute-force MVM on optimised hardware can be used to

implement almost all the foreseen ELT instruments but one implement almost all the foreseen ELT instruments but one Array of GPUs or FPGAsArray of GPUs or FPGAs

Still need them to compute the CMStill need them to compute the CM Anyway, would you throw away a factor 1000 speed-up?Anyway, would you throw away a factor 1000 speed-up?

Can make room for more advanced control schemesCan make room for more advanced control schemes

It is now a matter of a design decisionIt is now a matter of a design decisionFirenze, 26-31 May 2013; AO4ELT3Firenze, 26-31 May 2013; AO4ELT3




Vibration rejectionVibration rejection

A trend (and hot topic) on his ownA trend (and hot topic) on his own

Several groups at work with different solutionsSeveral groups at work with different solutions

Two main categories: LQG-based or RLS-basedTwo main categories: LQG-based or RLS-based


See talkSee talkSivoSivo

FridayFriday




ELT: the Instrument RTCELT: the Instrument RTC

ESO is developing a development planESO is developing a development plan


RequirementsRequirements

RequirementsRequirementsAnalysisAnalysis

TechnologicalTechnologicalsurveysurvey

CommunityCommunitysurveysurvey

LESIA





Major drivers:Major drivers: Compatibility with ELT established standards wherever possibleCompatibility with ELT established standards wherever possible Obsolescence management, upgradability, maintenanceObsolescence management, upgradability, maintenance Scalability both in performance (small to big systems) and in cost Scalability both in performance (small to big systems) and in cost

(laboratory to instrument systems)(laboratory to instrument systems) Structure of development, development phases, industrializationStructure of development, development phases, industrialization Define need for a common development (a platform) and at which levelDefine need for a common development (a platform) and at which level Flexibility to accommodate varying requirements/algorithms during the Flexibility to accommodate varying requirements/algorithms during the

development and AIT phases (maybe with degraded performance)development and AIT phases (maybe with degraded performance) Strong decoupling between the I/O and computing modes technology Strong decoupling between the I/O and computing modes technology

choices, allowing separate upgrade paths/roadmaps.choices, allowing separate upgrade paths/roadmaps. High SW component reusability through loosely-coupled development High SW component reusability through loosely-coupled development

techniques and standard libraries.techniques and standard libraries.






Technological surveyTechnological survey Operating systemOperating system

VxWorks, LinuxVxWorks, Linux

Parallel programming and architecturesParallel programming and architectures Cilk, OpenMP, OpenCL, NUMA, SSECilk, OpenMP, OpenCL, NUMA, SSE

Interconnects (PCIe, GbE)Interconnects (PCIe, GbE) CPU-based implementationsCPU-based implementations AcceleratorsAccelerators

GPU for soft and hard real time, GPU direct, multi-GPU systemsGPU for soft and hard real time, GPU direct, multi-GPU systems Role of PhiRole of Phi

FPGA as FPGA as protocol offload engineprotocol offload engine Computing engineComputing engine





10-40-100 GbE Interconnect10-40-100 GbE Interconnect

Raw UDPRaw UDPWith FPGA-to-FPGA: <1µs latencyWith FPGA-to-FPGA: <1µs latencyFull bandwidth (10Gb) reachedFull bandwidth (10Gb) reached

Optimised switching with multicastOptimised switching with multicast<2.5µs latency, switch only<2.5µs latency, switch onlyFull bandwidth on all ports (48) reachedFull bandwidth on all ports (48) reached

March 29th, ESO GarchingMarch 29th, ESO Garching




Concept for successor of SPARTAConcept for successor of SPARTA

WPUWPU

sw

itc

hs

wit

ch

RECREC

sw

itc

hs

wit

ch

CTRCTR

sw

itc

hs

wit

ch

CODECODEDETDET

ClusterCluster

• Communication based on 10-40-100 GbE• Distribution based on UDP or RTPS • Directly managed by FPGAs where latency/jitter is important• Metrology derived from the switch• Switch can deliver low latency (proved by Cisco)





My own experienceMy own experience

Portability of MACAO Portability of MACAO codecode

CPUs catch up: 2007: CPUs catch up: 2007: NAOS UpgradeNAOS Upgrade

Obsolescence of Obsolescence of SPARTASPARTA

Modularity to fight Modularity to fight obsolescenceobsolescence

Real time performance Real time performance vs feasibilityvs feasibility

The rest of the The rest of the development is the development is the biggest partbiggest part

Importance of shared Importance of shared developmentdevelopment

Lack of closed loop Lack of closed loop testing toolstesting tools


MACAO VLTIMACAO VLTI

CRIRESCRIRES

SINFONISINFONI

MADMAD

SPHERESPHERE

GRAALGRAAL

GALACSIGALACSI

ERISERIS

NAOMINAOMI

GRAVITYGRAVITY

NAOSNAOS

44

11

1 - LGS1 - LGS

LGSLGS

MA

CA

OM

AC

AO

11

11

11

11

44

44

SP

AR

TA

SP

AR

TA

2003

2006

2006

2013

2015

2015

2004/2006

2007

1.5M-MAC1.5M-MAC

1.5M-MAC1.5M-MAC

1.5M-MAC1.5M-MAC

27M-MAC27M-MAC

15M-MAC15M-MAC

4.1G-MAC4.1G-MAC

12G-MAC12G-MAC

12G-MAC12G-MAC

7.5M-MAC7.5M-MAC

5 M-MAC5 M-MAC

4.1G-MAC4.1G-MAC




The importance of being a PlatformThe importance of being a Platform





ConclusionsConclusions Heterogeneous computing, with GPUs playing a very important roleHeterogeneous computing, with GPUs playing a very important role MVM coming backMVM coming back Deterministic transport settling on GbEDeterministic transport settling on GbE Use of embedded systems more and more limitedUse of embedded systems more and more limited Emergence and importance of optical bench simulatorsEmergence and importance of optical bench simulators Need to find “space” for more complex control schemesNeed to find “space” for more complex control schemes

Anti wind-up, saturation management, vibration rejection, modal control. LQGAnti wind-up, saturation management, vibration rejection, modal control. LQG They add complexityThey add complexity

Use of commodity hardware Use of commodity hardware upgradeabilityupgradeability Maintainability of commodity hardware imposes continuous upgradesMaintainability of commodity hardware imposes continuous upgrades

Importance of software development costsImportance of software development costs Minimizing it key to success Minimizing it key to success shared developments, collaborations shared developments, collaborations

Portability and modularity recognised but need more developmentPortability and modularity recognised but need more development Need to harness computing power of the different technologies in a Need to harness computing power of the different technologies in a

portable/maintainable way portable/maintainable way template programming or metaprogramming template programming or metaprogramming

Total Cost of Ownership rarely addressedTotal Cost of Ownership rarely addressedFirenze, 26-31 May 2013; AO4ELT3Firenze, 26-31 May 2013; AO4ELT3

Trends on real time control for adaptive optics

Documents

Transcript of Trends on real time control for adaptive optics