Signal Processing in EU R&D Projects

Current activities and future perspectives

EUSIPCO 2000

September 6, 2000Tampere, Finland

Eric BadiquéEric.Badique@cec.eu.int

OutlineOutline

Why Europe ?Why Europe ? The IST ProgrammeThe IST Programme Signal processing and related issues in ISTSignal processing and related issues in IST Trends and opportunitiesTrends and opportunities Future calls for proposalFuture calls for proposal RecommendationsRecommendations

The European Commission’s 5The European Commission’s 5thth Framework ProgrammeFramework Programme

Selection criteria European “value added” and the

subsidiarity principle (critical mass, problem of European dimension...)

Social objectives (employment, environment…)

Economic development and scientific and technological prospects

Seven Specific Programmes

Quality of life and management of living resources

User-friendly information society

Competitive and sustainable growth

Energy, environment and sustainable development.

International aspects of Community research

Promotion of innovation and encouragement of SMEs

Improving human research potential

Types of actions supportedTypes of actions supported

Shared-cost actions R&D projects Demonstration projects Combined R&D and demonstration projects “SME Co-operative” research projects “SME Exploratory” awards

Training fellowshipsThematic networks, concerted actions etc..

Collaborative R&D modelCollaborative R&D model

Workplanconsultation

(yearly) Call for proposal Evaluation

by independentexperts

Contract

Yearly auditby independent

experts

Dissemination of results

Exploiting synergy

Cluster

Programme

World

Collaborative, pre-competitive R&D by multi-national consortia building European teams

Working in a wide « infocom » environment contribute to formulation and implementation of European policies

contributing to global standardisation

Project

Complementary and synergetic clusters of projects exploring options, optimising coverage

Research networkingResearch networkingResearch networkingResearch networking

Future & emerging technologiesFuture & emerging technologies

Systems & ServicesSystems & Services for the citizen for the citizen

New methods of workingNew methods of working & Electronic Commerce & Electronic Commerce

MultimediaMultimediacontent & toolscontent & tools

Essential technologiesEssential technologies& infrastructure& infrastructure

Information Society Information Society Technologies ProgramTechnologies Program

The Information Society Technologies The Information Society Technologies (IST) Programme(IST) Programme

The IST vision: Our surrounding is the interface

Build on Europe’s strengths (mobile and fixed coms, consumer electronics, digital broadcasting, content...)

Start creating the ambient intelligence landscape for seamless delivery of services and applications in Europe

- Ambient intelligence landscape - Priorities related to Signal Processing

Embedded intelligenceEmbedded intelligence Multi-lingual and Multi-lingual and

multi-modal interfacesmulti-modal interfaces Personalised servicesPersonalised services Cross-media contentCross-media content Trust, confidence and Trust, confidence and

securitysecurity

Distributed Distributed applicationsapplications

Mobile and wireless Mobile and wireless internetinternet

Converging networksConverging networks Open source softwareOpen source software

Signal Processing in IST -Signal Processing in IST -OverviewOverview

Signal Processing is everywhere !Signal Processing is everywhere !

70+ projects have an SP component70+ projects have an SP component

IST contributes significantly to SP fundingIST contributes significantly to SP funding

Participants: Academia, consumer electronics, Participants: Academia, consumer electronics, broadcasters, network operators, SMEs... broadcasters, network operators, SMEs...

Signal Processing in IST -Signal Processing in IST -Overview Overview

Cryptography

Watermarking

Authoring

Restoration

Health Environment

Communication

Biometry

Transport

Representation &Coding

Processors

Multimodalinterfaces

Telepresence

Computer vision

Computer vision for 3D modelling

3D acquisition cluster in ACTS3D acquisition cluster in ACTS Laser Range Finder Laser Range Finder (Resolv)(Resolv)

Uncalibrated Uncalibrated (corner and line matching, trifocal tensor) (corner and line matching, trifocal tensor) (Vanguard)(Vanguard)

Calibrated trinocular images Calibrated trinocular images (Panorama)(Panorama)

New projects in IST New projects in IST (Hiscore, Visire)(Hiscore, Visire)

Extraction of 3D geometry and textures from video streamsExtraction of 3D geometry and textures from video streams Image-based renderingImage-based rendering

Applications to virtual museums, archeology, architecture, Applications to virtual museums, archeology, architecture, eCommerce...eCommerce...

AuthoringAuthoring

Content to transform TV watching into an immersive Content to transform TV watching into an immersive experience experience (eg. focus on sport, Piste):(eg. focus on sport, Piste): 2nd Gen iTV 2nd Gen iTV

Seamless blending of real and virtual objectsSeamless blending of real and virtual objects Augmented views of events, advanced interactionAugmented views of events, advanced interaction Large market is Large market is virtual advertisingvirtual advertising on TV and on Web on TV and on Web

(Enrevi) (Enrevi)

Requires segmentation, analysis, tracking, fusion, animationStandards (MPEG-4 over MPEG-2)

Agent-Based Hybrid Scene Creation and Management

Authoring tool for Authoring tool for telepresence telepresence

and augmented reality and augmented reality (Art.Live)(Art.Live)

Real-time model-basedReal-time model-based

and blind segmentationand blind segmentation

Multi-agent architectureMulti-agent architecture Mixed 2D-3D scene Mixed 2D-3D scene

reconstructionreconstruction Dynamic databaseDynamic database

Sensors/ Sensors/Effectors Computer Enviornment Effectors

Augmented Simulation an Analysis of reality Virtual WorldsReality TelepresenceSite Reality

Rea

l-lif

e E

nvio

nmen

t Individual Enviornm

ent

IndividualReality

IndividualEnviornment

Image processing and NO characterisation

SO characterisationImageprocessing

Video capture

Screen SO/NOVisualisation

Monitor NO/SOVisualisation

Dreamed SOCreation

NO

NO

NO

SO

SO

SO

Signal Processing in IST -Signal Processing in IST -Overview Overview

Cryptography

Watermarking

Authoring

Restoration

Health Environment

Communication

Biometry

Transport

Representation &Coding

Processors

Multimodalinterfaces

Telepresence

Computer vision

Representation and codingRepresentation and coding Indexing and retrieval

Strong contribution to MPEG standard (ACTS legacy)Strong contribution to MPEG standard (ACTS legacy) Initiatives to bridge the gap between coding and CG world Initiatives to bridge the gap between coding and CG world

(MPEG-4/VRML/Web3D)(MPEG-4/VRML/Web3D) JPEG2000 JPEG2000 Universal production system for TV and HDTVUniversal production system for TV and HDTV Indexing and retrieval Indexing and retrieval

Shot detection, feature extraction, segmentation, classification, Shot detection, feature extraction, segmentation, classification, indexing, semantic annotation indexing, semantic annotation (Divan, Avir, Cuidad, Vicar, Assavid…)(Divan, Avir, Cuidad, Vicar, Assavid…)

Use of dual audio and visual features to locate content, association Use of dual audio and visual features to locate content, association of text and audio fragments of text and audio fragments (Ciwos)(Ciwos)

Automatic recognition of audio titles based on fingerprints Automatic recognition of audio titles based on fingerprints (RAA)(RAA)

Communication - Software radio

Multicarrier access system Multicarrier access system (Adamas)(Adamas)

Terrestrial and satellite UMTS Terrestrial and satellite UMTS (Virtuous)(Virtuous)

Reconfigurable terminals Reconfigurable terminals (Pastoral)(Pastoral)

Smart antennas Smart antennas (Saturn)(Saturn)

Wireless indoor high bitrate modems Wireless indoor high bitrate modems (Wind-flex)(Wind-flex)

Partners: Philips, VTT, Intracom, Thomson, FT, ST, Alcatel, Partners: Philips, VTT, Intracom, Thomson, FT, ST, Alcatel, CSELT, Siemens, Telefonica, ASCOM, Lucent...CSELT, Siemens, Telefonica, ASCOM, Lucent...

MultimodalMultimodal Interfaces Interfaces

Speech recognitionSpeech recognition Multi-linguality, multi-acoustic environments Multi-linguality, multi-acoustic environments ((Speecon, CoretexSpeecon, Coretex))

Unsupervised learning techniques Unsupervised learning techniques Applications to consumer electronics and distributed environments Applications to consumer electronics and distributed environments

((SmadaSmada) )

Avatars and virtual humansAvatars and virtual humans Face detection, registration, facial expression classification, Face detection, registration, facial expression classification,

animation animation (Interface, Mesh, Visicast, Vpark..)(Interface, Mesh, Visicast, Vpark..)

Eye-tracking Eye-tracking (I-eye)(I-eye)

Gaze as active element in user interfaceGaze as active element in user interface Applications: assisted translation, hand-free assisted tutoringApplications: assisted translation, hand-free assisted tutoring

Haptics, telerobotics Haptics, telerobotics (Teledimos)(Teledimos)

Signal Processing in IST -Signal Processing in IST -Overview Overview

Cryptography

Watermarking

Authoring

Restoration

Health Environment

Communication

Biometry

Transport

Representation &Coding

Processors

Multimodalinterfaces

Telepresence

Computer vision

Biometrics & CryptographyBiometrics & Cryptography

Multimodal verification Multimodal verification (M2VTS)(M2VTS)

facial feature extraction and tracking, 3D analysis, face facial feature extraction and tracking, 3D analysis, face recognition, speech recognition, fusion speaker verificationrecognition, speech recognition, fusion speaker verification

Applications are access control (e-commerce, e-banking), Applications are access control (e-commerce, e-banking), surveillance, alarm verification surveillance, alarm verification (Banca)(Banca)

Signature, integrity and encryption Signature, integrity and encryption (Nessie)(Nessie)

Primitives for Gigabit networking, wireless comms, PDAs Primitives for Gigabit networking, wireless comms, PDAs and smart cardsand smart cards

Contribution to Advanced Encryption standard (AES)Contribution to Advanced Encryption standard (AES)

EnvironmentEnvironment

Water quality monitoring Water quality monitoring (Bluewater, Rapsodi)(Bluewater, Rapsodi)

Multi-cameras, image analysis of wave motion dampingMulti-cameras, image analysis of wave motion damping Integration of aircraft and satellite sensorsIntegration of aircraft and satellite sensors Integration of radar signal and image-processingIntegration of radar signal and image-processing Detection, estimation and trackingDetection, estimation and tracking

Robot-based measurement of volcanic environment Robot-based measurement of volcanic environment (Robovolcist)(Robovolcist)

Automatic, visual inspection for structural risk assessment Automatic, visual inspection for structural risk assessment (Robosense)(Robosense)

Flood analysis Flood analysis (Anfas)(Anfas)

Remote sensing imagery, computer vision, propagation simulationRemote sensing imagery, computer vision, propagation simulation

Healthcare

Confocal imaging for early diagnosis of skin cancer (Ediscim)

Pattern recognition of tumours using magnetic resonance (Interpret)

Reconstruction and visualisation for dynamic computed tomography (Dynct)

Home-based care and medical treatment (Telemedicare)

24 hour real-time medical monitoring at home, body-sensors for ECG, blood pressure, oxymetry and temperature

Segmentation and tracking for echocardiography (Adequate)

MRI-based minimally invasive therapy (Amit)

instrument tracking, visualisation

Rehearsal and planning of surgical interventions (Ierapsi, VR-SUR)

TransportTransport

Advanced Driver Assistance System Advanced Driver Assistance System (Carsense)(Carsense)

Stereo vision, image processing for multiple targetsStereo vision, image processing for multiple targets

Sensorial and communication systems to improve safety of Sensorial and communication systems to improve safety of vulnerable road users vulnerable road users (Protector)(Protector)

computer vision + cooperative means carried by road usercomputer vision + cooperative means carried by road user

Near impending crash detection Near impending crash detection (Chameleon)(Chameleon)

Sensor fusion and standardisationSensor fusion and standardisation

Automotive radar system Automotive radar system (RadarNet)(RadarNet)

Multi-agent architecture for road surveillance Multi-agent architecture for road surveillance (Modest) (Modest)

TelepresenceTelepresence(Virtue)(Virtue)

•Eye contact, •spatial awareness, •Look-around effect•dynamic view synthesis from multiple cameras •Semi-immersive display •life-size head and torso images•Same virtual table•Correct perspective

•Video segmentation & coding•3D audio•head-tracking•multi-view reconstruction•Fusion•Networking•Synchronisation•Human factors...

Trends, opportunitiesTrends, opportunitiesand issuesand issues

Signal processing as the great enabler ?Signal processing as the great enabler ? Web and mobile developments Web and mobile developments (flexibility, scalability…)(flexibility, scalability…) Processor architecture to cope with complexity ? Processor architecture to cope with complexity ? (beyond speed and (beyond speed and

memory) memory) Progress in sensors, displays and actuators Progress in sensors, displays and actuators (fusion)(fusion) Need for increased and embedded intelligence Need for increased and embedded intelligence (more than signals, higher (more than signals, higher

abstraction levels, it’s another world...abstraction levels, it’s another world...)) New representation standards as catalyst for SP ? New representation standards as catalyst for SP ? (we have got decoders; (we have got decoders;

what about the coders ?)what about the coders ?) Multi-disciplinarity ? Multi-disciplinarity ? (has been a major issue for years but..)(has been a major issue for years but..) Initiatives to structure some aspects of SP ?Initiatives to structure some aspects of SP ? (COST211 analysis model ? (COST211 analysis model ?

Evolving model ? Architecture ? Relation to human perception ?)Evolving model ? Architecture ? Relation to human perception ?) Mixed Realities Mixed Realities (Augmenting reality & virtuality, new imaging frontiers..)(Augmenting reality & virtuality, new imaging frontiers..)

On the edge ?On the edge ?

Meta-cameras Meta-cameras (beyond AV..)(beyond AV..) High-end SP for special effects: Back to reality High-end SP for special effects: Back to reality (Matrix..)(Matrix..) Embedded intelligence, smart devices Embedded intelligence, smart devices (MI2..)(MI2..) TranscodingTranscoding (the universal mail box) (the universal mail box) Personalised entertainment and work environments Personalised entertainment and work environments (Oz..)(Oz..) Super high resolution, real size, real color displays Super high resolution, real size, real color displays (Immersive (Immersive

TV, Natural Vision..)TV, Natural Vision..) Holodecks…..Holodecks…..

This is just the beginning

Current call for proposal

Most action lines relevant to SP are open

Deadline RTD: October 31, 2000

Call details www.cordis.lu/ist/calls/200004.htmwww.cordis.lu/ist/calls/200004.htm

Consultation process (WP2001)Consultation process (WP2001)

Consultation meeting “High Performance Signal Processing” was held Consultation meeting “High Performance Signal Processing” was held on May15on May15

15 experts (Imec, Thomson, ARM, Philips, UC Dublin, Infineon, KU 15 experts (Imec, Thomson, ARM, Philips, UC Dublin, Infineon, KU Leuven…)Leuven…)

Led to a Led to a proposalproposal for a cross-programme action (currently with ISTC) for a cross-programme action (currently with ISTC)

IST 2001 - CPA12: Advanced Signal Processing Systems and Applications

•Applications, algorithms, devices for high performance signal processing systems

•Biomedical processing, media, communications, security

•Advanced algorithms

•Implementation in embedded systems

•Standards, methodologies, support tools

If you want to submit a proposal,If you want to submit a proposal,don’t forget the following...don’t forget the following...

Objectives Partnership Eligible partners Management Contract type

Scope of CfP Presentation Completeness Ethical issues Competition

Information Society Technologies www.cordis.lu/ist

IST Unit E3: Overview of SP activities uploaded soon

www.cordis.lu/ist/ka4/vision/

More information on IST