!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0&

Physiology-based interaction for health, well-being and entertainment!!

Dr. Aleksander Väljamäe Senior research fellow,

Human Computer Interaction group, School of Digital Technologies,

Tallinn University

Venia Legendi lecture

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8&

"Breaking the Waves" by Lars Von Trier (1996)

ART AS A MACHINE THAT ALLOWS US TO AQUIRE NEW LIFE EXPERIENCES

Inspired by works of Lotman, Vygotsky and Mamardashvili

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =&

Born in Tallinn, Estonia, in 1978 1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn TU, Tallinn, Estonia 2001-2003 Second degree (M.Sc.) – Telecommunication, Chalmers University of Technology, Gothenburg, Sweden 2003-2007 Ph.D. on multisensory virtual reality – Chalmers University of Technology, Gothenburg, Sweden 2007-2010 First postdoc – neuroscience lab SPECS, University of Pompeu Fabra, Barcelona, Spain 2011-2012 Second postdoc – Graz BCI lab, TU Graz & Neuropsychology Laboratory, University of Graz, Graz, Austria 2012-2015 FP7 Marie Curie Action – International Outgoing Fellowship (MCA-IOF), Project “B-Reactable”, LiU & SPbSU 2015 - Senior member of Interaction Design Lab, Tallinn University

Short academic CV

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :&

Searching for a low-cost motion simulator

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& >&http://www.elumens.com/

Using perceptual optimization & multisensory design in VR

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 1&

User-centred methodology: POEMS

Subjective measures (self-reported presence & illusory self-motion)

auditory

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& ?&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& @&

Born in Tallinn, Estonia, in 1978 1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn TU, Tallinn, Estonia 2001-2003 Second degree (M.Sc.) – Telecommunication, Chalmers University of Technology, Gothenburg, Sweden 2003-2007 Ph.D. on multisensory virtual reality – Chalmers University of Technology, Gothenburg, Sweden 2007-2010 First postdoc – neuroscience lab SPECS, University of Pompeu Fabra, Barcelona, Spain 2011-2012 Second postdoc – Graz BCI lab, TU Graz & Neuropsychology Laboratory, University of Graz, Graz, Austria 2012-2015 FP7 Marie Curie Action – International Outgoing Fellowship (MCA-IOF), Project “B-Reactable”, LiU & SPbSU 2015 - Senior member of Interaction Design Lab, Tallinn University

Short academic CV

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 09&

SPECS-UPF research lines

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 00&

Establishing and running psychophysiology lab

Supervision of 10 MSc theses in the period of 2009-2010

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 08&

&EDA-based biofeedback via real-time video effects

MSc. thesis by S. Wierenga (2010)

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0<&

XIM – eXperience Induction Machine

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0=&

User experience and social interaction - Bernardet, U., Väljamäe, A., et al (2011). Brain Research Bulletin

Social collaboration and competition - Inderbitzin, M., Wierenga, S., Väljamäe, A., et al. (2009). Virtual Reality

Explicit and implicit interaction - Bermúdez i Badia, S., Väljamäe, A., et al. (2009). Presence: Teleoperators and Virtual Environments

Design and analysis of the human experiments in XIM

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0:&

& Studying the affective conditioning

Väljamäe, A., Pesquita, A., & Verschure, P.F.M.J. (2010) Studying cross-modal emotional conditioning using distance montage in database narratives. Talk at the Conference of Society for Cognitive Studies of the Moving Image (SCSMI’10). Roanoke, Virginia, USA

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0>&

& Experimental design

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 01&

& Electrodermal activity – SCR data N

orm

aliz

ed u

nits

Duration [s]

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0?&

Pilot clinical studies at SPECS Steffert, T., Väljamäe, A., Fernandez-Moralez, J., Kropotov, J., Verschure,

P.F.M.J., & Pozo-Rosich, P. (2010). Slowing of sensory event related potentials (ERPs) in migraine: Search for an endophenotype. Poster presentation at 16th Annual Meeting of the Organization for Human Brain Mapping, Barcelona.

Väljamäe, A., Steffert, T., Bernardet, U., Gruzelier, J., & Verschure, P.F.M.J. (2009). Studying the effectiveness of immersion for neurofeedback in mixed/virtual reality. Poster presentation at 14th Annual CyberTherapy (CT14) & CyberPsychology Conference, Lago Maggiore, Italy.

Väljamäe, A., Pozo-Rosich, P. & Verschure, P.F.M.J. (2009). Studying mixed feelings in migraineurs using affective picture pairs. Poster presentation at the International Society for Research on Emotion (ISRE) 2009 Conference, Leuven, Belgium.

& Neurofeedback experiments in XIM: self-motion in VR as a reward

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 0@&

Pilot clinical studies at SPECS Steffert, T., Väljamäe, A., Fernandez-Moralez, J., Kropotov, J., Verschure,

P.F.M.J., & Pozo-Rosich, P. (2010). Slowing of sensory event related potentials (ERPs) in migraine: Search for an endophenotype. Poster presentation at 16th Annual Meeting of the Organization for Human Brain Mapping, Barcelona.

Väljamäe, A., Steffert, T., Bernardet, U., Gruzelier, J., & Verschure, P.F.M.J. (2009). Studying the effectiveness of immersion for neurofeedback in mixed/virtual reality. Poster presentation at 14th Annual CyberTherapy (CT14) & CyberPsychology Conference, Lago Maggiore, Italy.

Väljamäe, A., Pozo-Rosich, P. & Verschure, P.F.M.J. (2009). Studying mixed feelings in migraineurs using affective picture pairs. Poster presentation at the International Society for Research on Emotion (ISRE) 2009 Conference, Leuven, Belgium.

Exploratory clinical studies at SPECS: autism, migraine, dementia, stroke, chronic pain

Väljamäe, A., Steffert, T., Bernardet, U., Gruzelier, J., & Verschure, P.F.M.J. (2009). Studying the effectiveness of immersion for neurofeedback in mixed/virtual reality. Poster presentation at 14th Annual CyberTherapy (CT14) & CyberPsychology Conference, Lago Maggiore, Italy.

Steffert, T., Väljamäe, A., Fernandez-Moralez, J., Kropotov, J., Verschure, P.F.M.J., & Pozo-Rosich, P. (2010). Slowing of sensory event related potentials (ERPs) in migraine: Search for an endophenotype. Poster presentation at 16th Annual Meeting of the Organization for Human Brain Mapping, Barcelona.

Väljamäe, A., Pozo-Rosich, P. & Verschure, P.F.M.J. (2009). Studying mixed feelings in migraineurs using affective picture pairs. Poster presentation at the International Society for Research on Emotion (ISRE) 2009 Conference, Leuven, Belgium.

Le Groux, S., Väljamäe, A., & Verschure, P.F.M.J. (2009). Towards Interactive Sonic Therapy. Poster presentation at 14th Annual CyberTherapy (CT14) & CyberPsychology Conference, Lago Maggiore, Italy.

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 89&

Multimodal Brain Orchestra performance at FET conference in 2009

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 80&

Multimodal Brain Orchestra technical implementation

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 88&

Born in Tallinn, Estonia, in 1978 1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn TU, Tallinn, Estonia 2001-2003 Second degree (M.Sc.) – Telecommunication, Chalmers University of Technology, Gothenburg, Sweden 2003-2007 Ph.D. on multisensory virtual reality – Chalmers University of Technology, Gothenburg, Sweden 2007-2010 First postdoc – neuroscience lab SPECS, University of Pompeu Fabra, Barcelona, Spain 2011-2012 Second postdoc – Graz BCI lab, TU Graz & Neuropsychology Laboratory, University of Graz, Graz, Austria 2012-2015 FP7 Marie Curie Action – International Outgoing Fellowship (MCA-IOF), Project “B-Reactable”, LiU & SPbSU 2015 - Senior member of Interaction Design Lab, Tallinn University

Short academic CV

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8<&

Brain-Computer Neural Interaction (BNCI)

Aleksander Väljamäe and Brendan Z. Allison

Laboratory for Brain-Computer Interfaces, Institute for Knowledge Discovery,

Technical University of Graz, Graz, Austria

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8=&

Future BNCI roadmap and videography (http://www.youtube.com/user/FutureBNCI)

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8:&

Embodied cognition and body awareness

Tajadura-Jiménez, A., Väljamäe, A., et al. (2012)., Action sounds recalibrate perceived tactile distance, Current Biology

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8>&

Born in Tallinn, Estonia, in 1978 1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn TU, Tallinn, Estonia 2001-2003 Second degree (M.Sc.) – Telecommunication, Chalmers University of Technology, Gothenburg, Sweden 2003-2007 Ph.D. on multisensory virtual reality – Chalmers University of Technology, Gothenburg, Sweden 2007-2010 First postdoc – neuroscience lab SPECS, University of Pompeu Fabra, Barcelona, Spain 2011-2012 Second postdoc – Graz BCI lab, TU Graz & Neuropsychology Laboratory, University of Graz, Graz, Austria 2012-2015 FP7 Marie Curie Action – International Outgoing Fellowship (MCA-IOF), Project “B-Reactable”, LiU & SPbSU 2015 - Senior member of Interaction Design Lab, Tallinn University

Short academic CV

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 81&

B-Reactable project: 2013-2015

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8?&

Collaborations: - Dr. Aleksei Ossadchi, Saint Petersburg State University - Prof. Juri D. Kropotov, Natalia Shemyakina, Institute of Human Brain, Russian Academy of Sciences - Prof. Daniel Västfjäll, Linköping University, Linköping, Sweden - Prof. Sergi Jorda, Sebastian Mealla, University of Pompeu Fabra, Barcelona, Spain

“B-Reactable: multimodal tabletop system for collaborative physiology monitoring and training" https://vimeo.com/14675468

B-Reactable project: 2013-2015

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& 8@&

&First B-Reactable experiments

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <9&

&Multi-brain setup at Saint-Petersburg

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <0&

&B-Reactable: signal processing block diagram

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <8&

!Physiology-based interaction strategies global control vs. local control

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <<&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <=&

Journal publications of B-Reactable

Mealla, S., Jorda, S., & Väljamäe, A. Physiopucks: increasing user motivation by combining tangible and implicit physiological interaction (in press). ACM Transactions on Computer-Human Interactions (TOCHI).

Steffert, T., Holland, S., Mulholland, P., Sheep, D., & Väljamäe, A. (in review) The Assessment Of Real-Time EEG Sonifications For Neurofeedback

Mealla Cincuegrania, S., Jorda, S., & Väljamäe, A. (in review) Sonic designs for Neurofeedback: Implementation, Personalization and Evaluation of Sonic Designs for Neurofeedback.

Mealla Cincuegrania, S., Jorda, S., & Väljamäe, A. (in preparation) Enhancing Musical Expression Through Physiology-based Sonic Interaction: A Validation of Control Mappings in a Performance Environment

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <:&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <>&

from Wolpaw et al. (2002)

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <1&

from Wolpaw et al. (2002)

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <?&

&Implicit, physiology-based interaction Physiological computing provides a new channel to the

conventional Human Computer Interaction paradigm My current area of interest is implicit interaction with

multisensory media and its use for health and entertainment

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& <@&

&Tools: affective chronometry and database cinema

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =9&

Controlled stimuli – affective databases (IAPS)

HAU

LAU

HAP

LAP

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =0&

&Exploration of emotional rhythms

MSc. thesis by M. Oliveira (2010)

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =8&

Scientific framework: Craig's awareness model

Craig, Nature, 2009

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =<&

Craig's awareness model

Craig, Nature, 2009 (and references therein)

Hypothesis: Anterior Insular Cortex (AIC) engenders human awareness

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& ==&

Integration of salient activity in the insula

Craig, p67, Nature, 2009

Producing " a ‘global emotional moment’, which represents the sentient self at one moment of time"

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =:&

Temporal changes of the sentient self ‘image’

Craig, Nature, 2009

a series of global emotional moments produce a self "image" that evolves in time...

...and modulates subjective time passage

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =>&

Other view on emotional moments: six phases of auteur film

van der Pol, 2011

1st phase: seeing story world as a real world 2nd phase: understanding the film as fiction 3d phase: understanding the characters as real people 4th phase: Confrontation with a cognitive dissonance 5th phase: True moral crisis due to character "immorality" 6th phase: Redemption

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =1&

Physiology-based interaction research areas

- temporal dynamics of cognitive & emotional processing - unconscious processing - aesthetics of physiology-based interaction - neuro- and biofeedback - brain to brain interaction, hyperscanning - decision making and prosocial behavior (compassion) - emotional synchronization, collective experience - imaging genetics + media = cinegenetics

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =?&

Physiology-based interaction research areas

- temporal dynamics of cognitive & emotional processing - unconscious processing - aesthetics of physiology-based interaction - neuro- and biofeedback - brain to brain interaction, hyperscanning - decision making and prosocial behavior (compassion) - emotional synchronization, collective experience - imaging genetics + media = cinegenetics

WHAT ARE THE NEXT STEPS TO MAKE IT HAPPEN?

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& =@&

Action 1: User eXperience laboratory within HCI group

UX lab will be used to assess emotional, cognitive and perceptual reactions to mixed and real environments using psychophysiological, behavioural and subjective responses from users.

Example projects: •! Aesthetics of implicit interaction [methodology

development] •!Hyperscanning studies on group decision making

[group evaluation] •!Neuro- and biofeedback applications [prosocial

behavior training]

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :9&

Experiments at the CineLab, Mallorca 2015

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :0&

Action 2: Dedicated courses within MSc programmes

Project based courses within HCI or Digital Learning Games programmes where students will work together using physiology-based interaction for designing new applications for health, well-being and entertainment

Example projects: •! Phenomenon of catharsis defined in physiological

terms •!Neuro and bio-feedback training in actors and

“Stanislavsky 2.0” •! Physiology-based interaction patterns vocabulary,

minimal experience units or "cognemes"

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :8&

Advanced interaction design course at UPF, 2010

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :<&

Action 3: Research and fundraising

I have been active in research fundraising from 2004 with several projects successfully funded:

•! FP6 Presenccia (7.4 Mln, 2006-2009) •! FP7 BrainAble (3 Mln, 2010-2012) •! FP7 CEEDS (8.6 Mln, 2010-2015) •! H2020 BrainHack (0.5 Mln, 2016-2018)

Example projects proposal ideas: •!Neurotheatre and neurocinema •!Wearable and affective computing for health •!Affective metadata as a intangible cultural heritage

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :=&

Brain-to-brain communication based on media

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& ::&

Brain to Brain communication

A".B4#%*&%C"+D5"&)$+D4#%.E&

A".B4#%*&%C"+D5"&)$+D4#%.E&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :>&

Brain to Brain communication

A".B4#%*&%C"+D5"&)$+D4#%.E&

A".B4#%*&%C"+D5"&)$+D4#%.E&

!"#$%&'"("#)$&*"+,-."/&01,2&345"67".&890:/&;%**$##& :1&

Thank you for your attention!

aleksander.valjamae@tlu.ee