Model-based Measurement of Interaction in MobileMultimodal Environments

Pedro Mateo – Stefan Hillmann

aMMI - October 2012

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 1 / 17

Agenda

Model-based Measurement of Interaction in Mobile MultimodalEnvironments:

1 Research Questions

motivation, questions and proposed solutions

2 Research and Development

MMI metrics, MMI model and Android implementation

3 Conclusion

advantages, disadvantages and research directions

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 2 / 17

IResearch Questions

Motivation

We analyze interaction in differentsystems...

...but independently for each system

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 4 / 17

Motivation

And now, for each system...

different quantification/evaluation metrics

different representation formats

different interaction analysis tools

multimodal communication is not analyzed as a whole

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 5 / 17

Research questions and proposed solutions

Question 1:

MMI assessed on a common level for different modalities?⇒ Use common and generic metrics for all modalities

Question 2:

Compare interactions observed in different MMI contexts?⇒ Structure metrics in a common representation (model)

Question 3:

Analyze MMI dinamically, step-by-step, and using real-time data?⇒ Use a run-time model

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 6 / 17

IIResearch and Development

Defining multimodal metrics

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 8 / 17

Modeling multimodal interaction in EMF

Turn DialogModality

Task

User Turn System Turn

User Metacommunication System Metacommunication

Metacommunication

Speech Input

GUI Input

Gesture Input

Speech Output

GUI Output

Gesture Output

Cooperativity

Synchrony Modality Change

Appropriateness

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 9 / 17

Modeling multimodal interaction in EMF

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 10 / 17

Android implementation

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 11 / 17

Android implementation

how does it work?

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 12 / 17

IIIConclusion

Good stuff...

common metrics + uniform representation⇒ unified criteria to quantify MMI in different scenarios

comparison is allowed, model transformation is possible

different modalities are on a level ⇒ single thread

dynamic representation of MMI ⇒ run-time applications

reuse of metrics and tools is improved

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 14 / 17

Not so good stuff...

some data can’t be automatically collected

criteria to record parameters in different modalities?

lack of plasticity⇒ user and context information to enrich the analysis process

the HOW of interaction is not described

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 15 / 17

And now what?

Integrate this solution in real projects

Use the framework for real analysis

Make the model context-aware, more “mobile”

Explore new possibilities of the model

E.g. behavioral biometric authentication

Project info: http://www.catedrasaes.org/wiki/MIM

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 16 / 17

Acknowledgements

Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 17 / 17