The Unintended Use of Mobile Phone Features:Teaching Mobile Phones new tricks

Dr Eiman KanjoMobile Sensing and DataScience, Senior Lecturer

Computing Department

Nottingham Trent University

eiman.kanjo@ntu.ac.uk

@eimankanjo

https://www.linkedin.com/in/eimankanjo/

CodeMobile April 2017

Background• Senior Lecturer at Nottingham Trent University

• Have been Developing for smart phones since 2004 (started with Symbian)

• I was one of the first researchers to utilise mobile phones as research tools.

• Moved between various research organisations including Nottingham University, Cambridge University.

• Started teaching mobile Development 2010 (Android)

• My PhD was in Computer Games at Abertay University, Dundee (2005).

Overview

• Mobile phone development for academic research

• Built-in and external Mobile Phones features and capabilities

• Mobile Sensing Research

• Utilising Mobile Phone Notifications and Mobile to model behaviour

• Programming mobile Phones to sense mood

• Utilising the latest development in Beacons to enable context and physical computing

• (Data Annotation Tools) the need for hardware and software tools to enable data collection

• Next steps for Mobile Phones in research

• Mobile Development at NTU

Mobile Phones for Academic Research

Social Science

Psychology

Health

Mental Health

GIS and Spatial Analysis

Data Collection Tools

Animal and wildlife Tracking

Smart Farms

Disaster ManagementMarketing & Media Studies

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Retail and Marketing

Examples of mobile Phone Adoption for Research

Studies that have used mobile phones for mood disorder research (Torous, John et al ,2015)

1065 studies. Ten studies on major depressive disorder and 4 on bipolar disorder were

included. Nine out of 10 studies on depression related smartphone applications featured activedata collection and all 4 studies on bipolar disorder featured passive data collection.

Torous, John et al. Current research and trends in the use of smartphone applications for mood disorders Internet Interventions , Volume 2 , Issue 2 , 169 - 173

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Mobile Phone Sensing

We take advantage of Mobile Phones sensing Capabilities

Data Collection

Monitoring Measurement Intervention DisseminationVisualisation

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Mobile Sensing

Notifications

Audio

Camera

Motion

Location

Wireless Signals

Screen Activities

Wireless connectivity

Messages

Calls

Social Media

Web browsing

Messages

Web browsing

Light Sensor

Heart Rate

Sources of Information on Mobile Phones

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Apps

MobSens• Mobile phones can provide coverage where static

sensors are hard to deploy and maintain, and largenumbers of cell phones already exist around theworld, providing the physical sensinginfrastructure.

• Availability of more powerful operating systemsand the transfer of standardized programminglanguages on ever-smaller computing platformshave spurred the recent development of softwareapplications for mobile computers

• Such systems can benefit from localcommunities as the driving element for environ-mental sensing. This approach, sometimesreferred to as “citizen science,” uses mobilesensor technology to help individuals personallycollect, share, compare, and participate inresearch work.

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

http://pages.experts-exchange.com/processing-power-compared/

MobSensMobSens software components installed on the phones often performthe following operations: sensing, filtering, processing, and loggingsensor data; rendering screen displays, including graphs, maps, and userinterfaces; and uploading data streams to back-end servers in real time.

Usually, each sensor data entry is combined with various informationsuch as the last valid GPS location plus additional information such Time ,Date (or UTC date), time, the phone’s International Mobile EquipmentIdentity (IMEI), user name, journey ID..etc.

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Comparison of graphs showing the discrepancy between the two noise measurements generated by the N95 phone microphone NoiseSpy and the Norsonic meter

NoiseSpy: Turning Phone Microphone into Noise Sensor

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

NoiseSpy: Turning Phone Microphone into Noise Sensor

Screenshots of NoiseSPY(on Symbian)

Noise data collected by cyclists and overlaid on

Maps along with a time graph

Eiman Kanjo: NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping. MONET 15(4): 562-574 (2010)

Sound measurements a trial in Cambridge, the coloured traces represent the loudness in dBA

A cycling courier user collecting data using NoiseSpy system

Eiman Kanjo: NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping. MONET 15(4): 562-574 (2010)

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

PollutionSpy

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Eiman Kanjo, Jean Bacon, David Roberts, Peter Landshoff: MobSens: Making Smart Phones Smarter. IEEE Pervasive Computing 8(4): 50-57 (2009)

mBreath

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Fresh: Cell-ID based Mobile Forum for CommunityEnvironmental Awareness

• Fresh is a Mobile interface that utilizes GPRS networking based on cell-IDs from mobile phones. This mobile utility (forum) encourages users to interact at a specific geographical area to allow people to discuss issues related to their local environment. .

• (e.g.using GsmCellLocation)• Get CellID()

• GetLac() gsm location area code..

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

First Location based Forum based on CellIDs

ViralNet for Crowd Managemennt

• Utilizes forwarding Protocol which adapts to the messages sizes, types, frequencies and initiators.

• Connection to furthest node rather than the nearest.

• Based Mainly on Mobile phones communications over Bluetooth

• Extends the Bluetooth range to hundreds of meters depending on the crowed size.

Mobile Affective Sensing System

Eiman Kanjo, Alan Chamberlain, “Emotions in Context: Examining Pervasive Affective Sensing", Personal and Ubiquitous Computing Journal, 2015.

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

mFeel

Kanjo, E., "mFeel: An affective mobile system", IEEE Pervasive Computing, vol. 11, issue 3, pp. 43-45, 2012.

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Shopmobia

We leverage mobile phone processing power and physiological sensor in order to perceive and record customer emotions. Data are aggregated around different zones using NFC tagging technology.

Understanding Brain Activities and responses to Outdoor Places

Luluah Albarrak and Eiman Kanjo, "NeuroPlace: Making sense of a place", 4th International Conference on Augmented Human in Cooperation with ACM, Stuttgart Germany, ACM, March, 2013.

Eiman Kanjo, Alan Chamberlain,” ShopMobia: Emotion based Shop Rating System”, Affective Computing and Intelligent Interaction Conference, Geneva, Switzerland, IEEE, September, 2013.

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Mobile Phone Notification

and ResponsesAs Affective

Sensors

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Mobile Phone Notification

and ResponsesAs Affective

Sensors

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Physical Computing: Beacons

• Bluetooth beacons are hardware transmitters - a class of Bluetooth low energy (LE) devices that broadcast their identifier to nearby portable electronic devices. The technology enables smartphones, tablets and other devices to perform actions when in close proximity to a beacon.

• iBeacon introduced by Apple in 2013 to enable retail/location based payment.

• Then few versions of Beacons have followed.

• Eddystone is a Google's standard for Bluetooth beacons (released by Google in July 2015).

http://www.travelonart.com/new-media/che-cosa-sono-i-beacons/

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Eddystone

• Eddystone-UID functions in a very similar way to Apple'siBeacon, however, it supports additional data exchange.

• The beacon information available includes battery voltage,beacon temperature, number of packets sent since laststartup, and beacon uptime.

• Many Eddystone beacons have sensors (such as motionsensor, temperature, humidity), internal memory, longlifebattery, some have NFC chips in them.

Eddystone• There are two basic ways for apps to interact with beacons:

Monitoring and Ranging. • Monitoring allows detecting if there are beacons in range, • while Ranging is used to estimate proximity from beacons.

• Additionally, you can use the Physical Web/Eddystone standard tobroadcast URLs instead of beacon IDs.

• Android natively supports this as Nearby Notifications and on iOSit's integrated with Chrome.

• It's important because, on Android, Chrome is the default browserwhich makes it possible to reach millions of users without buildinga dedicated app.

BeaconsPhysical Computing/IoT enablers

Beacons

• Long Range Location Beacons: For indoor location, IoT prototyping or enterprise-grade deployments choose Location Beacons - the most robust beacons on the market. They support multiple packets, mesh networking, and have built-in sensors features: 5 sensors, NFC, mesh, GPIO, 1Mb EEPROM.

• Stickers are much smaller, but that also comes with shorter battery life (up to a year) and range. The idea behind stickers is to create nearables: smart objects broadcasting data about their location, motion, and environment.

https://estimote.com

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Current Projects based on Beacons

• For Fitness and to track Physical Movement

• Location based games (Pokemon Go Style)

•Monitoring Customer Behaviour in Retail Environment

• For Smart City applications and infrastructure

CodeMobile 2017 eiman.kanjo@ntu.ac.uk

Annotation on the go for Data Collection

1. Screen Buttons (Image Buttons)

2. Screen Swipe:(left and right).

3. Device Gesture: based on the phone accelerometer

motion.

4. Volume Buttons: logs volume levels.

5. Speech Labelling: For this, we utilised Android

Speech API for voice recognition to count words.

NFC as Annotation

Enablers (Proximity)

NFC on-Body: Two tags are positioned on the left and right

shoulder.

NFC tags on-Wall: scans NFC tags on the wall

Activity MonitoringGPS/Accelerometer data can be used to classify a user’s movement:

Running

Walking

Stationary

Combining motion classification with GPS tracking can recognize the user’s mode of transportation:

Subway, bike, bus, car, walk…

The Nike+iPod Sports Kit is an activity tracker device, developed by Nike, Inc., which measures and records the distance and pace of a walk or run.

Large number of Researchers are currently working on various acticty recognition algorithms which involves machine Learning (DeepLearning) on the go

Next Steps in the research world (1)

• Artificial Intelligence and Deep Learning to enable real-time training and prediction.

• E.g. On Mobile GPU, FPGAs(e.g. Lattice chip on Apple 7), Mobile TensorFLow…

Custom Sensors

Mobile phones still lack many sensing capabilities such as :

Blood pressure, heart rate, EEG

Barometer, temperature, humidity

Air quality, pollution, Carbon Monoxide

Specialized sensors can be embedded into peripherals:

Earphones

Dockable accessories / cases

Next Steps in the research world (2)

Next Steps in the research world (3)

• Think of it as a mini robot that can feel and monitor your behaviour.

• Or a form-changing device for gaming, entertainment or to deliver intervention(e.g. manage anxiety and stress).

Not just applications but new type of hardware, embedded devices and style are needede.g. Tactile biofeedback gadget

Actuators

Vibration

Heat/cooling

Electrical stimulation

Aroma generator

Mobile DevelopmentNottingham Trent University• Few Mobile Development modules are taught at NTU (Android &iOS)

• Undergraduate and Postgraduate Levels

• Mobile Development is popular among Final year projects

• New MSc Mobile Computing Course is planned

• Always on the look for Guest Lecturers

• Dedicated labs with a wide range of equipment including mobile phones for testing (one phone per student)

• This year our students have developed a wide range of applications and made use of a many native and external libraries.• E.g. most of our students have made use of Firebase to store and access their data.

• Many of the developed applications are professionally designed and tested.

• We encourage our students to choose ideas that are lied to real-business.

• Companies involvement is encouraged at early stages.

• If interested in delivering a guest lecture, engaging with students projects, or proposing a project idea, then please get in touch:

eiman.kanjo@ntu.ac.uk

Mobile DevelopmentNottingham Trent University