MSR populations talk v2.key

65
A Population Approach to System Design Matthew Chalmers 5 60 30 Zippy FanBook 5

description

Overview of population approach, with more tech detail of core analytics, done for MSR Cambridge

Transcript of MSR populations talk v2.key

Page 1: MSR populations talk v2.key

A Population Approach to System Design

Matthew Chalmers

560

30

Zippy

FanBook

5

Page 2: MSR populations talk v2.key

A Population Approach to Ubicomp System Design

Matthew Chalmers

560

30

Zippy

FanBook

5

Page 3: MSR populations talk v2.key

Plus ça change...

“Users’ control of [ubicomp] systems should be maximised and, given the heterogeneity of likely patterns of use, such systems must also be tailorable by the user. It is not possible to predict the social conventions governing the modes of use of ubiquitous computing that will come into being, so empirical studies of new applications [are] essential.”

Perceptions and Expectations of Ubiquitous Computing: Experiments with BirdDog, a Prototype Person Locator, O’Shea et al., Proc. IEE ITaP, 1991.

Page 4: MSR populations talk v2.key

Seamful Design: Treasure

Limits, variation and dynamics in wifi infrastructure made into resources for designers and users

Page 5: MSR populations talk v2.key

Seamful Design: Feeding Yoshi

Limits, variation and dynamics in wifi infrastructure made into resources for designers and users

Page 6: MSR populations talk v2.key

Seamful Design: Castles

Limits, variation and dynamics in software module infrastructure made into resources for designers and users

Page 7: MSR populations talk v2.key

Analysis Tools: Replayer

Page 8: MSR populations talk v2.key

Analysis Tools: Replayer

Page 9: MSR populations talk v2.key

Analysis Tools: Replayer

Page 10: MSR populations talk v2.key

Analysis Tools: Replayer

Page 11: MSR populations talk v2.key

Shifting to iPhones

Page 12: MSR populations talk v2.key

Shifting to iPhones

Page 13: MSR populations talk v2.key

Shifting to iPhones

Page 14: MSR populations talk v2.key

Yoshi (2009)

Page 15: MSR populations talk v2.key

28GB of quantitative log data representing 18,000 hours of use by 182,000 users, and qualitative data

from 8,000 users

Page 16: MSR populations talk v2.key

Analysis tools: SGVis

Page 17: MSR populations talk v2.key

SGVis: Categorising users

Page 18: MSR populations talk v2.key

SGVis: Categorising users

Page 19: MSR populations talk v2.key

SGVis: Categorising users

Calculate centroids of each identified cluster

4 categories of users: top players, commuters, static players, beginners

Page 20: MSR populations talk v2.key

Seamful Design: FlexBook

Mobile social networking app

Dynamic integration of software modules at runtime

Shared via our “module store” server, or ad hoc peer-to-peer

Reviewing and user feedback

Implicit logging of use

Explicit comments and ratings shown in app and Facebook

Page 21: MSR populations talk v2.key

Post Factory

Page 22: MSR populations talk v2.key

A software engineering revolution

Conventional models of the software development process all claim that “at some point, a product is delivered to a user community, at which point, for that revision of the software, the design process is over.”Paul Dourish

Page 23: MSR populations talk v2.key

“When we look at an interactive system as an evolving artifact in use, it follows that the process of design does not end with the delivery of the system to some community of users. Instead, it continues as they use and adapt the system.”

A software engineering revolution

Page 24: MSR populations talk v2.key

2011: 85% of users had extended v4 with at least 1 add-on

(or module or plug-in or add-in or component or...)

No single structural definition—or class—defines it, and no one pattern of use either

Similar situation in IE, Photoshop, Outlook and beyond...

In structural terms, what is Firefox?

Page 25: MSR populations talk v2.key

Established approaches are based on program structure being the same wherever and whenever it is used

How to deal with the scale of human and software interaction?

How to reason predictively about structure and use?

How to make tools for designers, evaluators and users that support adaptation and appropriation?

Practice ahead of theory and engineering

Page 26: MSR populations talk v2.key

A population approach to system design

Class as a varied population of software structures, being changed in the real world of use

Class as population:

Dynamic instead of static

Stochastic instead of deterministic and discrete

Coupled with programs in use, instead of standing apart from them

Complementary to (and to be used along with) existing class concepts

Page 27: MSR populations talk v2.key

A simple example: EdinburghFestivalGuide

Modules FestivalEvents & FestivalVenues in 100% of instances

Module NextBusToMyNextVenue in 50% of instances and the module QuickestWalkingRoute in 40%

All recorded instances have a variable meanLocationOfUse that shows 90% of its values within the city of Edinburgh

Page 28: MSR populations talk v2.key

The population approach

Multiple representations of a software class (or type)

Tools to use, create or adapt these representations

Social interactions and practices involving those tools

Page 29: MSR populations talk v2.key

The traditional representation

A class is a structure, and all instances conform to it

Data structure: variables or state

Code structure: methods, procedures or functions

These form a structural description of potential behaviour and use

Assumption: exact match between the structure of the class definition, and the structure of each instance

Page 30: MSR populations talk v2.key

The traditional representation

Tests and changes need only be done on one thing at one time: the class structure

What’s true for the class is true for the deployed instances

Methodologically convenient but not consistently true

Page 31: MSR populations talk v2.key

Borrowing from biology: population thinking

A species is a varied and changing population of individuals

Assumed to be undergoing continual change and evolvution

Each individual member may differ from others, and variation among individuals is natural and vital to adaptation

Described by family resemblances among observable attributes

The species can be described by the set of current (or recently recorded) members: the aggregate of each individual’s genes, size, colour, shape, etc.

Page 32: MSR populations talk v2.key

A new representation: class as population

A class as a varied and changing population of instances

Assumed to be undergoing continual change and evolvution

Each individual instance may differ from others, and variation among individuals is natural and vital to adaptation

Described by family resemblances of structure and use

A class can be described by the set of current (or recently recorded) members: the aggregate of each instance’s structure, context, use, etc.

Page 33: MSR populations talk v2.key

Populations: variation and dynamism

What’s true for one instance may not be for another

90% of unmodified instances crash, but those with module A added don’t crash. Overall, 75% of instances crash

Analysis results may be different at different times

We advertise A and the percentages; a week later only 50% crash

Results for different contexts may differ

News of module A spreads faster in one country than another, so in Spain 25% crash while in the US 60% crash

Page 34: MSR populations talk v2.key

New and old representations

Variation/dynamism in the match between class & instances

Probabilistic statements/models, coupled to ongoing use, at the core of designs, analyses and interactions

Population serves as overarching and connecting concept

Page 35: MSR populations talk v2.key

Populations with family resemblances

Fit with how ‘classes’ are used in everyday language

Moving away from narrow reliance on necessary and sufficient membership properties

Application to software structures demands a holistic approach

Page 36: MSR populations talk v2.key

Why ubiquitous computing?

It’s about more than just mobile devices...

Systems that fit with context, interaction and activity

...which are varied and dynamic in ways difficult to predict

The system is part of the user’s context

System interactions, and how users are modelled and presented to others in them, are resources for new user activity

Page 37: MSR populations talk v2.key

Ubiquitous computing

A holistic view spanning technology, use and users

Mark Weiser: The unit of design should be social people, in their environment, plus your device

Robin Milner: Here we look for synergy between the societal vision on the one hand, and the development of scientific models and engineering principles on the other.

Page 38: MSR populations talk v2.key

New tools and new interactions

Analysis of modules, configurations and contexts

Which configurations and contexts are popular? Which are problematic? How are they used? Are there clusters that should be separately managed and designed for?

How does my setup compare to my friends’? Are there configurations like my own that doesn’t crash so much? How are they used?

Access to modules and usage data

May support free dissemination (P2P) or retain a degree of control

‘Lead users’ keen to try out new or specially instrumented versions

Page 39: MSR populations talk v2.key

A third representation: ostension

One or more population members are pointed out as examples

Many potential ways to make an ostension

e.g. a category or cluster selected from a population of instances

Page 40: MSR populations talk v2.key

From observed use patterns to software structure

MapTool is usually run along with DGPSLocationStream

UKTrainSchedule is frequently logged as being used in UK locations tagged as train station

FestivalEvents and FestivalVenues modules are frequently used with an unofficial module AlternativeGuideToTheFringe when the user is in Edinburgh Fringe Festival venues

Page 41: MSR populations talk v2.key

SGVis: Categorising users

Page 42: MSR populations talk v2.key

Girolami: statistical analysis of populations

Inferring significant family resemblances among disparate features of large dynamic populations

Reducing a chosen set of instances to representative code and usage data

Page 43: MSR populations talk v2.key

Girolami: statistical analysis of populations

Extending RJMCMC and TI approaches to online environment via Dirichlet process approach

Heterogeneous data => non-parametric hierarchical mixture model

Stochastic structures abstracted into formal models

e.g. CTMCs, reliant on MCMC posterior inference

Page 44: MSR populations talk v2.key

Calder: formal modelling of populations

An online framework set within end-user apps

Real-time, critical feedback to users

Longer-term feedback to evaluators and developers

Page 45: MSR populations talk v2.key

Calder: role-based interface automata

Methods are classified as Master, Servant or Valets, and as Input or Output :

Master (M) actions must be performed

Servant (S) actions are offered

Valet (V) actions are either servant or hidden

Input (?) actions receive data

Output (!) actions send data

45

Page 46: MSR populations talk v2.key

Example: a simple GPS system

46

0,0,0,0(c1,c4).(M,S)loc

1,0,0,0

2,0,2,0 2,0,1,0

(c1,c8).(M,S)req

(c8,c5).(M,S)log3,0,0,0 (c1,c8).(S,M)map

c1.M!display

Page 47: MSR populations talk v2.key

Calder: component replacement

When can we replace/substitute one component by another?

C1 ⊗ C2 ⊗ C3 ⊗... ⊗ Cn ⤳ C1’ ⊗ C2 ⊗ C3 ⊗... ⊗ Cn

C1’ has to offer the environment C2 ⊗ C3 ⊗... ⊗ Cn whatever the environment needs to work

Formalise work by services specified in a temporal logic

C1 ⊗ Env ⊨ Σ : the current system works

C1’ ⊗ Env ⊨ Σ : replace C1 by C1’ and the system still works

Not aiming to check C1’ ⊗ Env ⊨ Σ, but derive it...

Page 48: MSR populations talk v2.key

Deploy app at a large scale, and collect log data

Infer statistical patterns of module structure and use

Make event-based formal models of module configurations

Make probabilistic judgements about failures/exceptions

Feed patterns and judgements into role-specific tools for awareness and analysis

Support people in these roles in adapting the app and its use

Page 49: MSR populations talk v2.key

Scenario: FanPhoto

FanPhoto: two core components, for text and photo sharing

We give it out to 100 people

and they start using it

We start developing new modules

one for sharing data via Facebook

one for fast compression and sharing of text, photos and video via MANETs

Page 50: MSR populations talk v2.key

Scenario: FanPhoto

A few participants are interested in new modules

download them and show off that they are trying them out

Analysts keep track via log data streaming back

start to see a new subcluster forming

973

Page 51: MSR populations talk v2.key

Scenario: FanPhoto

In the IDE, developers can see the FanPhoto class

Modules used by a minority of users are shown in grey

▽FanPhoto▷TextSharing▷PhotoSharing▷FacebookSharing▷MANETmodule

Page 52: MSR populations talk v2.key

Scenario: FanPhoto

After some time, developers and evaluators meet to play back several weeks’ use

973

Page 53: MSR populations talk v2.key

Scenario: FanPhoto

After some time, developers and evaluators meet to play back several weeks’ use

919

Page 54: MSR populations talk v2.key

Scenario: FanPhoto

After some time, developers and evaluators meet to play back several weeks’ use

5126

23

Page 55: MSR populations talk v2.key

Scenario: FanPhoto

After some time, developers and evaluators meet to play back several weeks’ use

560

30

FanBook

5

Page 56: MSR populations talk v2.key

Scenario: FanPhoto

After some time, developers and evaluators meet to play back several weeks’ use

560

30

Zippy

FanBook

5

Page 57: MSR populations talk v2.key

Scenario: FanPhoto

A developer sets his IDE to do automatic updates

Code defining each new subclass appears

labelled with users’ and analysts’ names

linked to tools to analyse the ostensions that made them

FanPhoto

FanBook Zippy

Page 58: MSR populations talk v2.key

Ontology as structure

“An ontology is a formal specification of a shared conceptualization” Gruber, via SemanticWeb.org

Traditionally, the core of software design is about finding the perfect hierarchical structure

One structure for all known uses, contexts and people

Adaptation is the problem of changing it

Page 59: MSR populations talk v2.key

Ontology as process

Population approach couples process and structure

Design process in which structure is a resource for use, and in which use creates or adapts structure

Giddens’ duality of structure, Heidegger’s hermeneutic circle

Part of the process is adaptation of a structure to new contexts and uses

A design requirement for ubicomp... and CompSci?

Page 60: MSR populations talk v2.key

Designing for duality of structure

Moving from structure to use is commonplace

That’s what happens when you compile code and users run it

The trick is how to move back again

Making code for a class from a set of instances and usage histories

A process that allows for gradual adaptation of a class

Moving between complementary forms of class definition

Each move is (or should be) a social interaction

Page 61: MSR populations talk v2.key

Code as intension

A class’ code, compiled to create an executable shared among users

Population as extension

The deployed instances, that are adapted by users and which create log data shared with evaluators and developers (and other users)

Sub-population as ostension

A subset of instances selected by one of these people as particularly interesting or useful, and shared among them for comment and use

Reduction to a code signature reflecting family resemblances among the configurations and logs of that subset

Page 62: MSR populations talk v2.key

Ostension revisited

Most forms of ‘analysis’ afford ostensive definition

A way to focus on a subset of structures, contexts, uses and users

e.g. an evaluator’s video of particular users from a system trial, a user’s list of friends on Facebook, a developer choosing a name of a city to customise design for

Log data allow us to link to the software used

Extend previous work that links from video to log data

Page 63: MSR populations talk v2.key
Page 64: MSR populations talk v2.key

A population approach

A combination of theory, design, evaluation and use

Advances in all four needed for success

A circular process designed for duality of structure

Change and human agency as essential elements of the process

Multiple ways of representing a software class

Tools to use, create or adapt these representations

Users’, evaluators’ and developers’ social interactions and practices involving those tools... that feed back into low-level representations

Page 65: MSR populations talk v2.key

560

30

Zippy

FanBook

5

[email protected]

www.dcs.gla.ac.uk/~matthew