Crowdsourcing the notation of domain specific languages
-
Upload
marco-brambilla -
Category
Software
-
view
364 -
download
1
Transcript of Crowdsourcing the notation of domain specific languages
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Using Crowdsourcing to Shape the Notation of
Domain-Specific Languages
Marco Brambilla, Jordi Cabot, Javier Luis Cánovas Izquierdo, Andrea Mauri
Contact: [email protected] @marcobrambi
Better call the crowdBetter call the crowd
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Prologue:
Context and
Motivation
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Context - 1
Domain-Specific Languages (DSLs) should be all about
• Domain
• Specificity
And yet, frequently challenged in terms of acceptance and adoption
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Context - 2
Software and modeling people: we tend to…
• Rely on tools
• Improve on methods
• Focus on performance, quality
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Context - 3
Software and modeling people: we tend to…
• Be anti-social
• Forget we are actually humans
• Reflect this on the research and solutions: not
for humans
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Problem Statement
Domain-Specific Languages (DSLs) should be all about
• Domain
• Specificity
And yet, frequently challenged in terms of acceptance and adoption
• We stick to MDE, testing, SwEng tools only
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Proposed Solution
Rely on COGNIFICATION for improving process
and quality of MDE practices1
Especially: rely on the human side
1 Jordi Cabot, Robert Clarisò, Marco Brambilla, Sébastien Gerard. Cognifying Model-Driven
Software Engineering. GrandMDE (Grand Challenges in Modeling) at STAF 2017.
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Episode 1:
Crowdsourcing and
Crowd Management
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Call the Crowd
Crowdsourcing: The process of building a computation
or information collection job
by using computers as organizers
by organizing the work as several tasks
by dealing with dependencies on data and tasks
by assigning tasks to humans
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
9-1-1, what’s your emergency?
initial hypothesis
Collecthuman
feedback
MDSE methods
Crowd or Social Platform
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Crowd Management: Design Process
A simple task design and deployment process, based on specific data
structures
Task Specification Task PlanningTask Execution
& Control
• Task Spec: task operations, objects, and performers Dimension Tables
• Task Planning: work distribution Execution Table for task monitoring
• Control Specification: task control policies Control Mart
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Task Design
Which are the input objects of the crowd interaction?
Which operations should the crowd perform?
How should the task be split into micro-tasks assigned to each
person?
How should a specific object be assigned to each person?
How should the results of the micro-tasks be aggregated?
Which execution interface should be used?
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Crowd task management problems
Task splitting: the input data collection is too complex
relative to the cognitive capabilities of users.
Task structuring: the query is too complex or too critical
to be executed in one shot.
Task routing: a query can be distributed according to
the values of some attribute of the collection.
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Splitting Strategy
Given N objects in the task
Which objects should appear in each MicroTask?
How many objects in each MicroTask?
How often an object should appear in MicroTasks?
Which objects cannot appear together?
Should objects be presented always in the same order?
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Crowd Management is hard
Task Type Object Type
AttributeOperation Type
*
Performer
Task Politician*
*
Object Attr.
Output Attr.
⎧
⎫
⎨
Politician: ObjectType
LastName: Attribute
Photo: Attribute
Classify by Political Party:
TaskType
MODEL
TRANSF.
(MT1)
MODEL
TRANSF.
(MT2)
Classify: OperationType
µTask723: µTask
J. F. Kennedy: Politician
J. F. Kennedy: Politician
Classify by Political Party:
Task
MO
DE
L L
EV
EL
INS
TA
NC
E L
EV
EL K
enne
dy
JFK.jp
g
LastNam
e
Photo
Party
NULL
3
4 6
Op. Parameter
Republican Party:
OpParameter
Democratic Party:
OpParameter
2
*
*
Classify by Political Party:
Task
Luigi: Performer
Mario: Performer
George Bush: Politician
µTask
Performer
TaskPolitician
*
Status
5
*
*
Mario: Performer
George Bush: Politician
Barak Obama: Politician
<<instance-of>>
<<instance-of>>
<<instance-of>>
1
OpT
ypes
Para
met
ers
Name
Platform
Oba
ma
BO.jp
g
NULL
METAMODEL STRUCTURAL MODEL
LastName
PartyPhoto
Non-spammer
*
Status
StartTsEndTs
Status
WORKPLAN MODEL
Bush
GB.jpg
NULL
Party
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Crowd Control Mart
µTObjExecution
Performer
TaskPoliticianSta
tus
Sta
rtTs
End
Ts
µTas
kID
Object Control
Performer Control
Task Control
Objec
tID
Com
pObjs
Task
ID
Com
pExe
cs
Name
PlatformStatus
PID
Eval
RightWrong
Eva
l
Dem
RepA
nswer
StatusLastName
Party
Photo
Par
ty
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Reactive System
Guaranteed
termination
Extensibility
Control Production Rules
Result Production Rules
Execution Modifier Rules
EXECUTION
OBJECT PERFORMER TASK
OBJECT CONTROL
PERFORMER CONTROL
TASK CONTROL
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Episode 2:
Crowdsourcing
DSL Notations
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
A (Simplistic) Traditional Language Design Process
Different phases
Feedback completely separated
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Levels of Crowd Involvement
Incremental role
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Our Current Challenge: Level (b)
Abstract Syntax
Concrete Syntax
Concrete Syntax
Concrete Syntax
Language in use
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Process in Use
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Input
(1) the abstract syntax of the language
(metamodel)
(2) a set of symbols representing the candidate
concrete syntax
(3) the mapping
(4) optional model examples
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Task Definition Principles
Set tasks and questions that grant coherency of
the language
Cover the whole language structure
Don’t ask many times the same question
Grant support of agreement
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Task Definition
Tasks with many correlated questions
Majority voting on tasks
Expertise and demographics considered
Pattern based approach
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Pattern-based Task Design
Patterns in the metamodel are used to create
Tasks and Questions
Tasks:
• Question(s)
• Alternative notations
• Interactive examples
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Generation of Tasks: Patterns
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Episode 3:
Use Case and
Experiment
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
BPMN
Standard
Widely adopted
Widely understood (20% notation)
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
BPMN
Standard
Widely adopted
Widely understood (20% notation)
Widely misinterpreted and underused
(80% notation)
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
INPUT 1: A Metamodel (Excerpt)
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
INPUT 2: Challenging the Notation
Can we do better
than the standard?
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Generation of Tasks
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Generation of Tasks
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
CrowdUser Interface
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Experimental Setting
85 participants, 10 countries
• undergraduate/graduate computer science students
• modeling experts
• IT professionals
Assessed against modeling, language specification and BPMN experience
Answer aggregation strategies compared (effectiveness and affordability ):
• static majority policy
• targeted agreement
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Results: Task and Events
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Targeted Agreement
minimum number of answers = 10
agreement level = 60%
maximum tasks = 50
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Targeted Agreement
minimum number of answers = 10
agreement level = 60%
maximum tasks = 50
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
And the winner is…
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
And the winner is…
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Is the Solution Good?
Ask the crowd!
On AMT
Compare
1st, 2nd and 3rd
alternative notations
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
AMT Results
Question: shown two models, which one do you prefer?
• 0.05$ per answer
• 204 answer
Results:
• 1st best notation preferred 60% of the times over the 2nd
best, and 69% of the times over the 3rd best.
• 2nd best preferred 67% of times over the 3rd best
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Epilogue:
Past and Future
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Conclusions
The past:Best notation actually identified
Problems in coherency of the notation
Eat your own dogfood: We did MDE for MDE
Ugly UI -> crowdsource it?
The future:Combination of crowd + AI for MDE?
Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages
Using Crowdsourcing to Shape the Notation of
Domain-Specific Languages
Marco Brambilla, Jordi Cabot, Javier Luis Cánovas Izquierdo, Andrea Mauri
contact: [email protected] @marcobrambi
Thanks for Calling!