Iterative usability evaluation of DSLs

Ankica Barišić, PhD student

Supervisors: Vasco Amaral, Miguel Goualo

A language is a means of communication

The user interface is a realization of a language

A language is a model that describes the allowed terms and how to compose them into valid sentences

2

General Purpose (programming) Languages (GPLs)User has to master programming concepts User has to master domain concepts

Domain Specific (modeling) Languages (DSLs)Meant to close gap between PROBLEM DOMAIN and

SOLUTION DOMAINReduce the use of computation domain conceptsFocus on the domain concepts

3

Increasingly popularRaise the abstraction level (closer to the domain)Narrow the design space

Several benefits claimed, in well-defined domainsProductivity gainsBetter time to marketAvoid error-prone mappings between domain and software

development concepts Leverage the expertise of domain experts

4

[Mernik, 2005]5

REUSE?

a model of the User Different knowledge sets Characteristics chosen are dependent on application domain

the hardware-software Platform set of computing, sensing, communication, and interaction

resources e.g. operating systems, memory size, network bandwidth, input and

output interaction devicesthe social and physical Environment

Where the interaction is actually taking placeDifferent languages may have different contexts of use

Their users are likely to have different knowledge setsA minimum set of ontological concepts is required to use the

language6

The user's view of the Quality of a productMeasured in terms of the result of using the

product, rather than its properties

7

In general, software language engineers DO NOT evaluate their languages with respect to their impact in the software development process in which the DSLs will be integratedOr, if they do, they are

extremely shy about it

8

What to measure?

10

11

[Barisic, 2011c]

12

• Introduce DSLs’ Usability evaluation during DSLs’ life-cycle iterations

• Design an effective experimental evaluation of DSLs that will provide qualitative and quantitative feedback for DSLs developers

• Produce user-centered design of DSL• Foresee the Quality of a DSL while in an iterative evolution step• Merge the Software Language development process with the

Usability Engineering process

13

Barišić, Monteiro, Amaral, Goulão, Monteiro:

"Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012

14



15



16

Language engineer

Domain expert

Usability engineer

per group

18

Language goals: Support users in creating and manipulating

Evaluation:Compare two different DSL solutions within same

domain sharing the same goalFor the purpose of having means to reason about what

is more usable: Abstract syntax definition Transformation Interaction

19

Development environment and methodology: EpsilonModel-Driven development

Domain: Management and organization

Transformation output: MS Excel template

Place:Summer school on MDD of DSLs, Faculty of

Organizational Science's

20

Domain Experts: subjectsUsability Experts: experiment designersLanguage Developers: lecturers, subjectsEnd Users: subjects

21

Lecturers Experiment designers Subjects

2 2 39

22

Instrument: background questionnaire

Our evaluation technique was tested with two individuals (two physics experts) in order to verify it and to test the teaching materials and questionnaires

As time constrants and equipment turn out to be adequat there was no need to change prepared materials

23

[Howatt 1995]

RQ1:Is querying with Pheasant more effective than with C++/BEE?

RQ2:Is querying with Pheasant more efficient than with C++/BEE?

RQ3:Are participants querying with Pheasant more confident on their performance than with C++/BEE?

Our goal is to:analyze the performance of Pheasant programmers plug-ins for the purpose of comparing it with a baseline alternative (C+

+/BEE)with respect to the efficiency, effectiveness and confidence of

defying queries in Pheasant from the point of view of a researcher trying to assess the Pheasant

DSL, in the context of a case study on selected queries

24

Instrument: background questionnaire

H1null Using Pheasant vs. C++/BEE has no impact on the effectiveness of querying the analysis framework

H1alt Using Pheasant vs. C++/BEE has a significant impact on the effectiveness of querying the analysis framework

H2null Using Pheasant vs. C++/BEE has no impact on the efficiency of querying the analysis framework

H2alt Using Pheasant vs. C++/BEE has a significant impact on the efficiency of querying the analysis framework

H3null Using Pheasant vs. C++/BEE has no impact on the confidence of querying the analysis framework

H3alt Using Pheasant vs. C++/BEE has a significant impact on the confidence of querying the analysis framework

25

We focus on presenting six examples, each focusing in some of the features we chose to evaluate

Participants are asked to give themselves a mark for feeling of correctness of their trial

Session take the time needed for each group to understand the examples

26

Every participant has four queries, specified in English, to be rewritten in previously learned language

Subjects makes self-assessment of his replay rating his feeling of correctness

Example:Build the decay of a D0

particle to a Kaon Pion

27

Results obtained with Pheasant were clearly better then those with C++/BEE

Pheasant allowed non-programmers to correctly define their queries.

The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using Pheasant

The feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative.

28

The participants were asked to judge the intuitiveness, suitability and effectiveness of the query language. The goal was to evaluate:Overall reactions Query language constructs

Affect to query language was rated by:Query language constructsParticipants’ comments

29

30

Foster higher productivity of DSL end usersIncreased quality in use and quality of

experience while using a DSLReplication of experimentsReusing data and recommendations Reduce complexityTraceability of usability recommendations

and design changesReduced cost of evaluation, development

and supportIncreased reliability of usability evaluation

results

33

Foster higher productivity of DSL end usersIncreased quality in use and quality of

experience while using a DSLReplication of experimentsReusing data and recommendations Reduce complexityTraceability of usability recommendations

and design changesReduced cost of evaluation, development

and supportIncreased reliability of usability evaluation

results

Literature [Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to

develop domain-specific languages, 2005, ACM Computing Surveys [Gabriel2010] Gabriel, P., Goulão, M. & Amaral, V. (2010). Do Software

Languages Engineers Evaluate their Languages? in XIII Congreso Iberoamericano en "Software Engineering" (CIbSE'2010)

[Barisic2011a] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of DSLs: Current Evaluation Methods’. Proc. 3rd INForum - Simpósio de Informática (INForum2011), Coimbra, Portugal, September 2011

[Barisic2011b] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of Domain Specific Languages: a Case Study’. Proc. Evaluation and Usability of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011

[Barisic2011c] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘How to reach a usable DSL? Moving toward a Systematic Evaluation’, Electronic Communications of the EASST, 2011

[Barisic2012] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Evaluating the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI Global, 2012)

35

Iterative usability evaluation of DSLs

Education

Transcript of Iterative usability evaluation of DSLs