Design & Evaluation of the Goal-Oriented Design Knowledge Library

Design and Evaluation of the Goal-oriented Design Knowledge Library

Framework

iConference 2012

Andrew HiltsEric Yu

Faculty of Information, University of Toronto

February 9, 2012

iConference 2012Hilts & Yu 2

The GO-DKL Framework

•A model and method for extracting, codifying and storing relational excerpts of design knowledge from scholarly publications.

•A method for analyzing such a knowledge base to support information systems design.


Siloed e-Democracy Design

•Emerging scholarly community

•“Design knowledge” expressed textually in journal articles or white papers.

•A shared knowledge base could accelerate and strengthen advances in the domain.


Study objectives

•Unified model of scholarly findings for accessible and efficient retrieval.

•Support designers / analysts to leverage past work to apply to current problem contexts.

•Highlight related areas of application domain that may be relevant to a designer’s project.


Design Knowledge Management

•How to effectively manage information systems design knowledge?

•No one solution -- domain specific.


Design Knowledge: a metamodel



•Conceptualize

•What is design knowledge?

•What is not?

Fludd, R. (1619). Description of perception.



•Codify

•How do we obtain and store this knowledge?

Stohlman, T. (2008). Touching Rosetta. http://www.flickr.com/photos/namlhots/227505138/



•Retrieve

•How can this knowledge be accessed?

http://www.examiner.com/sites/default/files/styles/image_full_width/hash/StanfordLibrary-JustinSullivan_Getty%20Images-12-17-2004.jpg



•Represent

•How is retrieved knowledge depicted?

Magritte, R. (1935). The Human Condition.



•Situate

•How to contextualize?



•Evaluate

•How to determine if appropriate for usage context?


Methodology


Hevner et al’s model of I.S. research


A goal-oriented view of design knowledge

•Design knowledge as the relationship between I.S. design features and stakeholder objectives.

•Features may help or hurt goals.

•Assess potential designs based on potential goal satisfaction.


Building a design knowledge base

•How to codify design knowledge expressed in narrative textual form? Our approach:

•Qualitative, inductive coding scheme based on our notion of goal-oriented design knowledge.

•Coding scheme applied to academic papers; instances of scheme concepts tagged and converted into database records and relationships.

•29 codified publications.


Library and Project Data


Codification and Analysis


Retrieval

•How to effectively navigate these relationships?

•Goal-oriented browsing - start from goals, end up with design feature recommendations

•Directly browsing long lists of goals is not scalable.

•Solution - aggregate goal classifications

•User builds a custom project model, a collection of relevant goals from the retrieved information.


Application Example

•Facebook DSS•Design goals:

•Participation be incentivized

•External information be considered

•Degree of Consensus be obtained

•Social Pressures be reduced


Goal association

Users’ project goal

Users’ project goal

Aggregate goals

associated with

project goals

Aggregate goals

associated with

project goals


Goal browsing

Description of a library goalDescription of a library goal


Relationally guided retrieval

Impacted & Related library goalsImpacted & Related library goals


Tree-list representation and contextualization of project model

Alter contributionsto parent

Alter contributionsto parent

Deselect nodes to negatetheir contributions

Deselect nodes to negatetheir contributions

Export “model slice”Export “model slice”

Label colour calculatedbased on

received contributions

Label colour calculatedbased on

received contributions

Drill down to reveal high-level goals’ contributions

Drill down to reveal high-level goals’ contributions


A generated goal model slice


A closer look


Generated Design Feature Report


Generated Publication Report


e-Democracy design practitioner work practices: methods

•Semi-structured interviews with 6 individuals.

•Interviews transcribed and qualitatively analyzed.


e-Democracy design practitioner work practices: findings

•Primarily rely on tacit experience; little on codified guidelines.

•Inspired by popular websites; tacit evaluations thereof.

•Design goals very similar to those codified in the GO-DKL knowledge base.

“The things that inform me are not documentation about projects, but the

projects themselves.”


Practitioner Evaluation of GO-DKL

• Relevance

• Useful as a heuristic - “did I miss anything?”

• Anticipate problems in advance

• Suggestions

• Link to “off-the-shelf” solutions

• Goal Prioritization

• Designers can contribute

• Critiques

• Trustworthiness

• Vocabulary

• Usability

GO-DKL: A source of inspiration for

potential solutions to design problems that

must be primarily considered in their own

context.


Future Work

•Develop web-based community platform for goal-oriented design knowledge sharing.

•User-contributed design knowledge; less focussed on scholarly publications.

•Peer assessments of design relationship trustworthiness and contextual dependence.

•Extensive user testing and interface redesign.


Questions?

•[email protected]

•[email protected]


References•Ahmed, S. (2005). Encouraging reuse of design knowledge: a method to index knowledge. Design Studies, 26(6):565–592.

•Ahmed, S. and Wallace, K. M. (2004). Understanding the knowledge needs of novice designers in the aerospace industry. Design Studies, 25(2):155–173.

•Alexander, I. and Kiedaisch, F. (2002). Towards recyclable system requirements. In Proceedings Ninth Annual IEEE International Conference and Workshop on the Engineering of Computer-Based Systems, pages 9–16. IEEE Comput. Soc.

•Babar, M., Gorton, I., and Kitchenham, B. (2006). A framework for supporting architecture knowledge and rationale management. Rationale Management in Software.

•Davies, T. (2009). The Blossoming Field of Online Deliberation. In Davies, T. R. and Pena Gangadharan, S., editors, Online Deliberation: Design, Research, and Practice, pages 1–20, San Francisco, CA. CSLI Publications.

•Eilouti, B. H. (2009). Design knowledge recycling using precedent-based analysis and synthesis models. Design Studies, 30(4):340–368.

•Esfahani, H. and Yu, E. (2010). A Repository of Agile Method Fragments. In Munch, J., Yang, Y., and Schafer, W., editors, LNCS 6195: New Modeling Concepts for Todays Software Processes, pages 163–174, Berlin / Heidelberg. Springer.

•Muller, W. (1996). Typology and the organization of design knowledge. Design Studies, 17(2):111–130.

•Ormerod, T., Mariani, J., Ball, L., and Lambell, N. (1999). Desperado: Three-in-one indexing for innovative design. In Sasse, A. and Johnson, C., editors, 13th International Conference on Human-Computer Interaction, page 336, Edinburgh, UK. Ios PR Inc.

•Oxman, R. E. and Planning, T. (1994). Precedents in design: a computational model for the organization of precedent knowledge. Design Studies, 15(2):141–157.

•Wahid, S., Mccrickard, D. S., Chewar, C. M., and Chong, J. (2006). Entangled Design Knowledge : Relationships as an Approach to Claims Reuse.

Design & Evaluation of the Goal-Oriented Design Knowledge Library

Business

Transcript of Design & Evaluation of the Goal-Oriented Design Knowledge Library