Fall 2007 Informatics 231 Human-Computer Interaction 1 Methodology Matters: Doing Research in the...

Fall 2007Informatics 231 Human-Computer

Interaction1

Methodology Matters: Doing Research in the Social and Behavioral Sciences

Joseph E. McGrath

Gary Suh

Vesna Memisevic


Interaction2

Outline

1. Introduction2. Some Basic Features of the Research Process3. Substantive Domain4. Conceptual Domain5. Methodological Domain6. Research Methods as Opportunities and Limitations7. Research Strategies: Choosing a Setting For a Study8. Quadrant I: The Field Strategies9. Quadrant II: The Experimental Strategies10. Quadrant III: The Respondent Strategies11. Quadrant IV: The Theoretical Strategies12. Some Strategic Issues


Interaction3

Outline

13. Study design14. Comparison techniques15. Baserates16. The Correlation Question17. The Difference Question18. Randomization and “True experiments”19. Sampling, allocation and statistical inference20. Validity of findings21. Potential classes of measures in social psychology22. Strengths and Weaknesses of Types of Measures23. Techniques for Manipulating Variables24. Selection, direct intervention, induction25. Conclusion


Interaction4

Introduction

Doing research – the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events

Some of the tools with which

researchers in the social and

behavioral sciences go about

doing research (strategy,

tactics, and operations)


Interaction5

Some Basic Features of the Research Process

Always involves bringing together three sets of things:

(a) some content that is of interest,(b) some ideas that give meaning to that

content, and(c) some techniques or procedures by means

of which those ideas and content can be studied.


Interaction6


These three sets of things more formally, as three distinct, though interrelated domains:

(a) The Substantive domain, from which we draw contents that seem worthy of our study and attention;

(b) The Conceptual domain, from which we draw ideas that seem likely to give meaning to our results; and

(c) The Methodological domain, from which we draw techniques that seem useful in conducting that research.


Interaction7



Interaction8

Substantive Domain

Phenomena and Patterns of phenomena are the object of our study

The phenomena of interest involve the states and actions of some human systems and the conditions and processes that give rise to and follow from those states and actions.


Interaction9

Substantive Domain

Example – an individual casting a vote in a county election


Interaction10

Conceptual Domain

Properties of the states and actions of those human systems that are the focus of study

Relations refer to any of a variety of possible ways in which two or more elements can be connected

Examples – causal connections, logical relations, chronological relations


Interaction11

Conceptual Domain

Examples – attitude, cohesiveness, power, social pressure, status


Interaction12

Methodological Domain

Basic sets of elements or “tools” by which social and behavioral scientists systematically gather empirical information:

Measuring Manipulating Controlling Distributing Impact


Interaction13


Measuring– For assessing the state or magnitude of some property of

some actors-behaving-in-context so that the researcher can determine what value or level that feature has for each “case” to be studied

– Examples – questionnaire, rating scale, personality test, instruments for observing and recording communications, techniques for assessing the quality of some products resulting from individual or group task performance


Interaction14


Manipulating– Techniques for manipulating some property of an

actor-behavior-context– Experimental manipulation – making a feature

have one particular predetermined value or level for certain “cases” to be studied and another specific preordained value or level for certain other “cases,” so that the effect of differences in that property can be assessed by comparing those two sets of “cases”


Interaction15


Manipulating– Social psychologists have tried to manipulate features of

the systems they study by a number of techniques, such as:

(a) giving instruction to participants

(b) imposing constraints on features of the environment

(c) selecting materials for use

(d) giving feedback about prior performances

(e) using experimental confederates– (More is said about techniques for manipulating variables

near the end of this chapter)


Interaction16


Controlling– A set of techniques for controlling the impact of features

that are important but that you are not going to measure or manipulate in a particular study

– These include: Techniques for experimental control – you make certain

features take the same predetermined value for all cases in the study

Techniques for statistical control – you try to nullify the effects of variations in a given property within a study by “removing” those variations by statistical means

Techniques for distributing the impact – so that such impact can be taken into account in interpretation of results


Interaction17


Distributing Impact– Techniques for distributing the impact of a

number of features of the system and its context-without directly manipulating or controlling any one of them-so that such impact can be taken into account in interpretation of results

– Randomization – the most prominent means; refers to procedures for the allocation of “cases” among various conditions within the study


Interaction18

Research Methods as Opportunities and Limitations

(a) Methods enable but also limit evidence.

(b) All methods are valuable, but all have weaknesses or limitations.

(c) You can offset the different weaknesses of various methods by using multiple methods.

(d) You can choose such multiple methods so that they have patterned diversity; that is so that strengths of some methods offset weaknesses of others.


Interaction19

Research Methods as Opportunities and Limitations

The fundamental principle, in behavioral and social science is that credible empirical knowledge requires consistency or convergence of evidence across studies based on different methods.


Interaction20

Research Strategies: Choosing a Setting For a Study

When you gather a batch of research evidence, you are always trying to maximize three desireable features or criteria:A. Generalizability of the evidence over the populations of Actors.B. Precision of the measurement of the behaviors that are being studied (and precision of control over extraneous factors that are not being studied).C. Realism of the situation or Context within which the evidence is gathered, in relation to the contexts to which you want your evidence to apply.


Interaction21


Although you always want to maximize all three of these criteria, A, B and C simultaneously, you cannot do so.

This is the fundamental dilemma of the research process.


Interaction22



Interaction23

Quadrant I: The Field Strategies


Interaction24


The two research strategies in quadrant I are the Field Study and the Field Experiment.

Field study – the researcher sets out to make direct observations of “natural”, ongoing systems, while intruding on and disturbing those systems as little as possible.

Field experiment – also works within an ongoing natural system as unobtrusively as possible, except for intruding on that system by manipulating one major feature of that system.


Interaction25


The essence of both of the strategies in quadrant I, the field study and the field experiment, is that the behavior system under study is “natural”, in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study.


Interaction26

Quadrant II: The Experimental Strategies


Interaction27


Laboratory experiment – the investigator deliberately concocts a situation or behavior setting or context, defines the rules for its operation, and then induces some individuals or groups to enter the concocted system and engage in the behaviors called for by its rules and circumstances.

Experimental simulation – the researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (or apparent realism) of field studies.


Interaction28


The two strategies in Quadrant II, in contrast to those of Quadrant I, involve concocted rather than natural settings.

The laboratory experiment and the experimental simulation are strategies that involve “actor-behavior-context” systems that would not exist at all were it not for the researcher’s interest in doing the study.


Interaction29

Quadrant III: The Respondent Strategies


Interaction30


Sample survey – the investigator tries to obtain evidence that will permit him or her to estimate the distribution of some variables, and/or some relationships among them, within a specified population

Examples – public opinion surveys on voting intentions, political preferences, buying intentions


Interaction31


The strategies of Quadrant III concentrate on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system

Studies are usually done under “neutral” conditions of room temperature, lighting, chair comfort to nullify any effects of the behavior setting or context on the judgments that are the topic of study.


Interaction32

Quadrant IV: The Theoretical Strategies


Interaction33


Formal theory – the researcher focuses on formulating general relations among a number of variables of interest

Computer simulation – a complete and closed system that models the operation of the concrete system without any behavior by any system participants


Interaction34


The inclusion of these two strategies reminds us of the importance of the theoretical side of the research process.

One of the more powerful general strategies for research is the simultaneous use of one of the theoretical strategies (say, the formulation of a general theory) and one of the empirical strategies (for example, a laboratory experiment).


Interaction35

Some Strategic Issues

1. Does the material, as presented, properly reckon with the strengths and weaknesses of the research strategies it encompasses?

2. To what extent is the research evidence on each problem or topic based on use of only a single research strategy, and therefore limited by the weaknesses of that strategy; and to what extent is that body of evidence based on use of multiple, complementary strategies, with agreement or convergence among the findings attained via the different strategies?


Interaction36

Study design

1. Gathering observations

2. Aggregation and partitioning

3. Comparison on the data set


Interaction37

Study design

Comparison depends on:

What’s included in the study (what phenomena, what properties, what model of treatment for variables)

What system works (which substantive system we study, paradigms, strategies)

What conceptual relations have been posted (which properties are linked)

What comparison techniques are available


Interaction38

Comparison Techniques

Three base forms:

Baserates (how often?) Correlations (are properties related;

do they occur together?) Differences (comparison or

difference)


Interaction39

Base rates

How often Y occurs in the general case, as a basis for deciding whether the rate of Y in some particular case is “notably” high or low

Problems - difference in interpretation - various political, economic and social issues


Interaction40

The Correlation Question

Is there covariation in the values of two properties or features of system?

Correlations: High – Low; Positive – Negative; Zero Linear or nonlinear relation between two or more variables Can asses conceptual relations that imply covariation between two or

more variables, but cannot asses any conceptual relations that are causal in their implications

Whether Y is present under condition where X is present and whether Y is absent when X is absent?

Samples are separated in two groups, one with members which have performance X, and another with members who don’t have performance X.

Some set of tasks is given to both of groups, and average task performance is compared for both groups.

Groups must be comparable on factors that might affect task performances!

41

The Difference Question

Informatics 231 Human-Computer Interaction


Interaction42

Randomization and True Experiments

Randomization = using random assignment procedure to allocate cases to conditions (in order to remove artifacts and observations that happen by chance)

In order for the study to be called a “true experiment”, study must include some randomization

Random allocation procedure doesn’t guarantee an equal distribution of any of the potential factors among the conditions being compared


Interaction43

Randomization and True Experiments

Possible problems:

– Reduce the scope of study, as some variables are hold constant, and therefore experimental variables will occur only at a few levels

– Reduce the realism of context of your study, designed the tasks serves us - not the participants' - purposes.


Interaction44

Sampling, Allocation and Statistical Inference

The way we choose cases which will be included in our study (from larger population of potential cases) effect credibility of the evidence resulting from the study

How to choose right nature of the “random sample” population?


We need to use random sample in that way that we can apply results of the study to the population of which cases constitute a random sample.


Interaction45


Interaction46


Sampling case procedure: which cases from larger population will be included in study.

Allocation case procedure: which condition every given cases (already selected as a part of the study) will be assigned to

Random refers to procedure, not outcome!


Interaction47


Size of the samples? The larger the number the more distributed those

cases will approach the idealized random distribution Uneven distribution doesn’t occur often if only

chance is operating.


Interaction48

Validity of Findings

Four different types of validity (Cook & Campbell):

Statistical conclusion validity Internal validity Construct validity External validity


Interaction49


Statistical conclusion validity: – Difference arisen just by chance?– Relationship between cause and effect

Internal validity:– How close can you come to

asserting that the present of X

caused the altered level of Y values? – Difference in Y associated with a

difference in X does not necessarily

imply a causal role for X– Are there other factors which may

have been covary with X and they, rather

than X, might have produced the change in Y


Interaction50


Construct validity: – How well defined are the theoretical ideas in our study? – How clearly understood are the conceptual relations

being explored? – How well are mapping of concepts and relations

External validity:– How confident you can be that your

findings will hold upon replication or

how general are your our findings. – How confident you can be that

you can make predictions ?


Interaction51

Potential Classes of Measures In Social Psychology

Strengths and Weaknesses of Types of Measures


Interaction52

Self reports Observations by visible observer

Observations by hidden observer

Archival records of public behavior

Archival records of private behavior

Trace measures

Reactivity High potential(Some participants are aware that their response will be recorded, that may influence how they response – they might try to make good impression, to give socially desirable answers or to try to help investigation)

High Low High Low None

Versatility Versatile to potential contents and to the population to which they would apply to

Can only be used on overt behavior (not feelings, expectations, etc)

Med-Low Low Low Low (many concepts not available)

Costs Low High High Medium Medium High (sometimes)

Dross Rates (discarded info.)

Low High (per observer hour bases)

High(per observer hour bases)

High High Very high (links to concept sometimes very week)

Time Low High High Medium Medium High

Errors Vulnerable to observer error

Ethical concerns, Vulnerable to observer error

No possibility to cross validate

No possibility to cross validate

Techniques for Manipulating Variables

Techniques for manipulating variables are not very well specified

Three classes of techniques: Selection Direct intervention Inductions

Manipulating variables are use to create an experimental manipulation of features of a situation


Interaction53

Selection

Advantage: Selection is the most convenient means to make sure that all cases of a given conditions are alike on a certain variable and that all cases of another comparison condition differ o that variable

Disadvantage: conclusion drawn is more or less unreliable unless all other potential factors are removed (uncertainty in nature of variable manipulated (additional meaning))


Interaction54

Direct intervention


Interaction55

Advantages: – Surest way of achieving a definite and specific variable manipulation– Permits random allocation (distributing impact of other factors and

variables, that we are not studying)– Nor likely to be costly or time consuming– Low dross rate

Disadvantages:– Applicable only for relatively overt

and tangible variables– May suffer from reactivity effects

and experimental demands

Induction

Three major forms1. Use of misleading instruction to the participants 2. Use of false feed back3. Use of experimental confederates

Advantages:– Fairy versatile in context of variables– Normally low in cost and time

Disadvantages:– Involves deception (raises ethical concerns)– Risk of detection– Risk of experimental demands


Interaction56

Conclusion

Results depend on methods. All methods have limitations. Hence, any set of results is limited.

It is not possible to maximize all desirable features of method in any one study; tradeoffs and dilemmas are involved.

Each study (each set of results) must be interpreted in relation to other evidence bearing on the same questions.


Interaction57

Questions?


Interaction58

Fall 2007 Informatics 231 Human-Computer Interaction 1 Methodology Matters: Doing Research in the...

Documents

Transcript of Fall 2007 Informatics 231 Human-Computer Interaction 1 Methodology Matters: Doing Research in the...