March 4, 1999Lethbridge1 Empirical Studies of Software Engineers: Some Experience and Pragmatic...

March 4, 1999 Lethbridge 1

Empirical Studies of Software Engineers: Some Experience and

Pragmatic Considerations

Dr. Timothy C. Lethbridge

University of Ottawa


Outline

A Case Study: The KBRE Project

Results of the KBRE Work So Far

Key Difficulties and Risks University ESE Researchers Face

How Can More Companies be Interested in Empirical Work

Issues that Must be Agreed Between the Company and the Researchers

Working With Corporate Employees and Managers



A collaborative project supported by:–Mitel–CSER (Consortium for Software Engineering

Research)–Involving The NRC

An industrially-driven project



Objective–To improve the productivity of software

engineers who have to understand and modify large software systems

Situation–Mitel has a large real-time software system

undergoing continuous development



Problems to be tackled

–New hires take a long time to learn the system

–Experienced people take a long time to make changes



Overall research strategy:–Study work of software engineers to discover

opportunities to handle problem–Analyze result, model their work patterns and

develop tool requirements–Develop prototype tools–Evaluate tools in work environment



Types of empirical studies performed –Studied software engineers at work

• Interviews, work observation, synchronized shadowing

–Monitored their use of all tools–Did some modeling of their mental models and

knowledge–Usability studies


Outline








Results of the KBRE work so far

Synchronized Shadowing–Involves two ‘observers’–Working on laptops with simple data input

interfaces–Each observing a different kind of data–Clocks are synchronized–Data is merged later to obtain composite view


Results of the KBRE work so farUsing Use Case Maps to Represent SE Work Patterns

execute search

look at search results

5w. Copy entire search results

5x. Copy some text from search results

5y. Search results no good

Exit withoutcopying searchresults

5c. Enter to look at search results

5a. Enter with copied text to search for

pastesearchargument

Cancel searchcommand

Exit totemporarily dosomething else

5b. Enter without copied text

5t. Immediately enter file from results

5s. Save results for quick access

5h. Enter to perform query based on previous search results

execute query


Results of the KBRE work so farAnalyzing the data to discover frequencies of patterns

Event types Percent ofTotal

StandardDeviation

Percent ofSubtotals

StandardDeviation

Numberof Events

Total number of events 966Copy text 9. 2% 6. 0% 89 Copy from file (1c, 7c) 66.3% 22.0% 59 Copy from search results (4x, 5x) 34.8% 23.8% 31Search 28. 3% 10. 3% 273 Search in editor (1p, 1e, 7p, 7e) 59.3% 17.1% 162 Search in system 40.7% 17.1% 111 Using grep (4a, 4b) 23.8% 15.6% 65 Using other tool (5a, 5b) 16.8% 16.6% 46Study 28. 5% 13. 9% 275 Study in editor (1r, 7r, 4t, 5t) 87.6% 7.1% 241 Study search results (4c, 5c) 12.4% 7.1% 34Paste text 10. 6% 7. 0% 102 Paste to modify text (1q, 7q) 7.8% 23.5% 8 Paste to search in editor (1p, 7p) 32.4% 22.7% 33 Paste to open file (2a, 8a) 22.5% 16.6% 23 Paste to search in system (4a, 5a) 37.3% 33.0% 38



Requirements for a source browsing tool–Strong support for search of all types–Make good use of screen real-estate–Make transitions from one type of information to

another easy to do–Allow heterogeneous graph structures in

exploration–Allow user to build their own mental model



–Allow easy context switching and return–Make it fast–Make it work with multiple languages–Make it USABLE!!!!!


Results of the KBRE work so farA tool architecture to support program comprehension

SEE (SoftwareExplorationEnvironment)

XCDB (CommandLine-OrientedAdvanced Query Tool)

KGREP (KnowledgeableRegular ExpressionSearch Tool)

OtherTools

User InterfaceSubsystems

CDB (sourceCode DataBase)

WSDB(Work Statemanagement)

Data ManagementSubsystems

CODE4(knowledgemanagement)

GCPMInterface(Generic

Configurationand ProblemManagement)

SliSim(prepares datafor simplifiedviews of code)

ReformOOlate(prepares datafor alternateviews of theentire system)

VisGen(prepares VRMLvirtual worlds)

AbGen(Abstractionanalyser andGenerator)

AnalysisSubsystems

Mitel Pascalparser unit

ParserModule

Cparser unit

otherparser unit

Key:Provides data to

Software Information System


Results of the KBRE work so farA tool being actively used at Mitel

An experimental test bed for the research of the whole research group

–7 graduate students, 2 research associates

Performed usability analysis–Even as a research platform, it has to be usable

and used so we obtain accurate data about the validity of the ideas


Outline









Those planning empirical studies must perform risk management, taking these into account

Establishing relationships with industrial partners

–We had it easy–The industrial partner sought us out


Key Difficulties and Risks

Attracting graduate students–Not so easy–Students often like design and working alone–Are empirical studies a ‘marketable skill?’

Researchers tempted to take ‘short cuts’–Just work on theory, toy problems–Just work with a few participants in the university


Key Difficulties and RisksLearning empirical techniques

–Not a traditional part of our background

Having the work recognized by colleagues

Training grad students

Avoiding crossing the line from research to development

Analyzing data from empirical studies


Key Difficulties and RisksResearch fails or goes slowly

–Hypothesis proved wrong–Methodology inadequate–Co-operation does not work–Researchers lack skills–Inadequate time–Students interest drifts

Company Priorities Change


Outline








How Can More Companies be Interested in Empirical Work?

Tangible benefits–New or improved products

Semi-tangible benefits–New or improved processes

Intangible benefits–Exposure of students to companies (potential

employees)


Issues that Must be Agreed Between the Company and the

ResearchersCompany and university objectives for the research

–Recognising the company imperative for the bottom line

–University imperative for publication

Access to facilities and employees–time commitment; participant selection process

Where will the work actually take place?


Issues that Must be Agreed

Deliverables, and planning for changes in research focus.

–How detailed should the project plan be?

Intellectual property and publication of results–Patents, copyright, trade secrets, non-disclosure,

masking of information, approval of release–Dealing with short publication lead-times.

How to handle the risks described earlier


Working with Corporate Employees and Managers

There must be an informed consent process

Understanding differences in attitudes of employee-participants.

What participants need to know–Support of management–Nature of the research–Disposition of data (confidentiality etc.)–They should feel a net benefit of the research


Conclusions

University based software engineering researchers should do more empirical studies in industry

–Should be part of regular SE research

Benefits, risks, and techniques need to be better understood

March 4, 1999Lethbridge1 Empirical Studies of Software Engineers: Some Experience and Pragmatic...

Documents

Transcript of March 4, 1999Lethbridge1 Empirical Studies of Software Engineers: Some Experience and Pragmatic...