8th Sakai Conference

4-7 December 2007Newport Beach

What does Quality Assurance What does Quality Assurance Mean to Sakai?Mean to Sakai?

Alan Berg

Megan May

Seth Theriault

TopicsTopics• What does a QA Director do?• QA Today

– Who is in the WG– Types of testing currently done and release activities– Strides forward

• Resources available – QA Network

• 2.5 Release Activities Update• QA Tomorrow• Questions

Role of QA DirectorRole of QA Director

• Coordination! – Help bring parties interested in the same testing

efforts together– Foster initiatives that support quality throughout

the community– Assist in bridging the gap between test and

design & development teams– Ensure release activities are transparent

QA Today QA Today

Development

Testing & Software Release

Requirement Gathering and

Design

Focus on Functional Testing

Who is the QA WG?Who is the QA WG?

2.4.0

93 People

37 Institutions

11 Countries

2.2.1

23 People

10 Institutions

4 Countries

2.3.0

44 People

17 Institutions

5 Countries

2.5.0

56 People

17 Institutions

4 Countries

. . . How do they get involved?. . . How do they get involved?

Strides ForwardStrides Forward

• Deployment testing on wider variety of platforms

• Integration of Accessibility WG activities in general QA cycle

• Expanding/updating library of test scripts• Reflection on Release process

– http://confluence.sakaiproject.org/confluence/x/fbU

QA NetworkQA Network

Amsterdam Boston U

Cambridge Cape Town

Columbia Indiana

rSmart Georgia Tech

Coming soon servers in Japan and Australia

CollabCollabHow to be in the know!How to be in the know!https://collab.sakaiproject.org/portal

JiraJirahttp://jira.sakaiproject.org/

ConfluenceConfluencehttp://confluence.sakaiproject.org/confluence/display/QA/Home

Perceived Quality LevelPerceived Quality LevelDevelopment

Management

QA

Support

Overall Perceived level of Quality

Quality, like beauty, is clearly in the eyes of the beholder.

Perceived Quality PollPerceived Quality Poll

User Perceptions

QA Tomorrow QA Tomorrow

• Sakai Community Concerns – Several prominent campuses experience serious

performance problems with 2.4

• 2.5 Release Update– Further refinement of process– Goals for 2.5

Paths ForwardPaths Forward

• Common set of testing vocabulary – Unit Testing– Integration testing– System Testing– Performance

• Load testing (lmp@umich.edu) • Integration testing

Development

Testing & Software Release

Requirement Gathering and Design

http://confluence.sakaiproject.org/confluence/x/M4B5

Roadmap for TomorrowRoadmap for Tomorrow

Metrics to Describe QualityMetrics to Describe Quality

2.3.0 – 85% Verification

2.2.0 – 85% Verification

(1,345 JIRA Tickets)

(611 JIRA Tickets)

(504 JIRA Tickets)

2.4.0 – 82% Verification

2.5.0 – Past bug verification

Metrics to Describe QualityMetrics to Describe Quality

Unit Integration System PerformanceKernelAssignmentsGradebookResources

Static code ReviewStatic code Review• Search for bug patterns• What to process: Source code, Binary• Two types of bug pattern

– Generic• Classic (buffer overflow, input validation)

– Contextual• Language specific• Framework specific• Configuration specific

• Proximity effect

Hinting nightlyHinting nightly

http://qa1-nl.sakaiproject.org

Random ExampleRandom Example

Duplicate CodeDuplicate Code

Do we wish to use the reports more Do we wish to use the reports more aggressively?aggressively?

• Are the developers taking note?– Bugs reported are not always correct– Bugs found not always important– But duplicate code, unit test coverage, and failing to deal properly

with exceptions hint strongly at quality.– Not enough Contextual searching for Sakai and spring

configuration specific issues– Reports do not explain enough– Answer probably not, but thankfully trends in the marketplace will improve

the quality of results overtime.

A year onA year on

• Update prototype– Less noise more explanation– Graphical tracking of change

• Recommend Eclipse Plugins

• Look at emerging products

• See as part of a series of QA processes.

• Agree on metrics to act on and then act.

A good book to readA good book to read

• Secure Programming with Static Analysis– Brian Chess, Jacob West

• Addison Wesley

• ISBN: 9780321424778

• Includes CD with software and examples.

What you can doWhat you can do• Pledge Resources

– What happens in the community has a direct local impact– Easy point of entry into Sakai– All skill sets are needed

• Discussion– Why hasn’t this been a priority?– What are obstacles in the way and how do we remove

them?

Questions

Other Sessions of InterestOther Sessions of Interest

• 2.5 QA Workshop (Thursday, 13:00 - 14:30 in Salon 5)

• The Automated Testing Conundrum in Sakai (Thursday, 15:10 - 16:10 in Salon 5)