Automatically Repairing Broken Workflows for Evolving GUI Applications

2013.11.27

Gigon Bae

Sai Zhang, Hao Lü, and Michael D. Ernst

ISSTA 2013

Introduction

Motivation

Overall Approach

Method

Empirical Evaluation

Related Work

Conclusion

Discussion

Contents

2 | 27

Broken Workflow Problem

3 | 27

The software developers evolve the GUI over time to improve the user experience. This can create usability problems for end-users, breaking an end-

user’s workflow --- a sequence of UI actions to complete a specific task.

Recovering from such broken workflows can be tedious, laborious, and frustrating. E.g., Microsoft Office products.

Crossword version 0.3.0

Crossword version 0.3.5

Test engineers often write test scripts to mimic end-user workflows to automate testing of a GUI application.

Such test scripts are fragile to UI changes. 74% of test scripts become unusable between successive

releases of a typical GUI application [1]. Even simple modifications to GUIs resulted in 30% to 70%

changes to test scripts [2].

GUI Evolution Problems to SW Developers

4 | 27

[1] A. M. Memon and M. L. Soffa. Regression testing of GUIs. In FSE, 2003.[2] M. Grechanik, Q. Xie, and C. Fu. Maintaining and evolving GUI-directed test scripts. In ICSE, 2009.

Manually repairing every broken workflows Too many GUI screens and UI actions

: tedious and frustrating

Programmatically comparing the GUIs of two versions, identifying changed GUI elements, and then locating UI actions that reference these modified GUI elements For some UI actions, their effects cannot be observed

without actually executing actions. The GUI can change substantially

: without knowing the precise “action semantics”, it is generally impractical to find a replacement GUI elements and a suitable action on it from the new application version.

Prior Approaches

5 | 27

During the evolution of a GUI application, the underlying system that implements a given functionality often stays relatively the same between versions, even when its GUI evolves rapidly.

Key Observations

6 | 27

A program-analysis-based solution: uses program analysis to repair broken workflows that are affected by GUI evolution.

The suggested technique, called FlowFixer, recommends replacement UI actions in the updated GUI application to complete the same workflow.

1. Dynamic Workflow Profiling

2. Static Method Matching

3. Random Action Execution

4. Replacement Action Recommendation

Approach

7 | 27

Input: two versions of the application

Output: replacement UI actions in the updated GUI application

Overview

8 | 27

Old version New version

FlowFixer first instruments the old version of the tested application offline by inserting code to monitor each method’s execution at run time.

FlowFixer ask user to demonstrate a workflow on its GUI.

Event handlers and other methods invoked at run time are recorded in the execution trace.

1. Dynamic Workflow Profiling

9 | 27

Previous work on matching program elements across program versions is unavailable.

does not support the latest Java version.

focuses on identifying changes by a small set of refactorings.

Created a tool to perform method matching Uses the first one that succeeds

1. Identical Method heuristic

2. Similar Name heuristic: using Levenshtein string similarity (default threshold: 0.9)

3. Co-evolving heuristic: observing the deleted method and the replacement methods in the same revision

2. Static Method Matching

10 | 27

FlowFixer employs random testing to generate and execute random UI actions on the new version, and observes the invoked event handlers and methods in the background.

For the sake of efficiency, it approximates the exploration by executing each UI action (at most) once.

Several kinds of UI events such as key pressing, mouse moving, and window disposing are ignored.

3. Random Action Execution

11 | 27

Example of Random Execution Algorithm

12 | 27

Window 1

Modal Dialog 1

Initial screen

Invokes:method A (event handler)method Bmethod C …

Recommends a list of replacement UI actions that may complete the same workflow in the new version.

Check each matched method in the new version and then infer which UI action is most likely to invoke it.

Focuses on the “uniqueness” of a method being invoked by UI actions.

4. Replacement Action Recommendation

13 | 27

4. Replacement Action Recommendation

14 | 27

1/3 1/3 1/3

1/1

Event handler

FlowFixer is implemented by using WALA framework and the UISpec4J library.

WALA: to perform offline instrumentation of Java bytecode.

UISpec4J: to execute UI actions and gather method invocation information.

The FlowFixer implementation is publicly available at

http://workflow-repairer.googlecode.com

Implementation

15 | 27

1. How accurate is FlowFixer in repairing broken workflows for real-world GUI applications? (what is the rank of the actual replacement UI actions in FlowFixer’s output?)

2. How long does it take for FlowFixer to repair a broken workflow?

3. How does FlowFixer’s effectiveness compare to an existing approach based on GUI comparison?

4. How does FlowFixer’s effectiveness compare to an alternative approach based on static analysis?

Research Question

16 | 27

Subject programs

Five popular GUI-based applications available at SourceForge.net• Chose two versions that contain significant GUI changes.

› Analyzed all documented workflows from user manual.

Subject Programs and Broken Workflows

17 | 27

FlowFixer repairs realistic broken workflows with high accuracy for evolving GUI applications with non-trivial GUI changes

Result: RQ1 - Accuracy

18 | 27

RQ1 : how accurate is FlowFixer in repairing broken workflows for real-world GUI applications? (what is the rank of the actual replacement UI actions in FlowFixer’s output?)

Example of Repairing Workflow

19 | 27

Invoked UndoableEditImpl.createTemporaryFile in common.

- Uses 27 minutes to execute random UI actions on each subject program.

- Spends an average of 3.2 minutes to recommend replacement UI actions for one workflow.

Result: RQ2 – Time Cost

20 | 27

FlowFixer repairs realistic broken workflows with acceptable time cost.

RQ2: how long does it take for FlowFixer to repair a broken workflow?

Comparing GUIs of two app. Versions without analyzing the program is insufficient in repairing realistic broken workflows.

Result: RQ3 – Comparison with a GUI-Comparison Approach

21 | 27

RQ3: how does FlowFixer’s effectiveness compare to an existing approach based on GUI comparison?

※ M. Grechanik, Q. Xie, and C. Fu. Maintaining and evolving GUI-directed test scripts. In ICSE, 2009.

※

Random testing, though neither sound nor complete, can provide more accurate and useful results than static analysis.

Result: RQ4 – Comparison with Static Analysis

22 | 27

RQ4: how does FlowFixer’s effectiveness compare to an alternative approach based on static analysis?

Analyzing and Testing GUI Applications Automated GUI Testing Techniques

• Maximize coverage.

Avoiding bugs in GUI applications• Prevent an existing bug from happening again.

Supporting Software Evolution Debugging techniques

• Help software developers understand divergent program behaviors between versions

Automatically repairing unit tests and GUI test scripts

Patching Software Errors Automated software error patching

Related Work

23 | 27

Presented a practical technique for repairing broken workflows for evolving GUI applications.

Showed that FlowFixer is accurate and efficient. The source code of FlowFixer is publicly available.

Future work would be performing a user study to evaluate FlowFixer’s usefulness to end-users and investigating some possible applications of the FlowFixer technique. E.g., automatic update of the documented workflows in

the user manual as a GUI app. evolves.

Conclusion

24 | 27

The technique finds only one broken action.

A broken action that is replaced to a sequence of actions might not be found.

Considering methods invoked at runtime would be useful for other GUI testing techniques.

Discussion

25 | 27

Thank you :)