AGILE

PrinciplesPractices

Search

Definitions

More…

-Remember that AGILE (and related methods) is not static nor dogmatic, so mind more following the goals than the actual practices since some of them are not practical nor apply for maintenance and others need tailoring to match maintenance process. -Maintenance work should not be putting out fires but instead being a doctor on call: knowing the component (functioning and interaction) and past/present behavior regarding -maintenance (does it show error with each new release, previous CPS discussions and PTF's, etc) allows developers to be prepared.

Introduction of AGILE principles into software support by implementation of specific elements of Agile methods like SCRUM, LEAN and XP (Extreme Programming):

-CPS process, PTF propagation and internal PTR's procedures can greatly benefit from SCRUM. -Coding of fixes and trap/tools can benefit from certain practices of XP -Identification and removal of waste can be accomplished by applying 6 of the 7 LEAN principles into maintenance.

PRACTICES

DEFINITIONS

PRINCIPLES

WHAT IS AGILE FOR SUPPORT?AGILE for SUPPORT simply takes from AGILE methods intended for development (some of them adapted to development from their original purpose, like LEAN which was intended for manufacturing principles for Toyota) and applies what makes sense of them into the support process.

WHERE DID AGILE FOR SUPPORT COME FROM?While managing OptiConnect project and attending AGILE meetings

someone made a casual remark about looking into applying AGILE to support. That and the added resource the project had at certain point facilitated trying out different elements of XP (SCRUM and LEAN were culled in the beginning when the project was in the red). That allowed

the research and trial of AGILE practices in support. While not nearly half of what is outlined here was tried and implemented, it developed from

that experience.

IS AGILE FOR SUPPORT SIMPLY A MIXTURE OF ESTABLISHED AGILE METHODS?

No. Instead of picking methods that at a glance look good, the focus is in understanding what the methodologies strife for, then deciding which ones are mostly in synch with the support process purpose, so then a decision can be made regarding which particular practices are practical to help achieve the support goals.

KEEP CUSTOMERS HAPPY AND RETURNINGBy

delivering a good produ

ct(Develop-

ment’s job)

By resolv

ing issues showing up on the field

Working software and Timely delivery.Documentations of the fixes is just as important to us and customers as the actual PTF’s.So we have:a) Fixesb) Deadlinesc) DocumentationAs metrics of support that help keep track of projects and thus achieve our goal of keeping customers happy and returning. To achieve this AGILE for SUPPORT incorporates the AGILE software principles into support by taking what is compatible from SCRUM, LEAN and XP.

OUR MEASUREMENTS OF SUCCESS ARE CLEAR:

Problem

determination

Coding &

Testing fixes

Propagating fixes

across the supported releases

BROKEN DOWN, SUPPORT CONSISTS OF THREE SECTIONS:

The question now is:

Can AGILE Principles help us achieve our goals?

 - Stable time-boxed, short iterations (each CPS discussion including resulting PTF, each internal PTR, and each propagation are treated as an iteration) -Stakeholder feedback (CPS discussions need customer's positive feedback to be closed, PTF's need customer feedback to be approved, only propagations and internal PTR's with no customer waiting lack customer feedback)- Self-directed teams (having full ownership and highly trained team members accomplishes it)- Sustainable pace (Crit Sits and personnel rotation aside, it can be maintained ).  

The main features from AGILE can be used to a degree, some

requiring minor tailoring to support work dynamic.

FOR A SUPPORT TEAM TO BE ABLE TO INCORPORATE AGILE PRINCIPLES,THEY HAVE TO MEET SOME REQUIREMENTS:

- Full ownership

-From one to nine team members, preferably no less than 3, and no more than 9 since SCRUM typically specifies a number between 5 and 9, but even a one man team can implement this approach, albeit in a limited capacity.

- Whole team on same geographic site preferably, while it can be worked around having remote team members, it's best to have everyone on same site.

REGRESION text bucket is required

:

A reality in support is that many support teams do not even have an organized proper test bucket for REGRESSION, in such case Test Driven Development (TDD) applied to a point (refactoring cannot be practiced in support) can be applied to correct the situation:No testcases for REGRESSION exist, automated or manual:Compile a list of testcases from all approved fixes for the component(s), so they can be run for each new fix, and with each new fixes, add the fixes testcase (which is only a part of the fix UT) to the list, thus building a REGRESSION test bucket with the list of fixes testcases.

This way, TDD not only helps improve fix coding and UT, but it also helps build a REGRESSION test bucket over time by providing clear and defined testcases for each new fix.

OptiConnect will be used as an example for measuring CPS and propagation processes. An explanation of how SCRUM, LEAN and XP began to be used in an intuitive way will precede the actual explanation of how they should be incorporated. For code delivery (fixes) there are two variants: PTF's for discussions (CPS) or internal PTR's, and PTF's resulting from propagating approved PTF's.

To be consistent with TDD, each new testcase should be designed, coded, added to the REGRESSION test bucket and the list should be run before the fix is coded, so only the new testcase fails.After coding the fix, run the REGRESSION test bucket again, and the new testcase should work this time (if not, rework).When this is implemented, an effort should be made to seek a way to automate the REGRESSION test bucket (automation saves many hours that a manual run would consume). REGRESSION test bucket exists, albeit testcases are not automated: automation is the remaining step to be achieved.REGRESSION test bucket is automated: each new testcase from current and prior fixes should be added to it (and later on automated) to maintain REGRESSION test bucket current and enrich it.

•Understand: the code and fix to be propagated, 1 day.•Propagate: minimum if not superseded by another fix.•UT: since most errors can be exposed in a rigorous UT, 1 to 2 days of UT with no rework, 2 extra days of possible rework. UT can be broken down into smaller sections:

 PROPAGATION PTF'S: PROCESS OF PROPAGATE/UT/CODE REVIEW

/REQUEST PTF/REGRESSION FOR EACH PTF TREATED AS AN ITERATION:

•Follow internal UT checklist: 3 hours.•Execute modified code: 1.5 days.•Verify reported error no longer surfaces: 2 hours•Rework in case UT checklist or execution/verification show error: 2 extra days.

•Code review: depending on code complexity either formal meeting or simply email review. 1 to 2 days when reviewers make themselves available(if team has only 2 members, external reviewers must be used, to allow team members to engage in XP practices)•Request PTF: a couple hours tops depending on builder.•REGRESSION: depending on complexity of component/product, when automated it may take between one day to even five (OPC being an example of product whose REGRESSION takes usually 5 days):

•DEX: jobs testing DDM path and basic functionality, between 30 to 44 hours (14 hours each run, one against reported release, one against lower release and another against current release).•FTP testcases: jobs testing IP over OPC path, 2 hours•TESTOPC: DDM testcases in BATS environment modified to run over OPC between 2 system, testing management of OPC basic functionality: 6 hours•IPL testing: issues 100 IPL's spread evenly in 24 hours testing QSOC subsystem starting, reconnecting to the other systems (which are running DEX between them) each time: 56 hours.•APPCREG: APPC tool for simulating sessions between systems using standard APPC communications, tests OPC's extended function path method: 2 hours

•Investigation/rework due to errors found or generated VLOG's or other type of error logs: •Investigate to see whether it's an error caused by us or unrelated, usually 1 day, if error is ours then follow PTF for CPS steps starting at problem identification section, if error not ours, move on.•Approve : 30 minutes.

PTF FOR CPS/INTERNAL PTR'S: BESIDES PROCESS DESCRIBED ABOVE, HAVE TO TAKE INTO ACCOUNT PROCESS THAT TAKES PLACE FROM THE MOMENT A CPS IS OPENED (HERE EACH CPS DISCUSSION IS TREATED AS AN ITERATION INCLUDING RESULTING PTF):

•Review CPS content: 1 hour (add to it if L2 explaining is necessary over sametime)•Review corresponding PMR(s): 1 hour

•Verify reported problem belongs to your component (not confirmation it's your problem indeed): 10 minutes•Review docs collected against those specified in (PD/PSI) Documents Queue for reported component: 30 minutes.

•Go over collected docs (for OPC they can be many and large): 6 hours•Confirm collected docs point to a problem in reported component: 1 hour•Look in active CPS, CPS archive and closed CPS discussions databases for similar symptoms (and hopefully a solution): 2 hours•Problem determination: this is the hardest part of the process to break down and measure, but once needed documentation is obtained, developer should be able to give estimate of needed time for next step, which of course varies for each iteration.

•Decide on whether final or test PTF will de delivered: 1 hour.•Request APAR: only then should APAR be requested to avoid impact on tracking metrics regarding PTF delivery: 1 hour•Constant updates on CPS discussion depending on PMR severity: 1 hour each if technical update.•Everything already described for propagation PTF except understanding and propagating sections•Fill out CPSE: it's supposed to be filled out as the process goes, but seasoned developers usually fill it out at a later stage and in one time: 2 hours for simple fix, 8 otherwise.•ARB review: produce evidence of code review, UT, REGRESSION, CPSE y Coverletter, 6 hours to 1 day (depending on reviewer availability)•Send PTF on to RETAIN and update CPS: 2 hours. Following section apply for test PTF's.•Wait for customer feedback, which can be anywhere from a week to 3 months.•Approval or rework: depending on feedback, either approve or start over from reviewing delivered docs.

•Early in problem determination (2 to 4 days into it), determine whether info is enough to track problem and commit a date for update regarding fixes, or trap/tool is required: •Brainstorm to determine what's

exactly needed (including what to codify to get it): 2 days•Test copy of coded trap/tool to verify it works: 6 hours•Negotiate acceptance of trap with level 2: usually 3 days (level 2 requires approval from customer).•Next steps skipping solution design and test or final PTF decision.

•Problem ownership: once needed info is on hand, estimate time needed to confirm ownership of problem or that problem is not ours (in which case our iteration would end alongside our participation in this issue).•Solution design: once ownership is determined, brainstorm to select best solution: 2 days, add 3 more days in case problem can be replicated and solution therefore tested.

The most crucial aspect of this is not just having each task measured for status update, but to be able to determine time estimates for each and every step of the CPS process, especially problem determination. An informed customer is a satisfied customer already. Even when an issue drags on due to its complexity or testing/doc collection difficulties, being able to inform customer of each and every stage and commit to a target date for each one keeps customer eased and aware of our efforts, which in turns will result in a more manageable and flexible customer.

AGILE METHODS CAN HELP ACHIEVE THE SUPPORT GOALS DESCRIBED ABOVE, FOR

THEIR OWN GOALS ARE VERY MUCH IN LINE WITH SUPPORT:

- XP goal: reduce cost of change.- LEAN goal: focus on what's needed only (considers the expenditure of resources for any means other than the creation of value for the presumed customer to be wasteful, and thus a target for elimination).- SCRUM goal: management of software development process/activities.

Following our case study, OptiConnect started some of the AGILE method's elements:- When it was in the red, SCRUM meetings were implemented to immediately deal with technical roadblocks and problems (servers not working, REGRESSION tools malfunctioning): it helped identify critical issues and prioritize them.- Also in the red, LEAN elements were intuitively applied to the handling the issues: delaying what could wait and focus on what mattered most.- When project progressed to a yellow status, XP was tried for propagations: several issues with apparently simple propagations emerged which prevented defects in original fix from being propagated.

•The SCRUM "pig" roles hold to a degree: ScrumMaster would be the most experienced team member (whomever but the team leader), the Product Owner would be the level 2 contact (although his almost daily interaction would be restricted to the CPS discussion mainly unless contacted via sametime, level 2 cannot be on site), and Team would be the rest of the team. However the "chicken" roles (Users, Stakeholders, Managers) would not apply (maybe first line manager few times a week).

LEARNING AND BUILDING FROM THIS EXPERIENCE, THE APPROACH AGILE FOR SUPPORT TAKES IS THIS:

- CPS process (and to an extent PTF propagation and internal PTR's) can benefit from Scrum:

•The "sprints" would be each CPS discussion, internal PTR, or PTR propagated across supported releases.•The documents are not very practical as intended: the product backlog could be used to set items, priorities and time assigned to them each two weeks, with progress being tracked by the SCRUM meeting, the sprint backlog the maintenance activities shorter than 16 hours, which would be REGRESSION sections (one can run DEX, another FTP, another TESTOPC, etc), UT checklist, while the Burn down could be used for propagations and maintenance process once past the problem determination phase.

•The SCRUM meetings can be implemented in their entirety (except for team members working remotely on occasion, they would require to call in): the 15 minutes duration, the three questions (What have you done since yesterday? What are you planning to do by tomorrow? Do you have any problems preventing you from accomplishing your goal?).

Codification can benefit from XP practices (some of them, and to an extent, since unlike in development projects, maintenance does not suffer from this need to adapt since the only thing they have to watch out for is difficulty of problem determination and fix coding, the rest is pretty constant):

•Fine scale feedback (Pair programming

for fixes and propagations, Planning game

only the iteration planning meeting to

assign tasks although this runs counter to

the intention of the sprint backlog from

SCRUM, Test driven development since

REGRESSION is already automated, Whole

team would not apply)

•Continuous process (Continuous

integration can be applied regarding

REGRESSION and propagation run time,

Refactoring as mentioned for TDD does

not apply since code must not be

changed except for fixes, Small releases

can be applied if we consider

propagations and fixes to be our releases)

•Shared understanding (Coding

standards does not apply since fix

must follow module used standard,

Collective code ownership applies,

Simple design applies, System

metaphor does not apply)

•Programmer welfare

(Sustainable pace applies

only when there are no crit

sits).

AVOID WASTING TIME AND EFFORT ON FUTILE ACTIVITIES CAN BENEFIT FROM LEAN PRACTICES:

•Eliminate waste: some activities in maintenance that contain some or can be total waste are:•Going back and forth for days on CPS discussions with level 2 regarding trivial stuff or anything that can be addressed much quicker on sametime and later posted on CPS discussion as evidence.•Waiting time for code reviews to take place (some teams like to take 3 days upon receiving review code to actually review it) or ARB reviews (wasted time looking for people willing to volunteer as moderators).•Trying to determine a problem with inadequate documentation when more can be obtained.•Fighting for test systems (they should be asked for in advance if dedicated cannot be obtained)

Build quality in: this does no apply since conceptual integrity is determined at design&development phase, not maintenance.

Defer commitment: Maintenance makes

crucial decisions based on customer info and

problem determination, so delaying them as

much as practical so they are based on facts

and not guesses or assumptions makes sense

to avoid hanging ourselves with activities like:

Deciding whether a fix can be delivered

already, or if a trap is needed and how to code

them.

Deciding whether test or final PTF will be

delivered.

Asking for an APAR since metrics measure time

since APAR open until fixes is delivered and

approved and time spent coding and testing

the fix can impact our metrics.

Deliver fast: obviously fast delivery of fixes and propagations is key.

Focus on learning:outside documenting components that were delivered

with lacking documentation (unfortunately a reality and necessity when it

happens) and testing tools (same thing), LEAN favors learning through

iterations feedback from both customers and team members. In CPS

customer feedback would not be as informative outside of errors/conditions

perceived by them that our test systems fail to show, or their environments

for us to take into account when testing. This practice can be implemented

only to a point, since in reality maintenance crews have to devote most of

their learning time to master their components and test tools. But using pair

programming as stated above can greatly enhance learning.

Respect people (empower the team):

this practice can be implemented to a point,

since in reality maintenance is where level

entry developers begin sharpening their

teeth. At that point they can't be trusted to

do their own work without supervision and

guidance. Only with experienced developers

can this be implemented, having clear goals

and metrics.

Optimize the whole: Maintenance requires not only a basic understanding of the components that interact with the owned components/product, but also problem determination, testing practices and even people skills when dealing with level 2 (and through level 2, the customer). It’s not just to deliver a good fix. It’s about the company’s interaction with its customers when problems are reported. All the things involved (having test systems available, keeping level 2 daily updated, committing to specific dates in every part of the maintenance process, be knowledgeable of testing practices and tools, reviewers commitment, constant education and training, etc) have a cost indeed, but they are all part of maintenance goal which is to keep customers happy with our service and coming back.

THIS WAY AGILE FOR SUPPORT USES SCRUM TO KEEP TRACK OF OVERALL PROGRESS, XP TO MAXIMIZE QUALITY OF CODE, AND LEAN TO REDUCE WASTE.

Has any project implemented AGILE for support?

OptiConnect: my project, where the idea originated. Here some of the practices were intuitively implemented and further developed. Implementation of XP has unearthed hidden defects in propagations. LEAN has helped cut wait time from code reviews and PMR discussions. SCRUM has helped keep track of all aspects of the project and improve handling of issues.Print CPS: pilot team under manager Leonel Perez (Leonel Perez/Mexico/IBM) for the full implementation of this approach. The team will begin implementation in a couple weeks.Surepos ACE and 4690: teams under manager Guillermo Hernandez (Guillermo Hernandez G/Mexico/IBM) have implemented XP as best practice and to bring new team members up to speed, SCRUM meetings.

YES, To a degree:

HAVE A NICE DAY!