Thoughts on SW Development:

One Person’s OpinionDave Ogle

Highlights of Dave’s Career

• Did ad-tech work for an IBM fellow (they are really smart folks)• Took ad-tech work from concept to shipping products (learned about

teams)• Did short stint making network code work on failing project ($70M+)• Worked on Olympics – applying technology to solve real problems• Worked with yet another IBM fellow on ad-tech project (cutting edge

work)• Left IBM, worked in a startup that lasted 18 months and then failed• Came back to IBM• Joined a sinking ship……a $100M project that tanked • Spent time in architecture group trying to change how we develop

software (made some small inroads)• Heard a talk, saw a light, made a change• Joined a test organization that is responsible for testing products that

have an annual revenue of over $1B US – have loved it for the past 5.5 years

Key Thing To Remember● Software Development is a BUSINESS

Failure….• Why do projects fail?

– No real understanding of the customer • The greatest “widget” in the world won’t sell if customers don’t

need widgets– No real design for the product

• Great ideas, poorly designed, will fail– Poor development skills and practices

• Good ideas, properly designed, but poorly constructed, will fail– Poor Leadership

• Key lead people in over their heads• General belief that putting in more hours coding solves basic

design flaws– Inadequate Marketing

• The best ‘widget’ in the world won’t sell if no one knows about it– Product released before it is ready

• People are so gung-ho to get the product released, they forget to ensure that it meets customers’ expectations

Why care about testing?• Customers pay big $$ for SW systems • Outages cost lots of money1

– Average across the industry is $45,000/hour – That cost can reach millions of dollars per minute in

some fields• It can cost up to $1M to fix a defect in the field• In some fields the cost of outages can cost more than

just $$$

• Testing is an important component of the checks and balances process in SW Development– Testing allows a ‘third party’ to try out the product and put it

through its paces. – It ensures the wheels will stay on when driving down the freeway.

1) By Motti Tal, OpTier, Special to ZDNet 10/2008

What Do I Mean by ‘Testing’…in SW Development

• Three basic phases of testing– Unit Testing – done by the developer to make sure the

code they write works the way they think it works– Function Testing – done by a tester (or developer) to

make sure that when you put the pieces together things still work as expected

– System Testing – done by a tester to ensure that the product works well in the ‘system’

Why do we need all three phases of testing?

How the World Really Is

Systems Verification Test – Value Proposition

The Mission of the System Test organization is to efficiently andeffectively validate that WPLC products meet our customers' high expectations in the areas of consumability, reliability, and interoperability.

Translation: § Test using real, c us tomer-representative, complex environments § Test the product using interactions that simulate c us tomer traffic§ Test the product using real c us tomer-like loads, volume of data,

number of transactions per second, number of users, number of logins, etc., and we need to keep the systems running for long periods of time

SW Development is a Business

• The Software margin was flat at 85.2 percent (from 2007 Annual report)

• Software Revenue (from 2007 Annual report): $19,982

• Software Gross Profit (from 2007 Annual report): $17,015

– Numbers are in Millions of US Dollars• Part of the cost of SW development is

‘warranty’ costs – this takes straight from the bottom line/profit

Why Focus on Testing?

• In some companies, management has encouraged ‘good’ people to leave test

• Test has not always been considered a noble profession

• Many people believe no real skills are required to test – just need to be able to bang on a keyboard

• Testers are often thought of as the least skilled people in the organization

One View of the Geek Hierarchy

• Visionary• Architect/Designer• Chief programmer/Tech lead• Team leader• Developer• Co-ops/Interns• Folks who turn off the lights in the building • Testers

So What is the Problem?

• Are designs flawed?• Do people just write bad code?• Do people just not understand how to

test?• How hard can this be?

One Problem: Perspectives• Software is usually developed in logical pieces, by

different teams– Developer ‘A’s view on how something should work

often differs from how Developer ‘B’ views things. When you put them together….

• You are working on teams of 2-4…and may have seen the problem

• Imagine 10-20 teams of 8-10 people each, spread out across the globe

• E xam ple of names with funny characters in them

• The developer’s view of how a piece of SW is going to be used, and how a customer actually uses it are often (wildly) different

– Readers/writers example

Another Problem: Environment– Very few software solutions are self-contained

• You need to interact with OS, LDAPs, DBs, Web Services, etc.

– Most developers don’t understand all the intricacies of those interactions, so they design and develop solutions based on their (potentially flawed) understanding of these interactions

• Example: LDAP – 10 threads• Example: DB – each component calls the DB

to get the same piece of data• Example: different DBs work differently

The 9am Phenomenon

Your Product

Company’s LDAP Server

LOGIN

•Your code requires people to login, which results in a call to the LDAP server.•Your assumption is that the LDAP server can handle ‘n’ calls per second•You design your system to handle ‘n’ logins per second

•What you didn’t know is that several other teams made the same assumptions….

Product ‘x’

Login

Product ‘y’

Login

Another Problem: Load• Developers make assumptions about how things will work,

these assumptions often break under load– Example: The classic ‘SuperBowl’ ad phenomenon –

everyone sees the ad, goes the website, clicks the button, and down comes the webserver

– Example: migrating data from release ‘n’ to release ‘n+1’

• Data kept in memory• Migration takes a long time, systems are

unresponsive, so the customer reboots the systems and tries again.

Another Problem: Duration

• Simple mistakes and problems get exacerbated over time

– JCB example of the space program– Memory leaks – even tiny ones, cause

problems over time

So What Does It All Mean?• Testing is a skilled science

– You need to understand complex environments, interactions, black and white box testing methods, you have to be able to improvise, understand all the ways in which things can break

– You need to apply technologies like code coverage analysis, OPC, memory leak analysis, etc

• Testers need to understand and empathize with customers/users– Need to understand how they think, how they operate, what their environments

are like, etc. • A developer should not be the sole tester of their SW

– When you know how you think something should work and how it will be used, you CANNOT adequately test it, since users never use it the way you expect

– Developers like to find the “1” w ay in w hic h S W w ill w ork, tes ters have to hig hlig ht the ‘99’ w ays in w hic h it w on’t w ork.

• Test and development teams should be separate– The U G LY B AB Y s yndrome

• Testers and developers need to interact more, earlier in the process– Most good test teams have an amazing wealth of knowledge on how things

break…most developers don’t. • There is a set of knowledge that the test community gains during a cycle

– This knowledge must be shared with development and design and architecture and marketing and….

Conclusion• Testing is a challenging, hard job that needs

more CS professionals to help balance the process

• Right now, many test organizations are out of balance with respect to their development teams

– They lack the skills necessary to properly perform their jobs

• Left unchecked, this imbalance will drive many companies out of the SW business in the foreseeable future…or it will result in SW that is even more flawed than what exists today