Software Fail Watch: 2015 in Review
14
© 2016 Tricentis. All rights reserved. © 2016 Tricentis All rights reserved. Software Fail Watch: 2015 in Review
-
Upload
chelsea-frischknecht -
Category
Data & Analytics
-
view
206 -
download
2
Transcript of Software Fail Watch: 2015 in Review
© 2016 Tricentis. All rights reserved.
© 2016 TricentisAll rights reserved.
Software Fail Watch:2015 in Review
© 2016 Tricentis. All rights reserved.
‘Software Fail Watch: 2015 in Review’ makes a strong case for how vital software testing is, and continues to be, despite those who proclaim the “death of testing“. As technology advances and enterprises race to develop the newest groundbreaking app, the demand for fast, efficient, and comprehensive testing only grows.
Comprehensive testing within short product sprints is not always possible however, making risk based testing the best way forward. Ultimately, testing harder isn’t the answer here - testing smarter is.
Ultimately this report serves as a crucial reminder of how easily a software bug can destroy enterprise value – making the role of testing as much about brand protection as it is about quality assurance.
Wolfgang PlatzFounder and CPO of Tricentis
Preface
© 2016 Tricentis. All rights reserved.
Table of contents
OverviewBugs per industryBugs in governmentBugs in transportationBugs in retailBugs in servicesBugs in financeBugs in entertainmentThe day of the bugTop 5 humorous storiesTop 5 shocking storiesConclusion
© 2016 Tricentis. All rights reserved.
483stories
4,365,767,463people affected
$428,455,568,503 assets affected
239companies
Tricentis, as the continuous testing company, takes a keen interest in software bugs. Most of the software bugs a company experiences neverbecome public knowledge, but those that do can have a severe impact. In an effort to track this phenomenon, we decided to catalogue the softwarebugs and failures that were big enough to gain media attention in 2015.
To do this we set news alerts for the phrases “software glitch“, “softwarebug“, and several common variants. All of the alerts were first sorted forrelevance and uniqueness, then placed into one of six broad industrycategories. Any statistics published in the news article were notedalongside each story, as well as external factors, such as whether the bug
originiated in a mobile app or posed the threat of physical injury. Stories that were covered by multiple sources and stayed in the news cycle for an extended period of time (days or weeks) were marked as having a high level of brand damage.
To the uninitiated, realizing the ubiquity of the software bug is an eyeopening experience. The process of collecting nearly a year‘s worth ofsoftware bugs revealed not only the sheer volume of bugs we encounteron a day-to-day basis, but also strong patterns in where and how theyoccur.
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
The software fail stories were sortedinto six broad industry categories asdisplayed on the left. Of thecategories, software bugs found in government were the mostcommon, but also generated theleast media coverage as a whole. Dare we say it seems that we expectbugs in government systems? Stories within the transportationand retail categories regularlyreceived more attention and generalconsumer ire, thoughtransportation earned the addeddistinction of posing the greatestpossibility of physical injury as a direct result of their bugs. The finance sector appeared uniquelyskilled at burying their softwarebugs; their stories tending to beparticularly vague on the facts. Several of the finance related storiesthat did come to light however, blewup to be among the biggest storiesof the year.
The latter half of the year saw a significant up-tick in software bugfrequency, which appear to matchthe travel and consumer purchasingtrends of the year.
20 1725
29 2722
53
88
7973
2922
Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sept. Oct. Nov. Dec.
39
48
70
89
92
145
0 20 40 60 80 100 120 140 160
Entertainment
Finance
Services
Retail
Transportation
Government
Software fails per industry
Software fails per month
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
Top 3 Government Bugs of 2015
DMV face recognition software denies twins driver’s licenses because the computer cannot tell them apart
Police department is forced to pay hackers data ransom in bitcoin after a malware attack
Software failure paralyzes health care services for over 10 days, denying care to over 10,850 patients
Is Agility as critical for Systems of Record as for Systems of Engagement?
Agility in both development and testing is just as critical for Systems of Record as it is for Systems of Engagement. Agility on enterprise level however, doesn’t necessarily imply continuous delivery – agility can be delivered at different speeds. Rather, it’s about applying the core essentials of agility (creating ownership, improving collaboration, enhancing transparency, increasing flexibility, amplifying feedback) to all software development regardless of the system’s character. Being a part of the “Digital Disruption”, businesses must leverage digital technologies to create new sources of customer value and increase operational agility. Agility in development, testing, and operations, therefore, is necessary to meet these needs. - Ingo Philipp
0 10 20 30 40 50 60 70 80 90
Healthcare
Emergency
Taxes
Dept. of transport
Justice
Subsidies
Bugs in government
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
Top 3 Travel Bugs of 2015
Fiat Chrysler scrambles to increase security after hackers hijack a Jeep Cherokee via bluetooth
Newly discovered Boeing 787 software bug capable of shutting down plane generator while in flight
360,000 passengers left stranded for over 4 hours after software glitch paralyzes Tokyo-Yokohama Tokyu trains
8,090,194cars recalled
11457 planes grounded
17 recorded deaths
Why is risk-based testing important?
As J. Bach has said, “Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous”. This implies that no matter how much you test, there is always an element of risk left over. You will simply never have enough time, budget, or resources to test everything as exhaustively as possible. Testing, therefore, must be selective. As a result, releasing a product is always directly related to the level of risk you are willing to run. Risk-based testing supports you in making that decision. It allows you to plan thorough testing in high risk areas, ordinary testing in medium risk ones, and light testing in low risk ones. It allows you to remove as much of the risk out of the product as early as possible, making testing one of the best investments you can make. - Ingo Philipp
Road
Air
Rail
Water
0 10 20 30 40 50 60
Bugs in transportation
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
33%mobile app bugs
2.5 Billionpeople affected
Top 3 Retail Bugs of 2015
A bug in Samsung Smart TVs shows Pepsi pop-up ads while watching live TV or home movies
Lenovo betrays customer security by pre-installing malicious Superfish adware on laptops
“Stagefright” bug capable of infecting almost 1 billion Android devices through a simple text message
The Growing Influence of Mobile Apps
With the increasing demand for instantaneous and easily accessible information, many analysts predict that tablets will surpass desktops and laptops within the next few years. With this in mind, the “Mobile First” strategy has become more and more popular as it focuses on first designing super lean applications for constrained mobile environments before developing that app out for use on desktop or laptop computers.
Mobile users however are very demanding in terms of quality –your app, and therefore your brand, will not recover from bad reviews in the app store. You have only one chance to get it right. Testing early in the development cycle therefore, is critical for releasing the great app you always wanted in the fastest possible time. – Georg Thurner
11
4 3
9
64
7 8
13
17
4 4
0
5
10
15
20
Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sept. Oct. Nov. Dec.
Bugs in retail
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
0 5 10 15 20 25
Other
Health
Communications
Utilities
Transport
Internet
Bugs in services
25% resulted in 4+ days of outage
1.65 yearsaverage life span of bug
1 known death
Top 3 Services Bugs of 2015
Telecom giant TalkTalk admits to having exposed 4 million customers’ personal information after hackers hold stolen data for ransom
The LightSail test mission placed on indefinite hold after the spacecraft software freezes while in orbit
A system “glitch” results in Yahoo Japan losing 2.58 million emails
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
Top 3 Finance Bugs of 2015
Bank of New York Mellon and U.S. mutual fund trading thrown into a panic after a software glitch generates inaccurate prices
275,000 HSBC customers find their payments delayed ahead of a holiday weekend due to a software bug
A ‘technical issue’ leads to the NY Stock Exchange grinding to a halt for the first time in 10 years
7.4 hoursavg. time lost
$2.5 billionbiggest single
loss
The Great Debate: Synthetic testdata vs. Production data
It’s common knowledge that the use of production data in testing offers only a small amount of risk coverage. Moreover, you are only testing for what has happened in the past and not what might cause defects as your system starts to evolve. This, in turn, leads to numerous situations where you “don’t know what you don’t know”. The fact is: your testing is only as intelligent as its test data. One way to improve testing then, is to improve the quality of test data. A synthetic test data approach offers a means to do this. It allows you to create the right data on demand to pair minimum test data volumes with maximum risk coverage. It also eliminates data security breaches and allows you to react proactively to test data consumption to ensure a repeatable and stable test execution at any time. With synthetic test data you can think about what you need in terms of quality, and not settle with what you have in terms of quantity. - Ingo Philipp
15
32
Bugs in finance
Stock Market Banking
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
Bugs in entertainment
1,369,075,725people affected
1/3mobile app bugs
Top 3 Entertainment Bugs of 2015:
WhatsApp bug puts 200 million users at risk of downloading malware
Twitter recants after blaming an Apple iOS update for losing 4 million users
“The Voice” tv show accidentally rejected all their stars thanks to an email glitch
0 2 4 6 8 10 12 14 16
Other
Video Games
Gambling
Film
Events
Social Media
© 2016 Tricentis. All rights reserved.
© 2016 Tricentis. All rights reserved.
08:26 – 09:47
4,900 flights grounded
United Airlines
“Automation issues“
TheDay of
the BugJuly 8, 2015
Trading suspended for 4 hours
New York Stock Exchange
“Configuration issues“
11:32 – 3:1011:18 – 15:05
2,500 residents lose power
In Washington D.C
Pepco | “Unknown cause“
Homepage replaced with
a 504 error page
Wall Street Journal | “Server issues“
11:34 – 12:50
“We are not big believers in coincidence“
FBI Director James Comey
© 2016 Tricentis. All rights reserved.
Top 5 Humorous Stories One lucky student found himself as
the Google webmaster for a full 60 seconds before Google realized they had just sold their domain name for $12. Sanmay Ved said he reported his purchase of the domain name to Google, who quickly caught and rectified the glitch. Full story here.
A bug in the Optus and Virgin cellphone networks gave thousandsof customers in Queensland, Australia an early wake up call whentheir phones mistakenly switched to daylight savings time. Customersfrom all over Brisbane reportedstarting their days – commuting to work, heading to the gym – only to find everything still closed. Full story here.
A bug in the Sims 4 “Get to Work” expansion pack released this year causes male characters to become pregnant after being abducted by aliens. The glitch has reportedly disappeared, much to the disappointment of some amused players. Full story here.
A “rapidly spreading” glitch in the New York City subway system reportedly caused subwy safety announcements to break off abruptly, leaving riders wondering what exactly they were supposed to “stand clear of” during their trip. Full story here.
Garmin satellite navigation offeredWestern Europeans a less-than-convenient detour when it beganrerouting everyone through thesleepy town of Preitenegg, Austria. Drivers across Europe and theUnited Kingdom reportedPreitenegg popping up as a suggested alternate route, even ifthe rerouting was just to avoid a traffic incident. Full story here.
© 2016 Tricentis. All rights reserved.
Top 5 Shocking Stories CareFusion recalled 15,905 Avea
ventilators after it became apparent that a defect in the ventilator could be deadly for patients. The ventilator was discovered as having a flaw that could cut off oxygen flow to patients, causing suffocation. CareFusion stated that they were not aware of any reported injuries. Full story here.
A convicted killer was released from prison 2 years into his 40 year conviction after a software glitch allowed for his release. The error in the system was noticed 5 days after the prisoner’s release, who has yet to be recaptured. Full story here.
The government of Queensland, Australia began investigating after an update to their OneSchoolsystem failed, preventing 644 suspected child abuse cases from being reported to the police. The glitch has reportedly been fixed, along with the cases being been recovered and delivered into the proper hands. Full story here.
Air traffic controllers at the Melbourne airport, Australia, lost the locations of at least 9 planes on the tarmac after their integrated tower automation suite failed. As it was reportedly a foggy night with no visibility, the planes were in danger of colliding on the tarmac –an incident which in the past killed
538 passengers and was regarded as “one of the worst accidents in aviation history”. Full story here.
A prepaid debit card called RushCards, marketed specifically to low-income families, hit a massive software bug, keeping hundreds of thousands of families from accessing their money for over a week. The story, one of the biggest of the year, brought the business of prepaid cards under heavy scrutiny in the United States. Full story here.
© 2016 Tricentis. All rights reserved.
History is full of unsinkable ships.
Regardless of how exceptional we are at our collective jobs, itremains true that no enterprise is immune to a software bug. Software bugs share a similar evolutionary pattern to their insectcounterparts. As the code changes, integrations are added, functions are introduced, and legacy systems migrated, the bugsfind a way to adapt and evolve. As engineers and developerscreate the next world-changing program (and the next, and thenext, and the next) they will always discover a new species ofbugs.
You could say that testers have the best job security possible: software quality testing will always be necessary for riskprevention. Just as with evolution however, risk is not a passive force. If we do not actively attack the risk of a software failure, the
risk will eventually attack us. The importance of software qualityassurance nevertheless, goes far beyond the mere avoidance ofproblems.
An enterprises‘ app is as much an extension of their brand astheir choice of logo. In the end, software quality assurance isabout making sure your enterprise doesn‘t end up on a list like this - or become synonymous with a meltdown of titanicproportions.
To view the full list of software bugs we collected in 2015, clickhere.
