Avoiding Technical Bankruptcy
-
Upload
markuskobler -
Category
Business
-
view
586 -
download
0
description
Transcript of Avoiding Technical Bankruptcy
![Page 1: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/1.jpg)
A story of reducing technical debt
Founder + Digital Craftsman
Markus Kobler
Avoiding Technical Bankruptcy
![Page 2: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/2.jpg)
Overview
The technical debt build up
Five Techniques that Worked
1) Getting a Canary
2) Rejecting Scrum for Kanban
3) Planning Poker
4) Automating Deployments with ‘Munge’
5) Sharing Ownership with Fez & Pirate Hats!
Dealing with Collapse : A Success Story
![Page 3: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/3.jpg)
Taglab did a bit of everything web related...
Strategy
Transactional Sites
Design
Email(Even built/hosted our own campaign system!?!)
BannersMicro-Sites
Brochure-ware
Social Media
Hosting
Ad Campaigns
24/7/365 Support J2EE Web Apps
Messaging Middleware
Workflow Systems
![Page 4: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/4.jpg)
Notable Clients Included
Betfair
Times Online
Starbucks
The Economist
Disney Channel
Turner
Global Switch
Channel Five
NBC UniversalSkandia
STA Travel
Yell Online
The Spectator
British Gas
Avis
One Water
Liverpool Victoria
![Page 5: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/5.jpg)
My First three years at Taglab had been exciting
- Six fold growth
- Wide range of big and small projects
- Rounded skillset from architectural & development to system administration
However after the initial exciting growth the company started to hit a celling
- Common in small business, 80% fail in their first 5 years*
- J2EE was loosing its perceived competitive advantage
*The E-Myth 2001
![Page 6: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/6.jpg)
Over time technical debt started to build up
Its around this time Taglab asked me to rejoin the company as CTO...
![Page 7: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/7.jpg)
Supporting 40-50 live production web apps
- 100+ Environments in total (UAT, Integration, etc)
- Key required 24/7/365 SLA’s
Hosted Apps had 45+ external integration points
- RESTful & SOAP API’s
- But many XML(sometimes)/RPC
Designed by Architects with Capital A...
...but built by developers (with a lowercase d)
Pain stemmed from multiple points
![Page 8: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/8.jpg)
Pain stemmed from multiple points
Legacy and modern Codebase
- 7+ year old code, Singleton galore with poor test coverage
- Some code even needing decompiling from bytecode...
- To modern lightweight clean GWT, Hibernate, Spring
Hosting Mix
- Mix of our own rack in Docklands to clients hardware and the cloud
- Although our rack provided many benefits some hardware dated back 5-7 years
![Page 9: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/9.jpg)
Projects were increasingly Death Marches
![Page 10: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/10.jpg)
Death March TypesH
appi
ness
Chances of Success
Kamikaze Mission Impossible
UglySuicideTeam are sacrificial
lambs at the slaughter for successful delivery
Team dreams of fame, glory and riches ‘if ’
they succeed
Projects and Team are doomed.Only alternative is to be fired
Projects are doomed, but everyone agrees it will be
glorious failure!
Death March - Edward Yourdon
![Page 11: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/11.jpg)
At its core this is a story aboutH
appi
ness
Chances of Success
Positively and creatively changing
peoples perspective to improve
delivery
![Page 12: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/12.jpg)
Problem 1 : Notification Hell
Nagios monitoring was already in place
- but needed updating
Were getting 4-5 alerts a day!
- Regular Database/Hardware failures
- Deployment, Code Quality problems
- Subtle networking issues (ISP or self inflicted)
You dreaded your phone
Make things worse, we needed more checks!
- CI still needed + much better test coverage
![Page 13: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/13.jpg)
Getting a Canary (or Nabaztag)
Flickr: julianbleecker
![Page 14: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/14.jpg)
Flickr: krissatmontreal
Could differentiate personal from work txt messages
Problems started feeling a little less ‘catastrophic’
The office knew when was not right time ask for a quote on ‘one more copy change...’
Over time major alerts dropped to 1 every few weeks
Getting a Canary (or Nabaztag)
![Page 15: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/15.jpg)
Problem 2 : Juggling Delivery + Support
We where hungry for work, so new projects varied wildly in size and complexity
- Some were multi year builds
- Others ‘OMG we need it done by the weekend’
Support’s unpredictability impacted new build work
- More common in-practice with teams impacted by 2nd/3rd line support issues
Mixing both functions can be efficient and satisfying
![Page 16: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/16.jpg)
'Known Knowns' - requirements captured in the specification
'Known Unknowns' - requirements flagged as ‘assumptions’
'Unknown Knowns' - requirements the client had not yet told you about
'Unknowns Unknowns' - requirements neither you or client knew about
Successful Delivery is about Accepting UncertaintyFlickr : ooocha
![Page 17: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/17.jpg)
Sounds like Agile Right?
Many competitors and clients were touting SCRUM
“Don’t you know the difference between the Chicken and Pig!”
But clients still found it hard to sign off budgets
“Need to know what is being delivered before I signing off costs!”
Work/Budget would often not fit neatly into an iteration(s)
How would unpredictable support be incorporated?
- Having a ‘batman’ would have been wasteful or not enough
We needed something less rigid
![Page 18: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/18.jpg)
Rejecting Scrum for Kanban
To-Do Doing Done
Task TaskTask Task
Task
Task
TaskTask
Task
Task
TaskTask
Task
So we started simply
![Page 19: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/19.jpg)
Rejecting Scrum for Kanban
To-Do Doing
Task TaskTask
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Added columns that made sense to us
UAT CAT Live
Task
TaskTask
TaskTask
![Page 20: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/20.jpg)
Rejecting Scrum for Kanban
Backlog In Progress
Task Task TaskTask
Task
Task
TaskTask
Task
Task
Task
Task
Task
Added columns that matched our internal processes
UAT CAT Live
Task
Task
Task
Task
Task
Planned DevInternal
Test
Task
Task
Task
Task
Task
Task
TaskTask
![Page 21: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/21.jpg)
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task TaskTask
Task
Task
Task
Task
Added swim lanes
UAT CAT Live
Task
Task
Task
Task
Planned DevInternal
Test
Task
Task
Task
Task
Task
TaskTask
Majo
r Proj
ect 1
Majo
r Proj
ect 2
Other P
rojec
ts
Intern
al
TaskTask
Task
Task
![Page 22: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/22.jpg)
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task TaskTask
Task
Task
Task
Task
Introduced Work In Progress (WIP) limits and SLA’s
UAT CAT Live
Task
Task
Task
Task
Planned Dev InternalTest
Task
Task
Task
Task
Task
TaskTask
Majo
r Proj
ect 1
Majo
r Proj
ect 2
Other P
rojec
ts
Intern
al
TaskTask
Task
Task
5 Day SLA
15 4 12
![Page 23: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/23.jpg)
IssueImprov-ement
Improv-ement
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task TaskTask
Task
Task
Task
Task
Visually distinguish types of tasks
UAT CAT Live
Task
Task
Task
Task
Planned Dev InternalTest
Task
Task
Task
Task
Task
TaskTask
Majo
r Proj
ect 1
Majo
r Proj
ect 2
Other P
rojec
ts
Intern
al
TaskTask
Task
Task
5 Day SLA
15 4 12
Improv-ement
Improv-ement
Issue
Issue
![Page 24: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/24.jpg)
Rejecting Scrum for Kanban
Work is continuously pulled through system
Visually provides a Micro & Macro view of work
Support issues easily absorbed
WIP and SLA’s provided key indicators of quality and resource planning
![Page 25: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/25.jpg)
Problem 3 : Estimation + Tracking Velocity
Flickr : kozumel
![Page 26: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/26.jpg)
Planning Poker
Cards going up in value
User Story discussed
Once happy, everyone estimates in private
Reveal at same time
Hi & Lo estimates discuss differences
Repeat estimate again if no consensus
Flickr: lejoe
![Page 27: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/27.jpg)
Shared ownership of projects problem solving
Could start analytically looking back at projects past progress
Enabled plotting projects into future based on previous tasks/estimates
As a PM you can often get team to develop simplest solution and feel good about it!
Planning Poker
![Page 28: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/28.jpg)
Problem 4 : Deployment
Deployment should be the last problem developers need to worry about
- Needed to re-instil confidence back in the team
Had to work well with Designers and Developers
Needed to adapt to different environments
- Our Rack, Client’s hardware, The cloud
Reducing time to deploy would reduce our costs
- Worst cases deployments took 5 hours!
![Page 29: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/29.jpg)
Designers Needed Something Simple
Flickr : thunderchild5
![Page 30: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/30.jpg)
Developers Demanded More Rigour
Flickr : shutupyourface
![Page 31: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/31.jpg)
Automating Deployments with ‘Munge’
Mix of automation and strict project conventions
- Minor changes could be released in seconds
- Major releases, tested and deployed in 5-20mins (depending on changes)
Clean build environment and version controlled released meant predictable builds and rollbacks
Used live environment metrics to determine deployed hosts
![Page 32: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/32.jpg)
Problem 5 : Shared Ownership of Prod Issues
Whole team needed to share ownership of dealing with production issues
Most problems only needed one person looking at it to begin with
Ideally ownership should be visual to rest of team
![Page 33: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/33.jpg)
So we got some hats of responsibility
Flickr : doctorow
![Page 34: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/34.jpg)
So we got some hats of responsibility
Flickr : c_r_i_s
![Page 35: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/35.jpg)
So we got some hats of responsibility
Flickr : striatic
![Page 36: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/36.jpg)
So we got some hats of responsibility
Flickr : blackcountrymuseums
![Page 37: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/37.jpg)
So we got some hats of responsibility
![Page 38: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/38.jpg)
Finally a Success Story
Flickr : dybarber
![Page 39: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/39.jpg)
Finally a success story
2009, Setanta Sports was on verge of collapse
- A key client dependant on their sports coverage
- Unsure if someone would step in with finance
- Or someone else would buy the rights
Particularly complex CRM/Sales intergration
- Estimated >5 weeks worth of effort
- Failure to deliver by immovable deadline was to be very damaging for all parties involved
Sign off received 3 days before changes were required
![Page 40: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/40.jpg)
Success Story
Work broken into 5 interleaved dev streams
Hit first key 3 day deadline
- 4 major releases followed over next 7 days
- More followed in the subsequent weeks/months
Sales in the days that followed dwarfed anything that came before with 0 downtime
One of the most satisfying projects have been lucky enough to be involved in
All thanks to hard prior work put in by the team
![Page 41: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/41.jpg)
Final Thoughts
Our biggest challenge was turning bleak situation into positive opportunities
- Some times you need a creative solution
- Continuously take educated technical risks
Software delivery is an art-form
- but backed by scientific approaches
Embrace change and unpredictability
- but always keep a clear guiding vision
An informative workspace is key
- Although is largely about ‘Psychological Manipulation’
![Page 42: Avoiding Technical Bankruptcy](https://reader033.fdocuments.in/reader033/viewer/2022052823/5556b4bdd8b42a9c798b5549/html5/thumbnails/42.jpg)
Credits
Death March (Yourdon Press Series) - http://www.amazon.co.uk/Death-March-Yourdon-Press-Edward/dp/013143635X/ref=sr_1_1?ie=UTF8&qid=1290856412&sr=8-1
Nabaztag - julianbleecker - http://www.flickr.com/photos/julianbleecker/156303245/
Nabaztag - krissatmontreal - http://www.flickr.com/photos/krissatmontreal/3027612184/
061108-F-5586B-137 - ooocha - http://www.flickr.com/photos/ooocha/3051551020/
Poker Phase - kozumel - http://www.flickr.com/photos/kozumel/4063207100/
Planning poker! I've a straight flush! - lejoe - http://www.flickr.com/photos/lejoe/4553607341/
The Big Easy - thunderchild5 - http://www.flickr.com/photos/thunderchild5/2397161370/
Factory - Robotic Arms - shutupyourface - http://www.flickr.com/photos/shutupyourface/105779625/
Me in Ada's pirate hat, monorail queue, Walt Disney World, Orlando, Florida - doctorow - http://www.flickr.com/photos/doctorow/2137686/
Fez / 87.365 - c_r_i_s - http://www.flickr.com/photos/c_r_i_s/2852508138/
past tense - striatic - http://www.flickr.com/photos/striatic/2418853122/
1940s flying helmet , LC1995_15_1 - blackcountrymuseums - http://www.flickr.com/photos/blackcountrymuseums/4970602229/
Fremantle Bridge Collapse, 22 July 1926 - dybarber - http://www.flickr.com/photos/dybarber/3461384691/