ContentsCover
Cast
Prologue
Chapter1
Chapter2
Chapter3
Chapter4
Chapter5
Chapter6
Chapter7
Chapter8
Chapter9
Chapter10
Chapter11
Chapter12
Chapter13
Chapter14
Chapter15
Chapter16
Chapter17
Chapter18
Chapter19
Chapter20
Chapter21
Chapter22
Chapter23
kindle:embed:0002?mime=image/jpg
Chapter24
Chapter25
Chapter26
Chapter27
Chapter28
Chapter29
Chapter30
Chapter31
Chapter32
Chapter33
Chapter34
Chapter35
Resources
Acknowledgements
Copyright
PartsUnlimited
PartsUnlimited:BusinessExecutives
SteveMasters,CEO,actingCIO
DickLandry,CFO
SarahMoulton,SVPofRetailOperations
MaggieLee,SeniorDirectorofRetailProgramManagement
BillPalmer,VPofITOperations,formerDirectorofMidrangeTechnologyOperations
WesDavis,DirectorofDistributedTechnologyOperations
BrentGeller,LeadEngineer
PattyMcKee,DirectorofITServiceSupport
JohnPesche,ChiefInformationSecurityOfficer(CISO)
ChrisAllers,VPofApplicationDevelopment
PartsUnlimited:Board
BobStrauss,LeadDirector,formerChairman,formerCEO
ErikReid,BoardCandidate
NancyMailer,ChiefAuditExecutive
FORIMMEDIATERELEASEFriday,August29
Company:PartsUnlimited(PAUD)
Rating:SELL
PriceTarget:$8(current$13)
Effectiveimmediately,PartsUnlimitedCEO,SteveMasters,issteppingdownfromhisroleaschairmanaftereightyearsofholdingthatposition.BoardDirectorBobStrauss,whoservedascompanychairmanandCEOtwodecadesago,isreturningfromretirementtoassumetheroleofchairman.
PartsUnlimitedstockhastumbled19percentinthelast30daysunderheavytrading,down52percentfromitspeakthreeyearsago.Thecompanycontinuestobeoutmaneuveredbyitsarchrival,famousforitsabilitytoanticipateandinstantlyreacttocustomerneeds.PartsUnlimitednowtrailsthecompetitioninsalesgrowth,inventoryturnsandprofitability.
Thecompanyhaslongpromisedthatits“Phoenix”programwillrestoreprofitabilityandclosethegapbytightlyintegratingitsretailingande-commercechannels.Alreadyyearslate,manyexpectthecompanytoannounceanotherprogramdelayinitsanalystearningscallnextmonth.
WebelievethatinstitutionalinvestorssuchasWayne-YokohamapressuredBobtoreconfiguretheboardasthefirstofmanyactionstorighttheshipinElkhartGrove.Agrowingnumberofinvestorsarepushingformoresignificantleadershipchangesandstrategicoptions,suchassplittingupthecompany.
DespiteMasters’pastachievementsthattransformedPartsUnlimitedintooneofthetopautomotivepartsmanufacturersandretailers,webelievesplittingupthechairmanandCEOrolesislongoverdue.PartsUnlimitedneedsfreshleadership,eitherfromtheoutsideorfromwithin.WebelieveSarahMoulton,SVPofRetailOperations,andarisingstaratthecompany,couldjustbewhatthecompanyneeds.
Accordingtooursources,theboardhasgivenStraussand
Masterssixmonthstomakedramaticimprovements.Iftheycan’tpullthisoff,expectmorechangesandturbulenttimes.
—KellyLawrence,ChiefIndustryAnalyst,NestorMeyers
###
Chapter1•Tuesday,September2
“BillPalmerhere,”Isay,answeringmycellphoneonthefirstring.
I’mlate,soI’mdrivingtenmilesperhouroverthespeedlimit,insteadofmyusualfive.Ispentthemorningatthedoctor’sofficewithmythree-year-oldson,tryingtokeeptheothertoddlersfromcoughingonus,constantlybeinginterruptedbymyvibratingphone.
Theproblemofthedayisintermittentnetworkoutages.AstheDirectorofMidrangeTechnologyOperations,I’mresponsiblefortheavailabilityandsmoothfunctioningofarelativelysmallITgroupatPartsUnlimited,a$4billionperyearmanufacturingandretailcompanybasedinElkhartGrove.
EveninthetechnologybackwatersI’vechosentomakemyturf,Ineedtotracknetworkissuesclosely.Becausetheseissuesdisrupttheservicesmygroupprovides,peoplewillblametheoutagesonme.
“Hi,Bill.ThisisLauraBeck,fromHumanResources.”She’snotthepersonIusuallydealwithfromHR,buthernameandvoicesoundfamiliar…
Holycrap.ItrynottoswearoutloudwhenIrememberwhosheis.Fromthemonthlycompanymeetings.She’stheVPinchargeofHR.
“Goodmorning,Laura,”Isaywithforcedcheer.“WhatcanIdoforyou?”
Sheresponds,“Whenwillyoubeintheoffice?I’dliketomeetassoonaspossible.”
Ihatevaguerequeststomeet.IonlydothatwhenI’mtryingtoscheduleatimetochewsomeoneout.Orfirethem.
Wait.IsLauracallingbecausesomeonewantstofireme?WasthereanoutageIdidn’trespondtoquicklyenough?AsanITOperationsguy,thecareer-endingoutageisthejokemypeersandItelloneanotherdaily.
Weagreetomeetatherdeskinahalfhour,butwhenshedoesn’tshareanymoredetails,Isayinmymostcajolingvoice,“Laura,what’sthisallabout?Isthereaprobleminmygroup?OramItheoneintrouble?”Ilaughextraloudly,soshehearsitoverthephone.
“No,it’snothinglikethat,”shesaysbreezily.“Youcouldevensaythisisgoodnews.Thanks,Bill.”
Whenshehangsup,Itrytothinkofwhatgoodnewswouldevenlooklikethesedays.WhenIcan’t,Iturntheradiobackonandimmediatelyhearacommercialfromourlargestretailingcompetitor.They’retalkingabouttheirunparalleledcustomerserviceandabreathtakingnewofferingthatallowspeopletocustomizetheircarswiththeirfriendsonline.
Theadisbrilliant.I’dusetheserviceinasecond,ifIweren’tsuchaloyalcompanyman.Howdotheykeepbringingsuchincrediblenewcapabilitiestomarketwhileweremainstuckinthemud?
Iturntheradiooff.Despiteallourhardworkandlatenights,thecompetitionkeepsleapfroggingus.WhenourMarketingpeoplehearthisad,they’llgoballistic.Becausethey’relikelyartormusicmajors,notpeoplewithatechnologybackground,they’llpubliclypromisetheimpossibleandITwillhavetofigureouthowtodeliver.
Eachyear,itgetsharder.Wehavetodomorewithless,tosimultaneouslymaintaincompetitivenessandreducecosts.
Somedays,Ithinkthatitcan’tbedone.MaybeIspenttoomuchtimeasasergeantintheMarines.Youlearnthatyouargueyourcaseasbestasyoucanwithyourofficer,butsometimesyouhavetosay,“Yes,sir,”andthengotakethathill.
Ipullintotheparkinglot.Threeyearsago,findinganemptyparkingspotwasimpossible.Now,afterallthelayoffs,parkingisrarelyaproblem.
WhenIwalkintoBuilding5whereLauraandherstaffreside,Iimmediatelynoticehownicelyfurnisheditis.Icansmellthenewcarpetingandthere’sevenclassywoodpanelingonthewalls.Suddenly,thepaintandcarpetinmybuildingseemdecadesoverdueforreplacement.
That’sIT’slotinlife.Atleastwe’renotinadingy,dimlylitdankbasement,likeintheBritishTVshow,TheITCrowd.
WhenIgettoLaura’soffice,shelooksupandsmiles.“Goodseeingyouagain,Bill.”Sheextendsherhand,whichIshake.“HaveaseatwhileIseewhetherSteveMastersisavailabletomeet.”
SteveMasters?OurCEO?
ShepicksupanddialsherphonewhileIsitdown,lookingaround.ThelasttimeIwasherewasacoupleofyearsagowhenHRnotifiedusthatweneededtodedicatearoomfornursingmothers.Wewerecriticallyshortofofficeandmeetingspace,andwehadbigprojectdeadlineslooming.
Wemerelywantedtouseaconferenceroominadifferentbuilding.However,Wesmadeitsoundlikewewereabunchof1950sMadMenNeanderthals.Shortlyafterward,wewerebothsummonedhereforahalfdayofpoliticalrehabilitationandsensitivitytraining.Thanks,Wes.
Amongotherthings,Wesisinchargeofthenetworks,whichiswhyItracknetworkoutagessoclosely.
Laurathanksthepersonontheotherendofthephoneandturnsbacktome.“Thanksforcomingdownonshortnotice.Howisyourfamilydoingthesedays?”sheasks.
Mybrowfurrows.IfIwantedtochitchat,therearemanypeopleI’drathertalktothansomeoneinHR.Iforcemyselftobanteraboutourfamiliesandkids,tryingnottothinkaboutmyotherpressingcommitments.EventuallyIsay,withoutmuchgrace,“So,whatcanIdoforyouthismorning?”
“Ofcourse.”Shepauses,andthensays,“Effectiveasofthismorning,LukeandDamonarenolongerwiththecompany.Thiswentallthewaytothetop,withStevegettinginvolved.He’schosenyoutobetheVPofITOperations.”
Shesmilesbroadly,holdingoutherhandagain,“You’reournewestVPinthecompany,Bill.Ithinksomecongratulationsareinorder?”
Holycrap.Inumblyshakeherhand.
No,no,no.ThelastthingIwantisa“promotion.”
LukewasourCIO,orChiefInformationOfficer.Damonworkedforhimandwasmyboss,inchargeofITOperationsacrosstheentirecompany.Bothgone,justlikethat.
Ididn’tseethiscoming.Therewasn’tanychatteronthesubspaceradio.Nothing.
Forthelastdecade,likeclockwork,newCIOswouldcomeandgoeverytwoyears.Theystayjustlongenoughtounderstandtheacronyms,learnwherethebathroomsare,implementabunchofprogramsandinitiativestoupsettheapplecart,andthenthey’regone.
CIOstandsfor“CareerIsOver.”AndVPsofITOperationsdon’tlastmuchlonger.
I’vefiguredoutthatthetricktoalongcareerinITOperationsmanagementistogetenoughsenioritytogetgoodthingsdonebuttokeepyourheadlowenoughtoavoidthepoliticalbattlesthatmakeyouinherently
vulnerable.IhaveabsolutelynointerestinbecomingoneoftheVPswhojustgiveeachotherPowerPointsalldaylong.
Fishingformoreinformation,Ijoke,“Twoexecutivesleavingatthesametime?Weretheystealingmoneyfromthestoreslateatnight?”
Shelaughs,butquicklyreturnstoherHR-traineddeadpan,“Theybothchosetopursueotherinterests.Morethanthat,you’llhavetofindoutfromthem.”
Asthesayinggoes,ifyourcolleaguetellsyouthey’vedecidedtoquit,itwasvoluntary.Butwhensomeoneelsetellsyouthey’vedecidedtoquit,itwasmandatory.
Ergo,mybossandhisbosswerejustwhacked.
ThisisexactlywhyIdon’twantapromotion.I’mextremelyproudoftheteamI’vebuiltoverthelasttenyears.It’snotthelargestgroup,butwe’rethemostorganizedanddependable,byfar.EspeciallycomparedtoWes.
IgroanatthethoughtofmanagingWes.Hedoesn’tmanageateam—he’sbarelyonestepaheadofachaoticmob.
AsIbreakoutinacoldsweat,IknowIwillneveracceptthispromotion.
Allthistime,Laurahasbeentalking,andIhaven’theardasingleword.“—andsowe’llobviouslyneedtotalkabouthowwe’regoingtoannouncethistransition.AndStevewantstoseeyouassoonaspossible.”
“Look,thanksfortheopportunity.I’mhonored.ButIdon’twantthisrole.WhywouldI?Ilovemycurrentjob,andtherearetonsofimportantthingsthatstillneedtobedone.”
“Idon’tthinkthisisoptional,”shesays,lookingsympathetic.“ThiscamestraightfromSteve.Hechoseyoupersonally,soyou’llhavetotalkwithhim.”
Istandupandreiteratefirmly,“No,really.Thanksforthinkingofme,butI’vealreadygotagreatjob.Goodluckfindingsomeoneelse.”
Minuteslater,LauraiswalkingmetoBuilding2,thetallestbuildingoncampus.I’mangryatmyselfforgettingsuckedintothisinsanity.
IfIrunnow,I’mprettysureshewouldn’tbeabletocatchme,butthenwhat?StevewouldjustsendawholesquadofHRgoonstofetchme.
Idon’tsayanything,definitelynotfeelinglikesmalltalkanymore.Lauradoesn’tseemtocare,walkingbrisklybesideme,noseburiedinherphone,
occasionallygesturingdirections.
ShefindsSteve’sofficewithouteverlookingup,obviouslyhavingmadethiswalkmanytimesbefore.
Thisflooriswarmandinviting,furnishedjustlikeitwasinthe1920s,whenthebuildingwasconstructed.Withdarkhardwoodfloorsandstainedglasswindows,it’sfromanerawheneveryoneworesuitsandsmokedcigarsintheiroffices.Thecompanywasboomingthen—PartsUnlimitedmadevariouswidgetsinsidealmosteverymakeofautomobile,whenhorseswerebeingvanquishedfromdailylife.
Stevehasacorneroffice,whereano-nonsensewomaniskeepingguard.She’saboutforty,radiatingcheerfulnessandasenseoforganizationandorder.Herdeskistidy,withPost-itnoteseverywhereonthewall.There’sacoffeemugwiththewords“Don’tMessWithStacy”byherkeyboard.
“Hi,Laura,”shesays,lookingupfromhercomputer.“Busyday,huh?So,thisisBill?”
“Yep.Intheflesh,”Laurareplies,smiling.
Tomeshesays,“StacykeepsSteveinline.You’llgrowtoknowherwell,Isuspect.YouandIcanfinishuplater.”Thensheleaves.
Stacysmilesatme.“Pleasure.I’veheardalotaboutyoualready.Steveisexpectingyou.”Shepointstohisdoor.
Iimmediatelylikeher.AndIthinkaboutwhatI’vejustlearned.It’sbeenabusydayforLaura.StacyandLauraareonveryfamiliarterms.StevehasHRonspeeddial.Apparently,peoplewhoworkforStevedon’tlastlong.
Great.
Walkingin,I’malittlesurprisedtofindSteve’sofficelooksjustlikeLaura’s.It’sthesamesizeasmyboss’office—orrather,myex-boss’office—andpotentiallymynewofficeifI’mstupid,whichIamnot.
MaybeIwasexpectingPersianrugs,waterfountains,andlargesculptureseverywhere.Instead,therearephotosonthewallofasmallpropellerairplane,hissmilingfamily,and,tomysurprise,oneofhiminaUSArmyuniformonarunwaysomewheretropical.Inotewithsurprisetheinsigniavisibleonhislapels.
So,Stevewasamajor.
Heissittingbehindhisdesk,scrutinizingwhatappeartobepaperspreadsheets.There’salaptopopenbehindhim,displayingabrowserfullof
stockgraphs.
“Bill,goodtoseeyouagain,”hesays,standingandshakingmyhand.“It’sbeenalongtime.Aboutfiveyears,right?Itwasafteryoupulledoffthatamazingprojecttointegrateoneofthemanufacturingacquisitions.Itrustlifehasbeentreatingyouwell?”
I’msurprisedandabitflatteredthatherememberedourbriefinteraction,especiallywhenitwassolongago.Ismileinreturn,saying,“Yes,verywell,thankyou.I’mamazedyouremembersomethingsofarback.”
“Youthinkwegiveoutawardslikethattojustanyone?”hesaysearnestly.“Thatwasanimportantproject.Tomakethatacquisitionpayoff,weneededtonailit,whichyouandyourteamdidsuperbly.
“I’msureLaurahastoldyouabitabouttheorganizationalchangesI’vemade.YouknowLukeandDamonarenolongerwiththecompany.IintendtofilltheCIOpositioneventually,butinthemeantime,allofITwillreporttome.”
Hecontinues,briskandbusinesslike,“However,withDamon’sdeparture,IhaveanorganizationalholeIneedtofill.Basedonourresearch,you’reclearlythebestcandidatetotakeoverasVPofITOperations.”
Asifhejustremembered,hesays,“YouwereaMarine.Whenandwhere?”
Iannounceautomatically,“222ndMarineExpeditionaryUnit.Sergeant.Iwasinforsixyearsbutneversawcombat.”
RememberinghowIjoinedtheMarinesasacockyeighteen-year-old,Isaywithasmallsmile,“TheCorpsreallystraightenedmeout—Iowethemalot,butIsurehopeneitherofmysonsjoinunderthesameconditionsIdid.”
“Ibet,”Stevelaughs.“IwasintheArmyforeightyearsmyself,slightlylongerthanIwasobligatedto.ButIdidn’tmind.ROTCwastheonlywayIcouldpayforcollege,andtheytreatedmewell.”
Headds,“Theydidn’tcoddleusliketheydidyouMarines,butIstillcan’tcomplain.”
Ilaugh,findingmyselflikinghim.Thisisthelongestinteractionwe’vehad.Isuddenlywonderifthisiswhatpoliticiansarelike.
Itrytostayfocusedonwhyhesummonedmehere:He’sgoingtoaskmetoundertakesomekamikazemission.
“Here’sthesituation,”hesays,motioningmetohaveaseatathis
conferencetable.“AsI’msureyou’reaware,wemustregainprofitability.Todothat,weneedtoincreaseourmarketshareandaverageordersizes.Ourretailcompetitorsarekickingourass.Thewholeworldknowsthis,whichiswhyourstockpriceishalfwhatitwasthreeyearsago.”
Hecontinues,“ProjectPhoenixisessentialtoclosingthegapwiththecompetition,sowecanfinallydowhatthecompetitionhasbeendoingforyears.Customersneedtobeabletobuyfromusfromwherevertheywant,whetherit’sontheInternetorinourretailstores.Otherwise,we’llsoonhavenocustomers,atall.”
Inod.Imightbeinthetechnologybackwaters,butmyteamhasbeeninvolvedwithPhoenixforyears.Everyoneknowshowimportantitis.
“We’reyearslatedelivering,”hecontinues.“OurinvestorsandWallStreetarehowling.Andnow,myboardislosingconfidenceinourabilitytohitourcommitments.
“I’llbeblunt,”hesays.“Thewaythingsaregoing,I’llbeoutofajobinsixmonths.Asoflastweek,BobStrauss,myoldboss,isthenewchairmanofthecompany.There’savocalgroupofshareholderstryingtosplitupthecompany,andIdon’tknowhowmuchlongerwecanfendthemoff.What’satstakehereisnotjustmyjob,butthenearlyfourthousandemployeeswhoworkhereatPartsUnlimited.”
Suddenly,StevelooksmucholderthantheearlyfiftiesIhadguessedhimtobe.Lookingrightatme,hesays,“AsactingCIO,ChrisAllers,ourVPofApplicationDevelopment,willreporttome.Andsowillyou.”
Hestandsupandstartstopace,“Ineedyoutokeepallthethingsthataresupposedtobeup,well,up.Ineedsomeonereliable,whoisn’tafraidtotellmebadnews.Aboveall,IneedsomeoneIcantrusttodotherightthing.Thatintegrationprojecthadmanychallenges,butyoualwayskeptacoolhead.You’vebuiltareputationassomeonewhoisdependable,pragmatic,andwillingtosaywhatyoureallythink.”
He’sbeencandidwithme,soIreplywiththesame.“Sir,withallduerespect,itseemsverydifficultforseniorITleadershiptosucceedhere.Anyrequestforbudgetorstaffisalwaysshotdown,andexecutivesarereplacedsoquickly,someneverevengetachancetofullyunpack.”
Withfinality,Isay,“MidrangeOperationsiscriticaltogettingPhoenixdone,too.Ineedtostaytheretoseethosethingsthroughtocompletion.Iappreciateyouthinkingofme,butIcan’taccept.However,IpromiseI’llkeepmyeyesopenforanygoodcandidates.”
Stevelooksatmeappraisingly,hisexpressionsurprisinglygrave.“We’vehadtocutbudgetsacrosstheentirecompany.Thatedictcamestraightfrommyboard.Myhandsweretied.Iwon’tmakepromisesIcan’tkeep,butIcanpromiseyouI’lldowhateverittakestosupportyouandyourmission.
“Bill,Iknowyoudidn’taskforthisjob,butthecompany’ssurvivalisatstakehere.Ineedyoutohelpmesavethisgreatcompany.CanIcountonyou?”
Oh,forchrissakes.
BeforeIcanpolitelydeclineagain,Isuddenlyhearmyselfsaying,“Yes,sir,youcancountonme.”
Ipanic,realizingthatStevesomehowusedsomeJedimindtrickonme.Iforcemyselftostoptalking,beforeImakemoredumbpromises.
“Congratulations,”Stevesays,standingupandshakingmyhandfirmly.Heclaspsmyshoulder.“Iknewyou’ddotherightthing.Onbehalfoftheentireexecutiveteam,we’regratefulforyousteppingup.”
Ilookathishandgraspingmine,wonderingifIcanbackpeddlemywayout.
Notachanceinhell,Idecide.
Swearingtomyself,Isay,“I’lldomybest,sir.Andcouldyouatleastexplainwhynoonewhoacceptsthispositionlastsverylong?Whatdoyouwantmostfromme?Andwhatdon’tyouwant?”
Witharesignedhalfsmile,Iadd,“IfIfail,I’lltrytomakesureit’sinanewandnovelway.”
“Ilikethat!”Stevesays,laughingloudly.“WhatIwantisforITtokeepthelightson.Itshouldbelikeusingthetoilet.Iusethetoiletand,hell,Idon’teverworryaboutitnotworking.WhatIdon’twantistohavethetoiletsbackupandfloodtheentirebuilding.”Hesmilesbroadlyathisownjoke.
Great.Inhismind,I’maglorifiedjanitor.
Hecontinues,“YouhaveareputationofrunningthetightestshipintheITorganization.SoI’mgivingyoutheentirefleet.Iexpectyoutomakethemallrunthesameway.
“IneedChrisfocusedonPhoenixexecution.AnythinginyourareaofresponsibilitythattakesfocusawayfromPhoenixisunacceptable.ThatappliesnotjusttoyouandChris,buteveryoneelseinthiscompany.Isthat
clear?”
“Absolutely,”Isay,nodding.“YouwanttheITsystemstobereliableandavailable,andforthebusinesstobeabletodependuponthem.YouwantdisruptionstonormaloperationskepttoanabsoluteminimumsothatthebusinesscanfocusongettingPhoenixdone.”
Lookingsurprised,Stevenods.“Exactly.Yes,wellput.Whateveryousaid,that’sexactlywhatIwant.”
Hehandsmeane-mailprintoutfromDickLandry,theCFO.
From:DickLandry
To:SteveMasters
Date:September2,8:27AM
Priority:Highest
Subject:ACTIONNEEDED:payrollrunisfailing
Hey,Steve.We’vegotseriousissueswiththisweek’spayroll.We’retryingtofigureoutiftheproblemiswiththenumbersorinthepayrollsystem.Eitherway,thousandsofemployeeshavepaychecksstuckinsystem&areatriskofnotgettingpaid.Seriouslybadnews.
Wemustfixthisbeforepayrollwindowclosesat5PMtoday.Pleaseadviseonhowtoescalatethis,giventhenewITorg.
Dick
Iwince.Employeesnotgettingpaychecksmeansfamiliesnotbeingabletopaytheirmortgagesorputfoodonthetable.
Suddenly,Irealizethatmyfamily’smortgagepaymentisdueinfourdays,andwecouldbeoneofthefamiliesaffected.Alatemortgagepaymentcouldscrewupourcreditratingevenmore,whichwespentyearsrepairingafterweputPaige’sstudentloansonmycreditcard.
“Youwantmetojumponthisandmanagetheincidenttoconclusion?”
Stevenods,givingmethethumbs-up.“Keepmepostedontheprogress,please.”Hisexpressionturnsgrave.“Everyresponsiblecompanytakescareofitsemployees.Manyofourfactoryworkerslivefrompaychecktopaycheck.Donotcreatehardshipfortheirfamilies,youhear?Thiscouldgetusintroublewiththeunion,maybeeventriggeringawork-stoppage,creating
someverybadpressforus.”
Inodautomatically.“Restorecriticalbusinessoperationsandkeepusoutofthefront-pagenews.Gotit.Thanks.”
Why,exactly,I’mthankinghimisnotclear.
Chapter2•Tuesday,September2
“How’ditgointhere?”Stacyaskskindly,lookingupfromherkeyboard.
Ijustshakemyhead.“Ican’tbelieveit.HejusttalkedmeintotakinganewjobIdon’twant.Howdidthathappen?”
“Hecanbeverypersuasive,”shesays.“Forwhatit’sworth,he’sone-of-a-kind.I’veworkedforhimfornearlytenyears,andI’llfollowhimanywhere.AnythingIcanhelpwithtomakeyourjobeasier?”
Thinkingforamoment,Iask,“There’sanurgentpayrollissuethatneedstobefixed.DickLandryisonfloorthree,right?”
“Hereyougo,”shesays,beforeI’vefinishedaskingmyquestion,handingmeaPost-itnotewithallofDick’scontactinformation.Officelocation,phonenumbers,andeverything.
Grateful,Ismileather.“Thanksalot—youarefantastic!”
IdialDick’scellphoneonmywaytotheelevator.“Dickhere,”heanswersgruffly,stilltypinginthebackground.
“ThisisBillPalmer.StevejustmademethenewVPofITOperations,andheaskedmeto—”
“Congratulations,”heinterrupts.“Nowlook,mypeoplefoundahugepayrollirregularity.Whencanyougettomyoffice?”
“Rightaway,”Ireply.Iheartheclickofhimendingthecall.I’vehadwarmerwelcomes.
Onthethirdfloor,IwalkthroughFinanceandAccounting,surroundedbypinstripedshirtsandstarchedcollars.IfindDickathisdesk,stillonthephonewithsomeone.Whenheseesme,heputshishandoverthemouthpiece.“YoufromIT?”heasksgruffly.
WhenInod,hesaysintothephone,“Look,Igottarun.Someonewho’ssupposedlygoingtohelpisfinallyhere.I’llcallyouback.”Withoutwaitingforananswer,hehangsupthephone.
I’veneveractuallyseensomeonewhoroutinelyhangsuponpeople.Ibracemyselfforaconversationthatislikelytobeshortonanycomforting“let’sgettoknoweachother”foreplay.
Asifinahostagesituation,Islowlyraisemyhands,showingDicktheprintede-mail.“Stevejusttoldmeaboutthepayrolloutage.What’sthebestwayformetogetsomesituationalawarenesshere?”
“We’reindeepkimchi,”Dickresponds.“Inyesterday’spayrollrun,alloftherecordsforthehourlyemployeeswentmissing.We’reprettydamnedsureit’sanITissue.Thisscrewupispreventingusfrompayingouremployees,violatingcountlessstatelaborlaws,and,nodoubt,theunionisgoingtoscreambloodymurder.”
Hemuttersunderhisbreathforamoment.“Let’sgoseeAnn,myOperationsManager.She’sbeenpullingherhairoutsinceyesterdayafternoon.”
Walkingquicklytokeepup,Inearlyrunintohimwhenhestopsandpeersthroughaconferenceroomwindow.Heopensthedoor.“How’sitgoinginhere,Ann?”
Therearetwowell-dressedwomenintheroom:one,aroundforty-fiveyearsold,studiesthewhiteboard,filledwithflowchartsandalotoftabulatednumbers,andtheother,inherearlythirties,typesonalaptop.Spreadsheetsarestrewnalloverthelargeconferenceroomtable.Theolderwomangestureswithanopenmarkeratwhatappearstobealistofpotentialfailurecauses.
Somethingaboutthewaytheydress,andtheirconcernedandirritatedexpressions,makesmethinktheywererecruitedfromalocalaccountingfirm.Ex-auditors.Goodtohavethemonourside,Isuppose.
Annshakesherheadinexhaustedfrustration.“Notmuchprogress,I’mafraid.We’realmostcertainthisisanITsystemsfailureinoneoftheupstreamtimekeepingsystems.Allofthehourlyfactoryworkerrecordsgotscrewedupinthelastupload—”
Dickinterruptsher.“ThisisBillfromIT.He’sbeenassignedtofixthismessordietrying,iswhatIthinkhesaid.”
Isay,“Hi,guys.I’vejustbeenmadethenewheadofITOperations.Canyoustartfromthebeginningandtellmewhatyouknowabouttheproblem?”
Annwalksovertotheflowchartonthewhiteboard.“Let’sstartwiththeinformationflow.Ourfinancialsystemgetspayrolldatafromallourvariousdivisionsindifferentways.Werollupallthenumbersforsalariedandhourlypersonnel,whichincludeswagesandtaxes.Soundseasy,butit’sextremelycomplex,becauseeachstatehasdifferenttaxtables,laborlaws,andsoforth.
“Tomakesuresomethingdoesn’tgetscrewedup,”shecontinues,“wemakesurethesummarizednumbersmatchthedetailednumbersfromeachdivision.”
AsIhurriedlyjotdownsomenotes,shecontinues,“It’saprettyclunkyandmanualprocess.Itworksmostofthetime,butyesterdaywediscoveredthatthegeneralledgeruploadforhourlyproductionstaffdidn’tcomethrough.Allofthehourlieshadzeroesfortheirhoursworkedandamountdue.
“We’vehadsomanyproblemswiththisparticularupload,”shesays,obviouslyfrustrated,“thatITgaveusaprogramthatweusetodomanualcorrections,sowedon’thavetobotherthemanymore.”
Iwince.Idon’tlikefinancepersonnelmanuallychangingpayrolldataoutsidethepayrollapplication.It’serror-proneanddangerous.SomeonecouldcopythatdataontoaUSBdriveore-mailitoutsideoftheorganization,whichishoworganizationslosesensitivedata.
“Didyousayallthenumbersforsalariedemployeesareokay?”Iask.
“That’sright,”shereplies.
“Buthourlyemployeesareallzeroes,”Iconfirm.
“Yep,”sheagainreplies.
Interesting.Iask,“Whydoyouthinkthepayrollrunfailedwhenitwasworkingbefore?Haveyouhadproblemslikethisinthepast?”
Sheshrugs.“Nothinglikethishashappenedbefore.Ihavenoideawhatcouldhavecausedit—nomajorchangeswerescheduledforthispayperiod.I’vebeenaskingthesamequestions,butuntilwehearfromtheITguys,we’restuckdeadinthewater.”
“Whatisourbackupplan,”Iask,“ifthingsaresohosedthatwecan’tgetthehourlyemployeedataintime?”
“Forcryingoutloud,”Dicksays.“It’sinthate-mailyou’reholding.Thedeadlineforelectronicpaymentsis5p.m.,today.Ifwecan’thitthatwindow,wemayhavetoFedExbalesofpapercheckstoeachofourfacilitiesforthemtodistributetoemployees!”
Ifrownatthisscenarioandsodoestherestofthefinanceteam.
“Thatwon’twork,”Annsays,clickingamarkeronherteeth.“We’veoutsourcedourpayrollprocessing.Eachpayperiod,weuploadthepayrolldatatothem,whichtheythenprocess.Intheworstcase,maybewedownload
thepreviouspayrollrun,modifyitinaspreadsheet,andthenre-uploadit?
“Butbecausewedon’tknowhowmanyhourseachemployeeworked,wedon’tknowhowmuchtopaythem!”shecontinues.“Wedon’twanttooverpayanyone,butthat’sbetterthanaccidentallyunderpayingthem.”
It’sobviousthatplanBisfraughtwithproblems.We’dbasicallybeguessingatpeople’spaychecks,aswellaspayingpeoplewhowereterminated,andnotpayingpeoplewhowerenewlyhired.
TogetFinancethedatatheyneed,wemayhavetocobbletogethersomecustomreports,whichmeansbringingintheapplicationdevelopersordatabasepeople.
Butthat’slikethrowinggasolineonthefire.Developersareevenworsethannetworkingpeople.Showmeadeveloperwhoisn’tcrashingproductionsystems,andI’llshowyouonewhocan’tfogamirror.Ormorelikely,isonvacation.
Dicksays,“Thesearetwolousyoptions.Wecoulddelayourpayrollrununtilwehavethecorrectdata.Butwecan’tdothis—evenifwe’reonlyadaylate,we’llhavetheunionsteppingin.So,thatleavesAnn’sproposalofpayingouremployeessomething,evenifit’stheincorrectamount.We’dhavetoadjusteveryone’spaycheckinthenextpayperiod.Butnowwehaveafinancialreportingerrorthatwe’vegottogobackandfix.”
Hepinchesthebridgeofhisnoseandcontinuestoramble.“We’llhaveabunchofoddjournalentriesinourgeneralledger,justwhenourauditorsarehereforourSOX-404audits.Whentheyseethis,they’llneverleave.
“Oh,Christ.Afinancialreportingerror?”Dickmutters.“We’llneedapprovalfromSteve.We’regoingtohaveauditorscampedouthereuntilthecowscomehome.Noone’llevergetanyrealworkdoneagain.”
SOX-404isshortfortheSarbanes-OxleyActof2002,whichCongressenactedinresponsetotheaccountingfailuresatEnron,WorldCom,andTyco.ItmeanstheCEOandCFOhavetopersonallysigntheirnames,attestingthattheircompany’sfinancialstatementsareaccurate.
Everyonelongsforthedayswhenwedidn’tspendhalfourtimetalkingtoauditors,complyingwitheachnewregulatoryrequirementdujour.
Ilookatmynotesandthenattheclock.Timeisrunningout.
“Dick,basedonwhatI’veheard,IrecommendthatyoucontinuetoplanfortheworstandwefullydocumentplanB,sowecanpullitoffwithout
furthercomplications.Furthermore,Irequestthatwewaituntil3p.m.beforemakingadecision.Wemaybestillabletogetallthesystemsanddataback.”
WhenAnnnods,Dicksays,“Okay,you’vegotfourhours.”
Isay,“Restassuredthatweunderstandtheurgencyofthesituationandthatyou’llbeapprisedofhowit’sgoingassoonasIfindoutmyself.”
“Thanks,Bill,”Annsays.DickremainssilentasIturnaroundandwalkoutthedoor.
Ifeelbetter,nowthatI’veseentheproblemfromthebusinessperspective.It’snowtimetogetunderthecoversandfindoutwhatbrokethecomplexpayrollmachinery.
Whilewalkingdownthestairs,Idigoutmyphoneandscanmye-mails.MyfeelingofcalmfocusdisappearswhenIseethatStevehasn’tsentoutanannouncementofmypromotion.WesDavisandPattyMcKee,whountiltodayweremypeers,stillhavenoideathatI’mnowtheirnewboss.
Thanks,Steve.
WhenIenterBuilding7,ithitsme.OurbuildingistheghettooftheentirePartsUnlimitedcampus.
Itwasbuiltinthe1950s,andlastremodeledinthe1970s,obviouslybuiltforutility,notaesthetics.Building7usedtobeourlargebrake-padmanufacturingfactoryuntilitwasconvertedintodatacenterandofficespace.Itlooksoldandneglected.
Thesecurityguardsayscheerfully,“Hello,Mr.Palmer.Howisthemorninggoingsofar?”
Foramoment,I’mtemptedtoaskhimtowishmeluck,sohecangetpaidthecorrectamountthisweek.Ofcourse,Imerelyreturnhisfriendlygreeting.
I’mheadedtowardtheNetworkOperationsCenter,oraswecallit,theNOC,whereWesandPattyaremostlikelytobe.They’renowmytwoprimarymanagers.
WesisDirectorofDistributedTechnologyOperations.HehastechnicalresponsibilityforoverathousandWindowsservers,aswellasthedatabaseandnetworkingteams.PattyistheDirectorofITServiceSupport.Sheownsallthelevel1and2helpdesktechnicianswhoanswerthephonesaroundtheclock,handlingbreak-fixissuesandsupportrequestsfromthebusiness.ShealsoownssomeofthekeyprocessesandtoolsthattheentireITOperationsorganizationreliesupon,likethetroubleticketingsystem,monitoring,and
runningthechangemanagementmeetings.
Iwalkpastrowsuponrowsofcubicles,thesameaseveryotherbuilding.However,unlikeBuildings2and5,Iseepeelingpaintanddarkstainsseepingthroughthecarpet.
Thispartofthebuildingwasbuiltontopofwhatusedtobethemainassemblyfloor.Whentheyconvertedit,theycouldn’tgetallthemachineoilcleanedup.Nomatterhowmuchsealantweputdowntocoatthefloors,oilstillhasatendencytoseepthroughthecarpet.
Imakeanotetoputinabudgetrequesttoreplacethecarpetsandpaintthewalls.IntheMarines,keepingthebarracksneatandtidywasnotonlyforaestheticsbutalsoforsafety.
Oldhabitsdiehard.
IheartheNOCbeforeIseeit.It’salargebullpenarea,withlongtablessetupalongonewall,displayingthestatusofallthevariousITservicesonlargemonitors.Thelevel1and2helpdeskpeoplesitatthethreerowsofworkstations.
It’snotexactlylikemissioncontrolinApollo13,butthat’showIexplainittomyrelatives.
Whensomethinghitsthefan,youneedallthevariousstakeholdersandtechnologymanagerstocommunicateandcoordinateuntiltheproblemisresolved.Likenow.Attheconferencetable,fifteenpeopleareinthemidstofaloudandheateddiscussion,huddledaroundoneoftheclassicgrayspeakerphonesthatresemblesaUFO.
WesandPattyaresittingnexttoeachotherattheconferencetable,soIwalkbehindthemtolistenin.Wesleansbackinhischairwithhisarmscrossedoverhisstomach.Theydon’tgetallthewayacross.Atsixfeetthreeinchestallandover250pounds,hecastsashadowonmostpeople.Heseemstobeinconstantmotionandhasareputationofsayingwhateverisonhismind.
Pattyisthecompleteopposite.WhereWesisloud,outspoken,andshootsfromthehip,Pattyisthoughtful,analytical,andasticklerforprocessesandprocedures.WhereWesislarge,combative,andsometimesevenquarrelsome,Pattyiselfin,logical,andlevelheaded.ShehasareputationforlovingprocessesmorethanpeopleandisofteninthepositionoftryingtoimposeorderonthechaosinIT.
She’sthefaceoftheentireITorganization.WhenthingsgowronginIT,
peoplecallPatty.She’sourprofessionalapologist,whetherit’sservicescrashing,webpagestakingtoolongtoload,or,asintoday’scase,missingorcorrupteddata.
TheyalsocallPattywhentheyneedtheirworkdone—likeupgradingacomputer,changingaphonenumber,ordeployinganewapplication.Shedoesallofthescheduling,sopeoplearealwayslobbyinghertogettheirworkdonefirst.She’llthenhanditofftopeoplewhodothework.Forthemostpart,theyliveineithermyoldgrouporinWes’group.
Wespoundsthetable,saying,“Justgetthevendoronthephoneandtellthemthatunlesstheygetatechdownherepronto,we’regoingtothecompetition.We’reoneoftheirlargestcustomers!Weshouldprobablyhaveabandonedthatpileofcrapbynow,cometothinkofit.”
Helooksaroundandjokes,“Youknowthesaying,right?Thewayyoucantellavendorislyingiswhentheirlipsaremoving.”
OneoftheengineersacrossfromWessays,“Wehavethemonthephonerightnow.Theysayit’llbeatleastfourhoursbeforetheirSANfieldengineerison-site.”
Ifrown.WhyaretheytalkingabouttheSAN?Storageareanetworksprovidecentralizedstoragetomanyofourmostcriticalsystems,sofailuresaretypicallyglobal:Itwon’tbejustoneserverthatgoesdown;it’llbehundredsofserversthatgodownallatonce.
WhileWesstartsarguingwiththeengineer,Itrytothink.NothingaboutthispayrollrunfailuresoundslikeaSANissue.Annsuggestedthatitwasprobablysomethinginthetimekeepingapplicationssupportingeachplant.
“ButafterwetriedtorollbacktheSAN,itstoppedservingdataentirely,”anotherengineersays.“Thenthedisplaystarteddisplayingeverythinginkanji!Well,wethinkitwaskanji.Whateveritwas,wecouldn’tmakeheadsortailsofthoselittlepictures.That’swhenweknewweneededtogetthevendorinvolved.”
AlthoughI’mjoininglate,I’mconvincedwe’retotallyonthewrongtrack.
IleanintowhispertoWesandPatty,“CanIgetaminutewithyouguysinprivate?”
Westurnsand,withoutgivingmehisfullattention,saysloudly,“Can’titwait?Incaseyouhaven’tnoticed,we’reinthemiddleofahugeissuehere.”
Iputmyhandfirmlyonhisshoulder.“Wes,thisisreallyimportant.It’s
aboutthepayrollfailureandconcernsaconversationIjusthadwithSteveMastersandDickLandry.”
Helookssurprised.Pattyisalreadyoutofherchair.“Let’susemyoffice,”shesays,leadingtheway.
FollowingPattyintoheroffice,Iseeaphotoonherwallofherdaughter,whoI’dguessiselevenyearsold.I’mamazedathowmuchshelookslikePatty—fearless,incrediblysmart,andformidable—inawaythatisabitscaryinsuchacutelittlegirl.
Inagruffvoice,Wessays,“Okay,Bill,what’ssoimportantthatyouthinkisworthinterruptingaSev1outageinprogress?”
That’snotabadquestion.Severity1outagesareseriousbusiness-impactingincidentsthataresodisruptive,wetypicallydropeverythingtoresolvethem.Itakeadeepbreath.“Idon’tknowifyou’veheard,butLukeandDamonarenolongerwiththecompany.Theofficialwordisthatthey’vedecidedtotakesometimeoff.Morethanthat,Idon’tknow.”
Thesurprisedexpressionsontheirfacesconfirmmysuspicions.Theydidn’tknow.Iquicklyrelatetheeventsofthemorning.Pattyshakesherhead,utteringatsk-tskindisapproval.
Weslooksangry.HeworkedwithDamonformanyyears.Hisfacereddens.“Sonowwe’resupposedtotakeordersfromyou?Look,nooffense,pal,butaren’tyoualittleoutofyourleague?You’vemanagedthemidrangesystems,whicharebasicallyantiques,foryears.Youcreatedanicelittlecushyjobforyourselfupthere.Andyouknowwhat?Youhaveabsolutelynoideahowtorunmoderndistributedsystems—toyou,the1990sisstillthefuture!
“Quitefrankly,”hesays,“IthinkyourheadwouldexplodeifyouhadtolivewiththerelentlesspaceandcomplexityofwhatIdealwitheveryday.”
Iexhale,whilecountingtothree.“YouwanttotalktoSteveabouthowyouwantmyjob?Bemyguest.Let’sgetthebusinesswhattheyneedfirstandmakesurethateveryonegetspaidontime.”
Pattyrespondsquickly,“Iknowyouweren’taskingme,butIagreethatthepayrollincidentneedstobeourfocus.”Shepausesandthensays,“IthinkStevemadeagoodchoice.Congratulations,Bill.Whencanwetalkaboutabiggerbudget?”
Iflashherasmallsmileandanodofthanks,returningmygazetoWes.
Acouplemomentsgoby,andexpressionsIcan’tquitedeciphercrosshisface.Finallyherelents,“Yeah,fine.AndIwilltakeyouuponyouroffertotalktoSteve.He’sgotalotofexplainingtodo.”
Inod.ThinkingaboutmyownexperiencewithSteve,IgenuinelywishWesluckifheactuallydecidestohaveashowdownwithhim.
“Thankyouforyoursupport,guys.Iappreciateit.Now,whatdoweknowaboutthefailure—orfailures?What’sallthisaboutsomeSANupgradeyesterday?Aretheyrelated?”
“Wedon’tknow,”Wesshakeshishead.“Weweretryingtofigurethatoutwhenyouwalkedin.WewereinthemiddleofaSANfirmwareupgradeyesterdaywhenthepayrollrunfailed.BrentthoughttheSANwascorruptingdata,sohesuggestedwebackoutthechanges.Itmadesensetome,butasyouknow,theyendedupbrickingit.”
Upuntilnow,I’veonlyheard“bricking”somethinginreferencetobreakingsomethingsmall,likewhenacellphoneupdategoesbad.Usingittorefertoamillion-dollarpieceofequipmentwhereallourirreplaceablecorporatedataarestoredmakesmefeelphysicallyill.
BrentworksforWes.He’salwaysinthemiddleoftheimportantprojectsthatITisworkingon.I’veworkedwithhimmanytimes.He’sdefinitelyasmartguybutcanbeintimidatingbecauseofhowmuchheknows.Whatmakesitworseisthathe’srightmostofthetime.
“Youheardthem,”Wessays,gesturingtowardtheconferencetablewheretheoutagemeetingcontinuesunabated.“TheSANwon’tboot,won’tservedata,andourguyscan’tevenreadanyoftheerrormessagesonthedisplaybecausethey’reinsomeweirdlanguage.Nowwe’vegotabunchofdatabasesdown,including,ofcourse,payroll.”
“ToworktheSANissue,wehadtopullBrentoffofaPhoenixjobwepromisedtogetdoneforSarah,”Pattysaysominously.“There’sgoingtobehelltopay.”
“Uh-oh.Whatexactlydidwepromiseher?”Iask,alarmed.
SarahistheSVPofRetailOperations,andshealsoworksforSteve.Shehasanuncannyknackforblamingotherpeopleforherscrewups,especiallyITpeople.Foryears,she’sbeenabletoescapeanysortofrealaccountability.
AlthoughI’veheardrumorsthatSteveisgroomingherashisreplacement,I’vealwaysdiscountedthatasbeingtotallyimpossible.I’mcertainthatStevecan’tbeblindtohermachinations.
“SarahheardfromsomeonethatwewerelategettingabunchofvirtualmachinesovertoChris,”shereplies.“Wedroppedeverythingtogetonit.Thatis,untilwehadtodropeverythingtofixtheSAN.”
ChrisAllers,ourVPofApplicationDevelopment,isresponsiblefordevelopingtheapplicationsandcodethatthebusinessneeds,whichthengetturnedovertoustooperateandmaintain.Chris’lifeiscurrentlydominatedbyPhoenix.
Iscratchmyhead.Asacompany,we’vemadeahugeinvestmentinvirtualization.Althoughitlooksuncannilylikethemainframeoperatingenvironmentfromthe1960s,virtualizationchangedthegameinWes’world.Suddenly,youdon’thavetomanagethousandsofphysicalserversanymore.They’renowlogicalinstancesinsideofonebig-ironserverormaybeevenresidingsomewhereinthecloud.
Buildinganewserverisnowaright-clickinsideofanapplication.Cabling?It’snowaconfigurationsetting.Butdespitethepromisethatvirtualizationwasgoingtosolveallourproblems,hereweare—stilllateindeliveringavirtualmachinetoChris.
“IfweneedBrenttoworktheSANissue,keephimthere.I’llhandleSarah,”Isay.“ButifthepayrollfailurewascausedbytheSAN,whydidn’tweseemorewidespreadoutagesandfailures?”
“Sarahisdefinitelygoingtobeoneunhappycamper.Youknow,suddenlyIdon’twantyourjobanymore,”Wessayswithaloudlaugh.“Don’tgetyourselffiredonyourfirstday.They’llprobablycomeformenext!”
Wespausestothink.“Youknow,youhaveagoodpointabouttheSAN.Brentisworkingtheissuerightnow.Let’sgotohisdeskandseewhathethinks.”
PattyandIbothnod.It’sagoodidea.Weneedtoestablishanaccuratetimelineofrelevantevents.Andsofar,we’rebasingeverythingonhearsay.
Thatdoesn’tworkforsolvingcrimes,anditdefinitelydoesn’tworkforsolvingoutages.
Chapter3•Tuesday,September2
IfollowPattyandWesastheywalkpasttheNOC,intotheseaofcubicles.Weendupinagiantworkspacecreatedbycombiningsixcubicles.AlargetablesitsagainstonewallwithakeyboardandfourLCDmonitors,likeaWallStreettradingdesk.Therearepilesofserverseverywhere,allwithblinkinglights.Eachportionofthedeskiscoveredbymoremonitors,showinggraphs,loginwindows,codeeditors,Worddocuments,andcountlessapplicationsIdon’trecognize.
Brenttypesawayinawindow,oblivioustoeverythingaroundhim.Fromhisphone,IheartheNOCconferenceline.Heobviouslydoesn’tseemworriedthattheloudspeakerphonemightbotherhisneighbors.
“Hey,Brent.Yougotaminute?”Wesasksloudly,puttingahandonhisshoulder.
“Canitwait?”Brentreplieswithoutevenlookingup.“I’mactuallykindofbusyrightnow.WorkingtheSANissue,youknow?”
Wesgrabsachair.“Yeah,that’swhatwe’reheretotalkabout.”
WhenBrentturnsaround,Wescontinues,“Tellmeagainaboutlastnight.WhatmadeyouconcludethattheSANupgradecausedthepayrollrunfailure?”
Brentrollshiseyes,“IwashelpingoneoftheSANengineersperformthefirmwareupgradeaftereverybodywenthome.Ittookwaylongerthanwethought—nothingwentaccordingtothetechnote.Itgotprettyhairy,butwefinallyfinishedaroundseveno’clock.
“WerebootedtheSAN,butthenalltheself-testsstartedfailing.Weworkeditforaboutfifteenminutes,tryingtofigureoutwhatwentwrong.That’swhenwegotthee-mailsaboutthepayrollrunfailing.That’swhenIsaid,‘GameOver.’
“Wewerejusttoomanyversionsbehind.TheSANvendorprobablynevertestedtheupgradepathweweregoingdown.Icalledyou,tellingyouIwantedtopulltheplug.Whenyougavemethenod,westartedtherollback.
“That’swhentheSANcrashed,”hesays,slumpinginhischair.“Itnotonlytookdownpayrollbutabunchofotherservers,too.”
“We’vebeenmeaningtoupgradetheSANfirmwareforyears,butwenevergotaroundtoit,”Wesexplains,turningtome.“Wecamecloseonce,butthenwecouldn’tgetabigenoughmaintenancewindow.Performancehasbeengettingworseandworse,tothepointwhereabunchofcriticalappswerebeingimpacted.Sofinally,lastnight,wedecidedtojustbitethebulletanddotheupgrade.”
Inod.Then,myphonerings.
It’sAnn,soIputheronspeakerphone.
“Asyousuggested,welookedatthedatawepulledfromthepayrolldatabaseyesterday.Thelastpayperiodwasfine.Butforthispayperiod,alltheSocialSecuritynumbersforthefactoryhourliesarecompletegibberish.Andalltheirhoursworkedandwagefieldsarezeroes,too.Noonehaseverseenanythinglikethisbefore.”
“Justonefieldisgibberish?”Iask,raisingmyeyebrowsinsurprise.“Whatdoyoumeanby‘gibberish’?What’sinthefields?”
Shetriestodescribewhatshe’sseeingonherscreen.“Well,they’renotnumbersorletters.There’ssomeheartsandspadesandsomesquigglycharacters…Andthere’sabunchofforeigncharacterswithumlauts…Andtherearenospaces.Isthatimportant?”
WhenBrentsnickersashehearsAnntryingtoreadlinenoisealoud,Igivehimasternglance.“Ithinkwe’vegotthepicture,”Isay.“Thisisaveryimportantclue.Canyousendthespreadsheetwiththecorrupteddatatome?”
Sheagrees.“Bytheway,areabunchofdatabasesdownnow?That’sfunny.Theywereuplastnight.”
Wesmutterssomethingunderhisbreath,silencingBrentbeforehecansayanything.
“Umm,yes.We’reawareoftheproblemandwe’reworkingit,too,”Ideadpan.
Whenwehangup,Ibreatheasighofrelief,takingamomenttothankwhateverdeityprotectspeoplewhofightfiresandfixoutages.
“Onlyonefieldcorruptedinthedatabase?Comeon,guys,thatdefinitelydoesn’tsoundlikeaSANfailure.”Isay.“Brent,whatelsewasgoingonyesterday,besidestheSANupgrade,thatcouldhavecausedthepayrollruntofail?”
Brentslouchesinhischair,spinningitaroundwhilehethinks.“Well,now
thatyoumentionit…Adeveloperforthetimekeepingapplicationcalledmeyesterdaywithastrangequestionaboutthedatabasetablestructure.IwasinthemiddleofworkingonthatPhoenixtestVM,soIgavehimareallyquickanswersoIcouldgetbacktowork.Youdon’tsupposehedidsomethingtobreaktheapp,doyou?”
WesturnsquicklytothespeakerphonedialedintotheNOCconferencecallthathasbeenonthiswholetimeandunmutesthephone.“Hey,guys,it’sWeshere.I’mwithBrentandPatty,aswellaswithournewboss,BillPalmer.SteveMastershasputhimchargeofallofITOps.Solistenup,guys.”
Mydesireforanorderlyannouncementofmynewroleseemslessandlesslikely.
Wescontinues,“Doesanyoneknowanythingaboutadevelopermakinganychangestothetimekeepingapplicationinthefactories?Brentsayshegotacallfromsomeonewhoaskedaboutchangingsomedatabasetables.”
Fromthespeakerphone,avoicepipesup,“Yeah,Iwashelpingsomeonewhowashavingsomeconnectivityissueswiththeplants.I’mprettysurehewasadevelopermaintainingthetimekeepingapp.HewasinstallingsomesecurityapplicationthatJohnneededtogetupandrunningthisweek.IthinkhisnamewasMax.Istillhavehiscontactinformationaroundheresomewhere…Hesaidhewasgoingonvacationtoday,whichiswhytheworkwassourgent.”
Nowwe’regettingsomewhere.
Adeveloperjamminginanurgentchangesohecouldgoonvacation—possiblyaspartofsomeurgentprojectbeingdrivenbyJohnPesche,ourChiefInformationSecurityOfficer.
Situationslikethisonlyreinforcemydeepsuspicionofdevelopers:they’reoftencarelesslybreakingthingsandthendisappearing,leavingOperationstocleanupthemess.
TheonlythingmoredangerousthanadeveloperisadeveloperconspiringwithSecurity.Thetwoworkingtogethergivesusmeans,motive,andopportunity.
I’mguessingourCISOprobablystrong-armedaDevelopmentmanagertodosomething,whichresultedinadeveloperdoingsomethingelse,whichbrokethepayrollrun.
InformationSecurityisalwaysflashingtheirbadgesatpeopleandmakingurgentdemands,regardlessoftheconsequencestotherestofthe
organization,whichiswhywedon’tinvitethemtomanymeetings.Thebestwaytomakesuresomethingdoesn’tgetdoneistohavethemintheroom.
They’realwayscomingupwithamillionreasonswhyanythingwedowillcreateasecurityholethatalienspace-hackerswillexploittopillageourentireorganizationandstealallourcode,intellectualproperty,creditcardnumbers,andpicturesofourlovedones.Thesearepotentiallyvalidrisks,butIoftencan’tconnectthedotsbetweentheirshrill,hysterical,andself-righteousdemandsandactuallyimprovingthedefensibilityofourenvironment.
“Okay,guys,”Isaydecisively.“Thepayrollrunfailureislikeacrimesceneandwe’reScotlandYard.TheSANisnolongerasuspect,butunfortunately,we’veaccidentallymaimeditduringourinvestigation.Brent,youkeepworkingontheinjuredSAN—obviously,we’vegottogetitupandrunningsoon.
“WesandPatty,ournewpersonsofinterestareMaxandhismanager,”Isay.“Dowhateverittakestofindthem,detainthem,andfigureoutwhattheydid.Idon’tcareifMaxisonvacation.I’mguessingheprobablymessedupsomething,andweneedtofixitby3p.m.”
Ithinkforamoment.“I’mgoingtofindJohn.Eitherofyouwanttojoinme?”
WesandPattyargueoverwhowillhelpinterrogateJohn.Pattysaysadamantly,“Itshouldbeme.I’vebeentryingtokeepJohn’speopleinlineforyears.Theyneverfollowourprocess,anditalwayscausesproblems.I’dlovetoseeSteveandDickrakehimoverthecoalsforpullingastuntlikethis.”
Itisapparentlyaconvincingargument,asWessays,“Okay,he’sallyours.Ialmostfeelsorryforhimnow.”
Isuddenlyregretmychoiceofwords.Thisisn’tawitchhunt,andI’mnotlookingforretribution.Westillneedatimelineofallrelevanteventsleadinguptothefailure.
JumpingtoinappropriateconclusionscausedtheSANfailurelastnight.Wewon’tmakethesekindsofmistakesagain.Notonmywatch.
AsPattyandIcallJohn,IsquintatthephonenumberonPatty’sscreen,wonderingifit’stimetoheedmywife’sadvicetogetglasses.Yetanotherreminderthatfortyisjustaroundthecorner.
Idialthenumber,andavoiceanswersinonering,“Johnhere.”
IquicklytellhimaboutthepayrollandSANfailureandthenask,“Didyoumakeanychangestothetimekeepingapplicationyesterday?”
Hesays,“Thatsoundsbad,butIcanassureyouthatwedidn’tmakeanychangestoyourmidrangesystems.SorryIcan’tbeofmorehelp.”
Isigh.IthoughtthatbynoweitherSteveorLaurawouldhavesentouttheannouncementofmypromotion.IseemdestinedtoexplainmynewroleineveryinteractionIhave.
IwonderifitwouldbeeasierifIjustsentouttheannouncementmyself.
Irepeattheabridgedstoryofmyhastypromotionyetagain.“Wes,Patty,andIheardthatyouwereworkingwithMaxtodeploysomethingurgentyesterday.Whatwasit?”
“LukeandDamonaregone?”Johnsoundssurprised.“IneverthoughtthatStevewouldactuallyfirebothofthemoveracomplianceauditfinding.Butwhoknows?Maybethingsarefinallystartingtochangearoundhere.Letthisbealessontoyou,Bill.YouOperationspeoplecan’tkeepdraggingyourfeetonsecurityissuesanymore!Justsomefriendlyadvice…
“Speakingofwhich,I’msuspiciousabouthowthecompetitionkeepsgettingthejumponus,”hecontinues.“Astheysay,onceiscoincidence.Twiceishappenstance.Thirdmustbeenemyaction.Maybeoursalespeople’se-mailsystemshavebeenhacked.Thatwouldsureexplainwhywe’relosingsomanydeals.”
Johncontinuestotalk,butmymindisstillstuckathissuggestionthatLukeandDamonmayhavebeenfiredoversomethingsecurityrelated.It’spossible—Johnroutinelydealswithsomeprettypowerfulpeople,likeSteveandtheboardaswellastheinternalandexternalauditors.
However,I’mcertainStevedidn’tmentioneitherJohnorInformationSecurityasreasonsfortheirdeparture—onlytheneedtofocusonPhoenix.
IlookatPattyquestioningly.Shejustrollshereyesandthentwirlsherfingeraroundherear.Clearly,shethinksJohn’stheoryiscrazy.
“HasStevegivenyouanyinsightsontheneworgstructure?”Iaskoutofgenuinecuriosity—Johnisalwayscomplainingthatinformationsecuritywasalwaysprioritizedtoolow.He’sbeenlobbyingtobecomeapeeroftheCIO,sayingitwouldresolveaninherentconflictofinterest.Tomyknowledge,hehadn’tsucceeded.
It’snosecretthatLukeandDamonsidelinedJohnasmuchaspossible,so
hecouldn’tinterferewithpeoplewhodidrealwork.Johnstillmanagedtoshowupatmeetings,despitetheirbestefforts.
“What?Ihavenocluewhat’sgoingon,”hesaysinanaggrievedtone,myquestionapparentlystrikinganerve.“I’mbeingkeptinthedark,likeusual.I’llprobablybethelasttofindout,too,ifhistoryisanyguide.Untilyoutoldme,IthoughtIwasstillreportingtoLuke.Andnowthathe’sgone,Idon’tknowwhoI’mreportingto.YougotacallfromSteve?”
“Thisisallabovemypaygrade—I’masmuchinthedarkasyouare,”Irespond,playingitdumb.Quicklychangingthesubject,Iask,“Whatcanyoutellusaboutthetimekeepingappchange?”
“I’llcallSteveandfindoutwhat’sgoingon.He’sprobablyforgottenInformationSecurityevenexists,”hecontinues,makingmewonderwhetherwe’lleverbeabletotalkaboutpayroll.
Tomyrelief,hefinallysays,“Okay,yeah,youwereaskingaboutMax.WehadanurgentauditissuearoundstorageofPII—thatis,personallyidentifiableinformationlikeSSNs—that’sSocialSecuritynumbers,obviously,birthdays,andsoforth.EuropeanUnionlawandnowmanyUSstatelawsprohibitusfromstoringthatkindofdata.Wegotahugeauditfindingaroundthis.Iknewitwasuptomyteamtosavethiscompanyfromitselfandpreventusfromgettingdingedagain.Thatwouldbefront-pagenews,youknow?”
Hecontinues,“Wefoundaproductthattokenizedthisinformation,sowenolongerhavetostoretheSSNs.Itwassupposedtobedeployedalmostayearago,butitnevergotdone,despiteallmybadgering.Nowwe’reoutoftime.ThePaymentCardIndustryauditors,that’sPCIforshort,areherelaterthismonth,soIfast-trackedtheworkwiththetimekeepingteamtogetitdone.”
Istareatmyphone,speechless.
Ontheonehand,I’mecstaticbecausewe’vefoundthesmokingguninJohn’shand.John’smentionoftheSSNfieldmatchesAnn’sdescriptionofthecorrupteddata.
Ontheotherhand:“LetmeseeifI’vegotthisright…”Isayslowly.“Youdeployedthistokenizationapplicationtofixanauditfinding,whichcausedthepayrollrunfailure,whichhasDickandSteveclimbingthewalls?”
Johnrespondshotly,“First,Iamquitecertainthetokenizationsecurityproductdidn’tcausetheissue.It’sinconceivable.Thevendorassuredusthatit’ssafe,andwecheckedalltheirreferences.Second,DickandStevehaveeveryreasontobeclimbingthewalls:Complianceisnotoptional.It’sthe
law.Myjobistokeepthemoutoforangejumpsuits,andsoIdidwhatIhadtodo.”
“‘Orangejumpsuits?’”
“Likewhatyouwearinprison,”hesays.“Myjobistokeepmanagementincompliancewithallrelevantlaws,regulations,andcontractualobligations.LukeandDamonwerereckless.Theycutcornersthatseverelyaffectedourauditandsecurityposture.Ifitweren’tformyactions,we’dprobablyallbeinjailbynow.”
Ithoughtweweretalkingaboutapayrollfailurenotbeingthrowninjailbysomeimaginarypoliceforce.
“John,wehaveprocessesandproceduresforhowyouintroducechangesintoproduction,”Pattysays.“Youwentaroundthem,and,onceagain,you’vecausedabigproblemthatwe’rehavingtorepair.Whydidn’tyoufollowtheprocess?”
“Ha!Goodone,Patty,”Johnsnorts.“Ididfollowtheprocess.Youknowwhatyourpeopletoldme?Thatthenextpossibledeploymentwindowwasinfourmonths.Hello?Theauditorsareon-sitenextweek!”
Hesaysadamantly,“Gettingtrappedinyourbureaucraticprocesswassimplynotanoption.Ifyouwereinmyshoes,you’ddothesamething.”
Pattyreddens.Isaycalmly,“AccordingtoDick,wehavefewerthanfourhourstogetthetimekeepingappup.NowthatweknowtherewasachangethataffectedSSNs,Ithinkwehavewhatweneed.”
Icontinue,“Max,whohelpedwiththedeployment,isonvacationtoday.WesorBrentwillbecontactingyoutolearnmoreaboutthistokenizationproductyoudeployed.Iknowyou’llprovidethemwithwhateverhelptheyneed.Thisisimportant.”
WhenJohnagrees,Ithankhimforhistime.“Wait,onemorequestion.Whydoyoubelievethatthisproductdidn’tcausethefailure?Didyoutestthechange?”
There’sashortsilenceonthephonebeforeJohnreplies,“No,wecouldn’ttestthechange.There’snotestenvironment.Apparently,youguysrequestedabudgetyearsago,but…”
Ishouldhaveknown.
“Well,that’sgoodnews,”PattysaysafterJohnhangsup.“Itmaynotbeeasytofix,butatleastwefinallyknowwhat’sgoingon.”
“WasJohn’stokenizationchangeinthechangeschedule?”Iask.
Shelaughshumorlessly.“That’swhatI’vebeentryingtotellyou.Johnrarelygoesthroughourchangeprocess.Nordomostpeople,forthatmatter.It’sliketheWildWestouthere.We’remostlyshootingfromthehip.”
Shesaysdefensively.“Weneedmoreprocessaroundhereandbettersupportfromthetop,includingITprocesstoolingandtraining.Everyonethinksthattherealwaytogetworkdoneistojustdoit.Thatmakesmyjobnearlyimpossible.”
Inmyoldgroup,wewerealwaysdisciplinedaboutdoingchanges.Noonemadechangeswithouttellingeveryoneelse,andwe’dbendoverbackwardtomakesureourchangeswouldn’tscrewsomeoneelseup.
I’mnotusedtoflyingthisblind.
“Wedon’thavetimetodointerrogationseverytimesomethinggoeswrong,”Isay,exasperated.“Getmealistofallthechangesmadeinthepast,say,threedays.Withoutanaccuratetimeline,wewon’tbeabletoestablishcauseandeffect,andwe’llprobablyendupcausinganotheroutage.”
“Goodidea,”shenods.“Ifnecessary,I’lle-maileveryoneinITtofindoutwhattheyweredoing,tocatchthingsthatweren’tonourschedule.”
“Whatdoyoumean,‘e-maileveryone?’There’snosystemwherepeopleputintheirchanges?Whataboutourticketingsystemorthechange-authorizationsystem?”Iask,stunned.ThisislikeScotlandYarde-mailingeveryoneinLondontofindoutwhowasnearthesceneofacrime.
“Dreamon,”shesays,lookingatmelikeI’manewbie,whichIsupposeIam.“Foryears,I’vebeentryingtogetpeopletouseourchangemanagementprocessandtools.ButjustlikeJohn,nooneusesit.Samewithourticketingsystem.It’sprettyhit-or-miss,too.”
ThingsarefarworsethanIthought.
“Okay,dowhatyouneedtodo,”Ifinallysay,unabletohidemyfrustration.“Makesureyouhitallthedeveloperssupportingthetimekeepingsystemaswellasallthesystemadministratorsandnetworkingpeople.Calltheirmanagers,andtellthemit’simportantthatweknowaboutanychanges,regardlessofhowunimportanttheymayseem.Don’tforgetJohn’speople,too.”
WhenPattynods,Isay,“Look,you’rethechangemanager.We’vegottodobetterthanthis.Weneedbettersituationalawareness,andthatmeanswe
needsomesortoffunctionalchangemanagementprocess.Geteveryonetobringintheirchangessowecanbuildapictureofwhatisactuallygoingonoutthere.”
Tomysurprise,Pattylooksdejected.“Look,I’vetriedthisbefore.I’lltellyouwhatwillhappen.TheChangeAdvisoryBoard,orCAB,willgettogetheronceortwice.Andwithinacoupleofweeks,peoplewillstopattending,sayingthey’retoobusy.Orthey’lljustmakethechangeswithoutwaitingforauthorizationbecauseofdeadlinepressures.Eitherway,it’llfizzleoutwithinamonth.”
“Notthistime,”Isayadamantly.“Sendoutameetingnoticetoallthetechnologyleadsandannouncethatattendanceisnotoptional.Iftheycan’tmakeit,theyneedtosendadelegate.Whenisthenextmeeting?”
“Tomorrow,”shesays.
“Excellent,”Isaywithgenuineenthusiasm.“I’mlookingforwardtoit.”
WhenIfinallygethome,it’saftermidnight.Afteralongdayofdisappointments,I’mexhausted.Balloonsareonthefloorandahalf-emptybottleofwinesitsonthekitchentable.Onthewallisacrayonpostersaying,“CongratulationsDaddy!”
WhenIcalledmywife,Paige,thisafternoontellingheraboutmypromotion,shewasfarhappierthanIwas.Sheinsistedoninvitingtheneighborsovertothrowalittlecelebration.Cominghomesolate,Imissedmyownparty.
At2p.m.,Pattyhadsuccessfullyarguedthatofthetwenty-sevenchangesmadeinthepastthreedays,onlyJohn’stokenizationchangeandtheSANupgradecouldbereasonablylinkedtothepayrollfailure.However,WesandhisteamwerestillunabletorestoreSANoperations.
At3p.m.,IhadtotellAnnandDickthebadnewsthatwehadnochoicebuttoexecuteplanB.Theirfrustrationanddisappointmentwerealltooevident.
Itwasn’tuntil7p.m.whenthetimekeepingapplicationwasbackupand11p.m.whentheSANwasfinallybroughtbackonline.
NotagreatperformanceonmyfirstdayasVPofITOperations.
BeforeIleftwork,Ie-mailedSteve,Dick,andAnnaquickstatusreport,promisingtodowhateverittakestopreventthistypeoffailurefromhappeningagain.
Igoupstairs,finishbrushingmyteeth,andcheckmyphoneonelasttimebeforegoingtobed,beingcarefulnottowakeupPaige.IcursewhenIseeane-mailfromourcompanyPRmanager,withasubjectof“Badnews.Wemaybeonthefrontpagetomorrow…”
Isitonthebed,squintingtoreadtheaccompanyingnewsstory.
ElkhartGroveHeraldTimes
PartsUnlimitedflubspaychecks,localunionleadercallsfailure‘unconscionable’
AutomotivepartssupplierPartsUnlimitedhasfailedtoadequatelycompensateitsworkers,withsomeemployeesreceivingnopayatall,accordingtoaPartsUnlimitedinternalmemo.Thelocallyheadquarteredcompanyadmittedthatithadfailedtoissuecorrectpaycheckstosomeofitshourlyfactoryworkersandthatothershadn’treceivedanycompensationfortheirwork.PartsUnlimiteddeniesthattheissueisconnectedtocashflowproblemsandinsteadattributestheerrortoapayrollsystemfailure.
Theoncehigh-flying$4billioncompanyhasbeenplaguedbyflaggingrevenueandgrowinglossesinrecentquarters.Thesefinancialwoes,whichsomeblameonafailureofuppermanagement,haveledtorampantjobinsecurityamonglocalworkersstrugglingtosupporttheirfamilies.
Accordingtothememo,whateverthecauseofthepayrollfailure,employeesmighthavetowaitdaysorweekstobecompensated.
“Thisisjustthelatestinalongstringofmanagementexecutionmisstepstakenbythecompanyinrecentyears,”accordingtoNestorMeyersChiefIndustryAnalystKellyLawrence.
PartsUnlimitedCFODickLandrydidnotreturnphonecallsfromtheHeraldTimesrequestingcommentonthepayrollissue,accountingerrorsandquestionsofmanagerialcompetency.
InastatementissuedonbehalfofPartsUnlimited,Landryexpressedregretatthe“glitch,”andvowedthatthemistakewouldnotberepeated.
TheHeraldTimeswillcontinuetopostupdatesasthestoryprogresses.
Tootiredtodoanythingmore,Iturnoffthelights,makingamentalnotetomyselftofindDicktomorrowtoapologizeinperson.Iclosemyeyesandtrytosleep.
AnhourlaterI’mstillstaringattheceiling,verymuchawake.
Chapter4•Wednesday,September3
IdrinkmycoffeeasIopenupmylaptopat7:30a.m.,hopingtogetthroughmye-mailsandvoicemailsbeforemy8a.m.meeting.Istareatthescreen.Inthetwenty-twohourssinceIwaspromoted,526newe-mailshavearrivedinmyinbox.
Holycrap.
Iskipallthemessagesaboutyesterday’sfailureandamstartledbyallthecongratulatorynotesfromvendors,wantingtomeetforlunch.Howdidtheyfindout?I’mprettysuremostofmyorganizationstilldoesn’tknow.
Ireadane-mailfromEllen,myformerboss’assistant,whoisnowassignedtosupportme,congratulatingmeandaskingwhenwecanmeet.Ireply,tellingherI’dliketotakeheroutforcoffeethismorning.IsendanotetotheITservicedesk,requestingthatEllenbegrantedaccesstomycalendar.
Ablinkingredlightonmydeskphonecatchesmyattention.Itreads,“7:50a.m.62newvoicemails.”
Myjawdrops.ItwouldtakeanhourIdon’thavejusttolistentothem.Ie-mailEllenagain,askinghertogothroughallmyvoicemails,transcribinganythatrequireaction.
BeforeIhitsend,Iquicklyadd,“IfthereareanymessagesfromSteveorDick,pleasecallmerightawayonmycellphone.”
Grabbingmyclipboard,Ihurrytowardmyfirstmeetingwhenmyphonevibrates.It’sanurgente-mail:
From:SarahMoulton
To:BillPalmer
Cc:SteveMasters
Date:September3,7:58AM
Priority:Highest
Subject:LatestPhoenixslip
Bill,asyouknow,ProjectPhoenixisthemostimportantprojectthiscompanyisundertaking.I’vehearddisturbing
rumorsthatyouareholdinguptherelease.
Idon’tneedtoremindyouthatourcompetitionisn’tstandingstill.Eachdaythatgoesby,ourmarketsharegoesdown.Ineedeveryonetohaveasenseofurgency.Especiallyfromyou,Bill.
Wehaveanemergencyprojectmanagementmeetingat10AMtoday.Pleasejoinus,andbepreparedtoexplaintheseunacceptabledelays.
Steve,Iknowhowimportantthisprojectisforyou,giventhecommitmentsyou’vemadetotheboard.Pleasefeelfreetoattend.We’dloveyourperspective.
Regards,
Sarah
Goddamnit.
Iforwardthee-mailtoWesandPatty,flaggingitashighpriority.Somethingseemswronginaworldwherehalfthee-mailmessagessentareurgent.Caneverythingreallybethatimportant?
IcallWes’cellphone.“Ijustgotyoure-mailfromSarah,”hesays.“Whatutterbullshit.”
“What’sthisallthisabout?”Iask.
Hesays,“I’mprettysureit’saboutBrentnotfinishingupthatconfigurationworkforthePhoenixdevelopers.Everyoneischasingtheirtailsbecausethedeveloperscan’tactuallytelluswhatthetestenvironmentshouldlooklike.We’redoingourbest,buteverytimewedeliversomething,theytelluswediditwrong.”
“Whendidtheytellusaboutit?”Iask.
“Twoweeksago.It’sthetypicalbullshitwithDevelopment,butworse.They’resofreakedoutabouthittingtheirdeadlines,they’reonlynowstartingtothinkabouthowtotestanddeployit.Apparently,they’remakingitourproblem.Ihopeyou’rewearingyourasbestosunderwearlikeme.Sarahisgoingtobeatthatmeetingwithtorches,wantingtothrowusontothebonfire.”
It’samazingtomehowhandoffsbetweenDevelopmentandITOperationsalwaysgetscrewedup.Butgiventheperpetualtribalwarfarebetweenthetwo
groups,maybeIshouldn’tbesurprised.
Ireply,“Igetthepicture.Look,makesureyoudigintothisDevspecificationissuepersonally.We’vegottogetthisnaileddown—grabeveryoneinvolved,whetherthey’reinDevorOps,andlocktheminaroomuntiltheycomeupwithawrittenspecification.Phoenixissoimportant,wecan’taffordtoscrewthisup.”
Wessayshe’sonit,andIask,“IsthereanythingelseSarahcouldpoponus?”
Hepausestothinkandfinallysays,“No,Idon’tthinkso.Wehaveaprettyvalidreason,withthepayrollrunfailure,forwhyBrentwasn’tabletocompletehiswork.”
Iagree.Feelinglikeourassesaresufficientlycovered,Isay,“Seeyouatten.”
Lessthananhourlater,I’mwalkingtoBuilding9inthehotsunshine,wheremanyoftheMarketingfolkscallhome.Tomysurprise,IjoinasmallarmyofITpeoplewalkingthesameway.Why?
Thenithitsme.Themajorityofourmarketingprojectscan’tbedonewithoutIT.Hightouchmarketingrequireshightech.Butifthere’ssomanyofusassignedtotheseMarketingprojects,shouldn’ttheybecomingtous?
IimaginethatSarahlikesitthisway,thespidersittingback,enjoyingseeingallthecompanyminionsmakingtheirwaytoherlair.
IarriveandimmediatelyseeKirstenFingle,whorunstheProjectManagementOfficesittingattheheadofthetable.Iamabigfanofhers.Sheisorganized,levelheaded,andasticklerforaccountability.Whenshefirstjoinedthecompanyfiveyearsago,shebroughtawholenewlevelofprofessionalismtoourorganization.
Atherright,Sarahleansbackinherchair,tappingawayonheriPhone,oblivioustotherestofus.
Sarahismyage:thirty-nine.She’sveryguardedaboutherage,alwayssayingthingsinawaythatwouldleadonetoconcludeshe’smucholder,butneveractuallylying.
YetanothermaddeningthingaboutSarah.
Thereareabouttwenty-fivepeopleintheroom.Manyofthebusinesslineownersarehere,someofwhomworkforSarah.ChrisAllersisalsohere.
Chrisisalittleolderthanmeandlooksleanandfit.He’lljustasoftenbeseenjokingwithsomeoneaskickingtheirassaboutmissingadeadline.Hehasareputationasacapableandno-nonsensemanager.Withnearlytwohundreddevelopersworkingforhim,heneedstobe.
TohelpwithPhoenix,histeamhasgrownbyfiftypeopleinthelasttwoyears,manythroughoffshoredevelopmentshops.Chrisisconstantlyaskedtodelivermorefeaturesanddoitinlesstime,withlessmoney.
Severalofhismanagersareintheroom,too.Wesisalsohere,sittingrightnexttoChris.AsIstarttolookforanopenchair,Inotehoweveryoneseemsunusuallytense.AndthenIseewhy.
There,sittingrightnexttotheonlyopenseatatthetable,isSteve.
Everyoneseemstobegoingtogreatlengthstonotstareathim.AsIcasuallytakemyseatnexttoSteve,myphonevibrates.It’satextmessagefromWes:
Shit.Stevehasneverattendedaprojectmanagementmeeting.Wearetotallyscrewed.
Kirstenclearsherthroat.“FirstonouragendaisPhoenix.Thenewsisn’tgood.Thisprojectwentfromyellowtoredaboutfourweeksago,andit’smypersonalassessmentthatthedeadlineisingravejeopardy.”
Shecontinuesinherprofessionalvoice,“Torefreshyourmemory,lastweekthereweretwelvetasksinthecriticalpathofPhoenixPhase1.Onlythreeofthosetaskswerecompleted.”
Thereisacollectivegroanintheroom,andseveralpeoplemuttertooneanother.Steveturnstolookatme.“Well?”
Iexplain,“ThecriticalresourceinquestionisBrent,whohasbeenonehundredpercentutilizedhelpingtorecoverfromthepayrollfailure,whichweallknowabout.Thiswasatotallyunforeseenemergencybutobviouslyonethatwehadtohandle.EveryoneknowshowimportantPhoenixis,andwearedoingeverythingwecantomakesureBrentcanstayfocused.”
“Thanksforthatsupercreativeexplanation,Bill,”Sarahimmediatelyresponds.“Therealissuehereisthatyourpeopledon’tseemtograsphowimportantPhoenixistothecompany.Ourcompetitioniskillingusinthemarket.You’veallseenandheardthecommercialsabouttheirnewservices.They’rebeatingusoninnovation,bothintheretailstoresandonline.They’vealreadyluredawaysomeofourbiggestpartners,andoursalesforceis
startingtopanic.I’mnotthetypetosay,‘Itoldyouso,’buttheirlatestproductannouncementshowswhywecan’tbeactingasifthisisjustbusinessasusual.”
Shecontinues,“See,Bill,inorderforustoincreasemarketshare,wemustshipPhoenix.Butforsomereason,youandyourteamkeepdraggingyourfeet.Maybeyou’renotprioritizingcorrectly?Ormaybeyou’rejustnotusedtosupportingaprojectthisimportant?”
Despiteallmymentalpreparation,Ifeelmyfacegethotwithanger.MaybeitwasthecondescendingwayshewasparrotingStevetome.Orhowshewasn’tevenlookingatmewhileshewasaddressingme,insteadlookingatStevetoseehowhereacts.Orthewayshebasicallycalledmeout-of-touchandincompetent.
EveryoneissilentasIforcemyselftotakeadeepbreath.
Myangerdissipates.Thisisalljustcorporatetheater.Idon’tlikeitbutacceptitforwhatitis.IalmostmadetheMarinesmycareerwhenIwasupforpromotiontostaffsergeant.Youdon’tbecomeaseniorNCOintheMarineswithoutbeingabletoplaypolitics.
“Interesting,”IsaytoSarah.“Youtellmewhichismoreimportant:gettingourfactoryemployeespaidorgettingthePhoenixtasksdone?Stevetoldmetoresolvethepayrollfailure.HowwouldyouhaveprioritizedthisdifferentlythanSteve?”
AtmymentionofSteve,Sarah’sexpressionchanges.“Well,maybeifITdidn’tcausethefailureinthefirstplace,youwouldn’thaveblownyourcommitmentstous.Idon’tthinkwecandependonyouandyourteam.”
Inodslowly,nottakingthebait.“Ilookforwardtoanysuggestionsyouhavetooffer,Sarah.”
Shelooksatme,thenatSteve.Apparentlydecidingtherearenomorepointstobegainedhere,sherollshereyes.IseeWesshakinghisheadindisbeliefatthisdiscussion,stayinguncharacteristicallyquiet.
Sarahcontinues,“We’vespentover$20milliononPhoenix,andwe’renearlytwoyearslate.Wemustgettomarket.”LookingoveratChris,sheasks,“GiventhedelaysfromBill’sgroup,whenisthesoonestwecangolive?”
Chrislooksupfromhispapers.“I’velookedintothissincewetalkedlastweek.IfweexpeditesomethingsandifthevirtualizedenvironmentsfromBill’steamworkasexpected,wecangointoproductiononeweekfrom
Friday.”
IgapeatChris.Hejustmadeupanarbitrarydatetogointoproduction,withcompletedisregardforallthethingsweneedtodobeforedeployment.
Ihaveasuddenflashback.IntheMarines,wehadaritualforalltheseniorNCOs.We’dhangoutwithbeersandwatchStarWars:ReturnoftheJedi.EverytimeAdmiralAckbarwouldcry,“It’satrap!”we’dalllaughuproariously,yellingforareplay.
Thistime,I’mnotlaughing.
“Nowjustwaitagoddamnedminutehere!”Wesinterjects,poundingthetable.“Whatthehellareyoutryingtopull?WejustfoundouttwoweeksagoaboutthespecificsofthePhoenixdeployment.Yourguysstillhaven’ttolduswhatsortofinfrastructureweneed,sowecan’tevenorderthenecessaryserverandnetworkinggear.Andbytheway,thevendorsarealreadyquotingusthree-weekdeliverytimes!”
HeisnowfacingChris,pointingathimangrily.“Oh,andI’veheardthattheperformanceofyourcodeissoshitty,we’regoingtoneedthehottest,fastestgearoutthere.You’resupposedtosupport250transactionspersecond,andyou’rebarelydoingevenfour!We’regoingtoneedsomuchhardwarethatwe’llneedanotherchassistoputitallinandprobablyhavetopayacustom-manufacturingfeetogetitintime.Godknowswhatthiswilldotothebudget.”
Chriswantstorespond,butWesisrelentless.“Westilldon’thaveaconcretespecificationofhowtheproductionandtestsystemsshouldbeconfigured.Oh,doyouguysnotneedatestenvironmentanymore?Youhaven’tevendoneanyrealtestingofyourcodeyet,becausethatfellofftheschedule,too!”
Myheartlurchesasalltheimplicationssinkin.I’veseenthismoviebefore.Theplotissimple:First,youtakeanurgentdate-drivenproject,wheretheshipmentdatecannotbedelayedbecauseofexternalcommitmentsmadetoWallStreetorcustomers.Thenyouaddabunchofdeveloperswhouseupallthetimeintheschedule,leavingnotimefortestingoroperationsdeployment.Andbecausenooneiswillingtoslipthedeploymentdate,everyoneafterDevelopmenthastotakeoutrageousandunacceptableshortcutstohitthedate.
Theresultsareneverpretty.Usually,thesoftwareproductissounstableandunusablethateventhepeoplewhowerescreamingforitendupsayingthatit’snotworthshipping.Andit’salwaysITOperationswhostillhastostay
upallnight,rebootingservershourlytocompensateforcrappycode,doingwhateverheroicsarerequiredtohidefromtherestoftheworldjusthowbadthingsreallyare.
“Guys,IunderstandthedesiretogetPhoenixintoproductionasquicklyaspossible,”IsaytoSteveandChrisascalmlyasIcan.“Butbasedonwhatwe’veheardfromWes,Ithinkitisincrediblyprematuretodeploy.Westilldon’tknowwhatequipmentweneedtohittheperformanceobjectives,norhavewedoneanycapacitytestingtoconfirmourguesses.It’sunlikelywehaveadequatedocumentationtorunthisthinginproduction,letalonegeteverythingmonitoredandbackedup.”
Inmymostpersuasivevoice,Icontinue,“IwantPhoenixinthemarketasbadlyasanyoneelse,butiftheuserexperienceisbadenough,we’llendupdrivingourcustomerstothecompetition.”
IturntoChris.“Youcan’tjustthrowthepigoverthewalltous,andthenhigh-fiveeachotherintheparkinglot,congratulatingyourselvesonhowyoumadethedeadline.Wesistellingusthatthepigwillprobablybreakitsleg,andit’llbemyguyswhoworkall-nightersandweekendstokeepthatpigalive.”
Chrisreplieshotly,“Don’tgivemethatbullshitabout‘throwingthepigoverthewall.’Weinvitedyourpeopletoourarchitectureandplanningmeetings,butIcancountononehandthenumberoftimesyouguysactuallyshowedup.Weroutinelyhavehadtowaitdaysorevenweekstogetanythingweneedfromyouguys!”
Thenhejustholdsuphishands,asifeverythingisoutsideofhiscontrol.“Look,I’dlikemoretime,too.Butfromtheverybeginning,weallknewthatthiswasadate-drivenproject.Thatwasabusinessdecisionweallmade.”
“Exactly!”SarahexclaimsbeforeIcanrespond.“ThisjustshowshowBillandhisteamlackthenecessarysenseofurgency.Perfectionistheenemyofgood.Bill,wesimplydonothavetheluxuryoftimetopolishthistowhatevergoldstandardyou’reproposing.Weneedtocreatepositivecashflow,andwecannotdothatwithouttakingbackmarketshare.Andtodothat,weneedtodeployPhoenix.”
ShelooksoveratSteve.“Weunderstandrisk,don’twe,Steve?You’vebeendoinganabsolutelyamazingjobsellingthistoanalystsandeventheguysonCNBC—Idon’tthinkwewanteggonourfacebyshippingevenlaterthanwealreadyare.”
Stevenodshisheadandrubshischin,rockingbackandforthinhischair
ashethinks.“Agreed,”hefinallysays,leaningforward.“We’vemadecommitmentstoourinvestorsandanalyststhatweweregoingtolaunchPhoenixthisquarter.”
Myjawdrops.Sarahhasbluntedallmyarguments,leadingStevedownareckless,destructivepath.
Exasperated,Isay,“Doesanyonethinkthisisreallyodd?I’vebeeninthisroomwhenwediscussedinstallingnewwaterfountainsinthefrontofeverystore.Wegavethatteamninemonthstoplantherollout.Ninemonths!Andallofusagreedthatwasreasonable.
“Nowwe’retalkingaboutPhoenix,whichimpactsthousandsofpointofsalesystems,andalloftheback-officeorderentrysystems.Thisisatleasttenthousandtimesmorecomplexthanrollingoutnewwaterfountains,withwaymorerisktothebusiness.Andyou’reonlygivingusoneweektoplanandexecutetherollout?”
Ithrowmyhandsup,imploringSteve,“Doesn’tthisseemabitrecklessandunfair?”
Kirstennods,butSarahsaysdismissively,“Bill,that’satouchingstorybutwe’renotdiscussingwaterfountains,we’rediscussingPhoenix.Besides,Ibelievethedecisionhasalreadybeenmade.”
Stevesays,“Yes,ithas.Thankyouforsharingwhatyouviewastherisks,Bill.”HeturnstoSarah.“Whenisthelaunchdate?”
Sarahrepliesquickly,“MarketinglaunchisnextSaturday,September13.Phoenixwilldeployat5p.m.thepreviousday.”
Stevewritesthedateinthebackofhisnotebookandsays,“Good.Keepmepostedonprogress,andletmeknowifthere’sanythingIcandotohelp.”
IlookoveratWes,whomimeswithhishandsanairplanecrashingintothetableinfrontofhimandburstingintoflames.
Inthehallway,Wessays,“Ithoughtthatwentprettywell,boss.”
I’mnotlaughing.“Whatthehellhappenedinthere?Howdidwegetintothisposition?Doesanyoneknowwhat’srequiredfromustosupportthislaunch?”
“Noonehasadamnedclue,”hesays,shakinghisheadindisgust.“Wehaven’tevenagreedonhowtodothehandoffwithDevelopment.Inthepast,they’vejustpointedtoanetworkfolderandsaid,‘Deploythat.’Thereare
newbornbabiesdroppedoffatchurchdoorstepswithmoreoperatinginstructionsthanwhatthey’regivingus.”
Ishakemyheadathisawfulimagery,buthe’sright.We’vegotaseriousproblemhere.
Hecontinues,“We’regoingtohavetoassembleahugeteam,includingChris’guys,tofigureouthowwe’regoingtopullthisoff.Wehaveproblemsateverylayer:networking,servers,databases,operatingsystems,applications,Layer7switching—thewholewadofcrap.It’sgoingtobelatenightsforallofusforthenextninedays.”
Inodunhappily.Thistypeofall-handseffortisjustanotherpartoflifeinIT,butitmakesmeangrywhenweneedtomakesomeheroic,divingcatchbecauseofsomeoneelse’slackofplanning.
Isay,“Getyourteamassembled,andaskChristoassemblehisrespectiveteamaswell.Stoptryingtodothisbye-mailandintheticketingsystem.Weneedeveryoneinthesameroom.”
“Speakingofcommitments,”Isay,“WhatwasChrisreferringtowhenhesaidthatourguysnevershoweduptothePhoenixarchitectureandplanningmeetings?Isthattrue?”
Wesrollshiseyesinfrustration.“Yeah,it’struethathispeoplewouldinviteusatthelastminute.Seriously,whocancleartheircalendaronlessthanaday’snotice?”
“Although,infairness,”hesays,afteramoment,“wedidgetamplenoticeonacoupleofthebigplanningmeetings.Andoneofthemostcriticalpeoplewhoneededtobetherewasn’tabletomakeit,duetoescalations.Youcanprobablyguesswhothatis…”
Igroan.“Brent?”
Wesnods,“Yep.He’stheguyweneedatthosemeetingstotellthosegoddamneddevelopershowthingsworkintherealworldandwhattypeofthingskeepbreakinginproduction.Theirony,ofcourse,isthathecan’ttellthedevelopers,becausehe’stoobusyrepairingthethingsthatarealreadybroken.”
He’sright.Unlesswecanbreakthiscycle,we’llstayinourterribledownwardspiral.Brentneedstoworkwithdeveloperstofixissuesatthesourcesowecanstopfightingfires.ButBrentcan’tattend,becausehe’stoobusyfightingfires.
Isay,“Weneedourbestmindstoprepareforthisdeployment,somakesureBrentisthere.”
Weslookssheepishforamoment.Iaskhim,“What?”
“Ithinkhe’sworkinganetworkoutagerightnow.”hereplies.
“Notanymore,”Isay.“They’regoingtohavetofixitwithouthim.Ifsomeonehasaproblemwiththat,sendthemtome.”
“Okay,whateveryouwant,boss.”hesays,shrugginghisshoulders.
Aftertheprojectmanagementmeeting,I’minnomoodtotalktoanyone.Isitatmydeskandgrumblewhenmylaptopdoesn’twakeup.Thediskdrivelightjustkeepsblinking.Whennothingshowsonthescreen,IgrabmyemptymugthatIkeeponmydeskbythepictureofPaigeandmytwosonsandwalktothecoffeemachineinthecorner.
WhenIgetbacktomydesk,awindowonthescreentellsmethatit’sgoingtoinstallsomecriticalnewupdates.Isitdown,click“OK”andwatchthestatusbarcrawlacrossthescreen.Suddenly,Iseethedreaded“bluescreenofdeath.”Mylaptopisnowcompletelylockedupandunusable.
IthappensagainevenafterIreboot.Imutterinfrustration,“You’vegottobekiddingme!”
Justthen,Ellen,mynewassistant,pokesherheadaroundthecorner.Holdingoutherhandshesays,“Goodmorning.Congratulationsonthepromotion,Bill!”Noticingmyblue-screenedlaptop,shesayssympathetically,“Ooh,thatdoesn’tlookgood.”
“Umm,thanks.”Isay,reachingouttoshakeherhand.“Yeah,aboutthislaptop,canyougetaholdofsomeoneindesktopsupport?There’ssomeseriouscrapheadedourwayfromPhoenix,andI’mgoingtoneedit.”
“Noproblem,”shesays,noddingwithasmile.“I’lltellthemournewVPishoppingmad,demandingthathislaptopgetfixed.Ofallpeople,youneedaworkingcomputer,right?
“Youknow,”sheadds,“I’veheardthatabunchofotherpeoplearehavingproblemslikethistoday.I’llmakesureyougettothetopofthelist.Youcan’taffordtowaitinline.”
Morebrickedlaptops?Thisissurelyevidencethattheuniverseisouttogetmetoday.
“Bytheway,IneedsomehelpcoordinatingsomeemergencyPhoenix
meetings.Hasanyonegrantedyouaccesstomycalendaryet?”Iask.
Sherollshereyes.“No.That’sactuallywhyIcamedownhere.Iwantedtoseeifyoucouldprintoutyournextcoupleofdays.Obviously,that’soutofthequestion.I’llhavethedesktopsupportpersondothatwhilehe’shere.Sometimesittakesweeksforthee-mailadministratorstogetaroundtostufflikethis.”
Weeks?That’sunacceptable.IquicklylookatmywatchandrealizeI’llhavetotacklethislater.I’malreadylate.
“Doyourbest,”Isay.“I’mofftoPatty’senterprisechangemanagementmeeting.Callmeifyouneedanything,okay?”
BeingtenminuteslatetoPatty’smeeting,Ihurryintotheroom,expectingtoseeeitherabunchofpeoplewaitingformeimpatientlyorperhapsameetingalreadyunderway.
Instead,IseeonlyPattysittingattheconferencetable,typingawayonherlaptop.
“WelcometotheCAB,Bill.Ihopeyoucanfindanemptychair,”shesays.
“Whereiseverybody?”Iask.
I’mbaffled.WhenIranthemidrangegroup,myteamwouldnevermissourchangemanagementmeetings.Itwaswherewecoordinatedandorganizedallourworktomakesurewedidn’tshootourselvesinthefoot.
“Itoldyouyesterdaythatchangemanagementaroundhereishit-or-miss,”Pattysays,sighing.“Somegroupshavetheirownlocalchange-managementprocess,likeyours.Butmostgroupsdonothingatall.Yesterday’soutageisjustproofthatweneedtohavesomethingattheenterpriselevel.Rightnow,thelefthandrarelyknowswhattherighthandisdoing.”
“So,what’stheproblem?”Iask.
Shepursesherlip.“Idon’tknow.WesentabunchofpeopletoITILtraining,sotheycouldgetuptospeedonallthebestpractices.Webroughtinsomeconsultants,whohelpedusreplaceourticketingsystemwithanITIL-compliantchangemanagementtool.Peopleweresupposedtoputchangerequestsintoit,whereitwouldgetroutedforapprovals.But,evenaftertwoyears,allwehaveisagreatprocessonpaperthatnoonefollowsandatoolthatnooneuses.WhenIpesterpeopletousethem,allIgetarecomplaintsandexcuses.”
Inod.ITILstandsforITInfrastructureLibrary,whichdocumentsmanyIT
bestpracticesandprocesses,andtheITILprogramhashadareputationofspendingyearsmerelywalkingincircles.
I’mbotheredthatWesisn’there.Iknowhe’sbusy,butifhe’snothere,whywouldanyofhispeoplebothertoshowup?Effortslikethismuststartandbecontinuallymaintainedfromthetop.
“Well,theycanbringtheircomplaintsandexcusestome,”Isayadamantly.“We’rerebootingthechangemanagementprocess.Withmytotalsupport.Steve’stoldmetomakesurepeoplecanstayfocusedonPhoenix.ScrewupsliketheSANfailuremadeusmissaPhoenixdeliverable,andnowwe’repayingforit.Ifsomeonewantstoskipachangemanagementmeeting,theyobviouslyareinneedofsomespecialcompassionatecoaching.Fromme.”
AtPatty’spuzzledexpressionatmyPhoenixreference,ItellherabouthowWesandIspentourmorningbeingrunoverbythebus.SarahandChriswereatthewheel,butStevewasinback,cheeringthemontofloorit.
“Notgood,”shesays,disapprovingly.“TheyevenranoverKirsten,huh?”
Inodsilentlybutrefusetosaymore.IalwayslikedthatphraseinSavingPrivateRyan:“There’sachainofcommand:gripesgoup,notdown.”
Instead,Iaskhertowalkmethroughthecurrentchangeprocessandthewayit’sbeenautomatedinthetools.Itallsoundsgood.Butthere’sonlyonewaytoseeiftheprocessworks.
Isay,“ScheduleanotherCABmeetingforthesametimeFriday.I’llsendoutane-mailtoalltheCABmemberslettingthemknowthatthisismandatory.”
WhenIgetbacktomycubicle,Ellenisatmydesk,bendingovermylaptop,writinganote.
“Everythingworking,Ihope?”Iask.
Shestartlesatthesoundofmyvoice.“Oh,myGod.Youscaredme,”shesayslaughing.“Supportleftyouareplacementlaptopbecausetheycouldn’tgetyourlaptoptoboot,evenafterahalfhouroftrying.”
Shepointsatthefarsideofmydesk,andIdoadoubletake.
Myreplacementlaptopappearstobealmosttenyearsold—it’stwiceaslargeasmyoldoneandlooksthreetimesasheavy.Thebatteryhasbeentapedon,andhalfthekeyboardletteringiswornofffromheavyuse.
Foramoment,Iwonderifthisisapracticaljoke.
Isitdownandbringupmye-mail,buteverythingissoslowthatseveraltimesIthoughtithadlockedup.
Ellenhasasympatheticexpressiononherface.“Thesupportguysaidthatthisisalltheyhaveavailabletoday.Overtwohundredpeoplearehavingsimilarproblems,andmanyaren’tgettingreplacements.Apparently,peoplewithyourlaptopmodelalsohavehadtheir’sbreakbecauseofsomesecuritypatch.”
Damn.Iforgot.It’sPatchTuesday,whenJohnandhisteamrolloutalltheirsecuritypatchesfromourmajorvendors.Onceagain,Johniscausinghugeissuesanddisruptionsformyteamandme.
Imerelynodandthankherforthehelp.Aftershe’sgone,Isitdownandtypeoutane-mailtoalltheCABmembers,mykeystrokesoftentakingtensecondstoshowuponthescreen.
From:BillPalmer
To:WesDavis,PattyMcKee,ITOperationsManagement
Date:September3,2:43PM
Priority:Highest
Subject:MandatoryCABmeetingFriday,2PM
Today,IattendedtheweeklyCABmeeting.IwasextremelydisappointedthatIwastheonlyonethere,besidesPatty,especiallygiventhetotallyavoidable,change-relatedfailureyesterday.
Effectiveimmediately,managers(ortheirassigneddelegates)arerequiredtoattendallscheduledCABmeetingsandtoperformtheirassignedduties.WeareresurrectingthePartsUnlimitedchangemanagementprocessanditwillbefollowedtotheletter.
Anyperson(s)caughtcircumventingchangemanagementwillbesubjecttodisciplinaryaction.
TherewillbeamandatoryCABmeetingFridayat2PM.Seeyouthere.
Callmeifyouhaveanyquestionsorconcerns.
Thanksforyoursupport,
Bill
Ihitsend,waitingfifteensecondsforthee-mailtofinallyleavemyoutbox.Almostimmediately,mycellphonerings.
It’sWes.Isay,“Iwasjustabouttocallyouaboutthelaptops.We’vegottogetreplacementstoourmanagersandemployeessotheycandotheirjobs,youhear?”
“Yeah,we’reonit.ButI’mnotcallingaboutthat.AndI’mnotcallingaboutPhoenix,either,”hesays,soundingirritated.“Look,aboutyourmemoonchangemanagement:Iknowyou’retheboss,butyoubetterknowthatthelasttimewedidoneofthesechangemanagementkumbayas,weranITstraightintotheground.Noone,andImeanabsolutelynoone,couldgetagoddamnedthingdone.Pattyinsistedonhavingeveryonetakeanumberandwaitforherpointy-headstoauthorizeandscheduleourchanges.Itwasabsolutelyridiculousandatotalwasteoftime.”
He’sunstoppable:“Thatsoftwareapplicationshemadeususeisatotalpieceofcrap.Ittakestwentyminutestofilloutallthegoddamnedfieldsforasimplefive-minutechange!Idon’tknowwhodesignedtheprocess,butIthinktheyassumedthatweallgetpaidbythehourandwanttotalkaboutdoingworkinsteadofactuallydoingwork.
“Eventually,theNetworkingandServerTeamstagedarebellion,refusingtousePatty’stool,”hecontinuesheatedly.“ButJohnwavedanauditfindingaroundandwenttoLuke,ouroldCIO.Andjustlikeyoudid,Lukesaidthatfollowingpolicieswasaconditionofemployment,threateningtofireanybodywhodidn’tfollowthem.
“MyguyswerespendinghalftheirtimedoingpaperworkandsittinginthatdamnedCABmeeting,”hecontinues.“Luckily,theeffortfinallydied,andJohnwastoocluelesstocatchonthatnoonewasactuallygoingtothemeetingsanymore.EvenJohnhasn’tgonetooneofthosemeetingsinoverayear!”
Interesting.
“Ihearyou,”Isay.“Wecan’trepeatthat,butwealsocan’thaveanotherpayrolldisaster.Wes,Ineedyouthere,andIneedyoutohelpcreatethesolution.Otherwise,you’repartoftheproblem.CanIcountonyou?”
Ihearhimsighloudly.“Yeah,sure.Butyoucanalsocountonmecalling‘bullshit’ifIseePattytryingtocreatesomesortofbureaucracythatsucksouteverybody’swilltolive.”
Isigh.
Before,IwasmerelyworriedthatITOperationswasunderattackbyDevelopment,InformationSecurity,Audit,andthebusiness.Now,I’mstartingtorealizethatmyprimarymanagersseemtobeatwarwitheachother,aswell.
Whatwillittakeforustoallgetalong?
Chapter5•Thursday,September4
Iwakeupwithajoltwhenthealarmclockgoesoffat6:15a.m..Myjawstillhurtsfromclenchingitallnight.ThedismalprospectsoftheupcomingPhoenixlaunchwereneverfarfrommymind.
Asusual,beforeclimbingoutofbed,Iquicklyscanmyphoneforanybadnews.Usually,Iwouldspendabouttenminutesreplyingtoe-mails—italwaysfeelsgoodtolobacoupleofballsoffmysideofthecourt.
IseesomethingthatmakesmeboltuprightsoabruptlythatIwakeupPaige.“Oh,myGod.What,what?”sheasksfrantically,notfullyawake.
“It’sanothere-mailfromSteve.Hangon,darling…”Isaytoher,whileIsquinttoreadit.
From:SteveMasters
To:BillPalmer
Cc:NancyMailer,DickLandry
Date:September4,6:05AM
Priority:Highest
Subject:URGENT:SOX-404ITAuditFindingsReview
Bill,pleaselookintothisASAP.Idon’tneedtotellyouhowcriticalitistohaveacleanSOX-404audit.
Nancy,pleaseworkwithBillPalmer,whoisnowinchargeofITOperations.
Steve
>>>Beginforwardedmessage:
WejustconcludedourQ3internalauditinpreparationfortheupcomingSOX-404externalaudit.Wediscoveredsomeveryconcerningdeficienciesthatweneedtodiscusswithyou.Duetotheseverityandurgencyofthefindings,weneedtomeetwithITthismorning.
Nancy
Indeed,there’satwo-hourmeetingscheduledfor8a.m.onmycalendar,
setupbyNancyMailer,ChiefAuditExecutive.
Holycrap.Sheisincrediblysmartandformidable.Yearsagoduringtheretailacquisitionintegration,Iwatchedhergrillamanagerfromthebusinesswewereacquiring.Hewaspresentingtheirfinancialperformance,whenshestartedarapid-fireinterrogation,likeacrossbetweenColumbo,Matlock,andScarface.
Hequicklybroke,admittingthathewasexaggeratinghisdivision’sperformance.
Recallingthatmeeting,myarmpitsfeeldamp.Ihaven’tdoneanythingwrong.Butgiventhetoneofthee-mail,sheisclearlyhotonthetrailofsomethingimportant,andStevejustthrewmeinherpath.
I’vealwaysrunaverytightshipinmyMidrangeTechnologygroup.ThiskeptAuditfrominterferingtoomuch.Sure,therewouldstillbealotofquestionsanddocumentationrequests,requiringustospendafewweekscollectingdataandpreparingresponses.Occasionally,theywouldfindsomething,butwewouldquicklyfixit.
Iliketothinkthatwebuiltamutuallyrespectfulworkingrelationship.However,thise-mailportendssomethingmoreominous.
Ilookatmywatch.Themeetingisinninetyminutes,andIdon’thaveaclueaboutwhatshewantstotalkabout.
“Shit!”Iexclaim,asIjostlePaige’sshoulder.“Honey,canyoudrivethekidsintoschooltoday?SomethingreallybadjustcameupinvolvingtheChiefAuditExecutiveandSteve.Ineedtomakesomephonecallsandgettotheofficerightaway.”
Annoyed,shesays,“Fortwoyearsyou’vealwaystakenthekidsonThursdays!Ihaveanearlystarttoday,too!”
“I’msorry,honey.Thisisreallyimportant.TheCEOofthecompanyaskedmetohandlethis.SteveMasters.Youknow,theguyonTVandwhogivesthebigspeechesatthecompanyholidayparty?Ican’tdropanotherballafteradaylikeyesterday.Andthenewspaperheadlinethenightbeforethat—”
Withoutaword,shestormsdownthestairs.
WhenIfinallyfindtheconferenceroomforthe8a.m.meeting
Top Related