Lecture2Morelogistics

Divide-and-conquer,MergeSort,andBig-Onotation

Somequestionsfromlasttime• What’sthebestwaytotakenotes?• Iwillpostslidesaheadoftime.• Sometimestheslideswillrefertoexplanationsontheboard– Iencourageyoutotakenotesforthatstuff.

• What’swiththewaitlist?• WegotanotherTA!(DannyWright).Enrollmentwillbeopeningup(stilllimited,buteveryonewhowasonthewaitlistat8amthismorningwillgetaspot).

Logisticssomemore!• Asbefore:http://web.stanford.edu/class/cs161/• Therewillbeofficehours!Andrecitationsections!

MTu WTh

(screenshotsubjecttochange)

Thedealwithsections:

• HeldonMondaysandTuesdays:goovercontentfrompreviousweek,withexampleproblems!

• Don’tneedtosignup,gotoany/eithersection!

• Problems/solutionsfrombothsectionswillbeposted!

Signupfor

Homework!We’regonna haveit!

• AssignmentwillbepostedFRIDAY.(3pm)• DuethenextFRIDAY.(3pm)• SeetheCOURSEWEBSITEforHWguidelines.• SolutionsmostthefollowingMONDAY.

MondayTuesdayWednesdayThursdayFriday

HWASSIGNED

HWDUE

SOLUTIONSRELEASED

HWRETURNED(target)

Lasttime

• Algorithmsareawesomeandpowerful!• Algorithmdesigner’squestion:CanIdobetter?• Interplaybetweenrigorand intuition.

• Karatsubaintegermultiplication• Exampleof“DivideandConquer”• Not-so-rigorousanalysis

Philosophy

TechnicalcontentPluckythepedanticpenguin

Luckythelackadaisical

lemur.

Today

• Sorting• Returnofdivide-and-conquer with MergeSort• Skills:• Analyzingcorrectnessofiterativeandrecursivealgorithms.• Analyzingrunningtimeofrecursivealgorithms.

• Howdowemeasuretheruntimeofanalgorithm?• Worst-caseanalysis• AsymptoticAnalysis

Sorting

• Importantprimitive• Fortoday,we’llpretendallelementsaredistinct.

6 4 3 8 1 5 2 7

1 2 3 4 5 6 7 8

Benchmark:insertionsort

• Saywewanttosort:

• Insertitemsoneatatime.

• Howwouldweactuallyimplementthis?

6 4 3 8 5

We’regoingtogothroughthisinsome

detail– it’sgoodpractice!

46 3 8 5

Insertionsortexample…

6 4 3 8 5

64 3 8 5

Startwiththesecondelement(thefirstelementissortedwithinitself…)

Pull“4”backuntilit’sintherightplace.

64 3 8 5 Nowlookat“3”

4 63 8 5 Pull“3”backuntilit’sintherightplace.

43 6 8 5 “8”isgood…lookat5

43 6 85 (thenfix5andwe’redone)

Insertionsortpseudocode

• (Discussiononboard)

Luckythelackadaisicallemur

Goone-at-a-timeuntilthingsareintherightplace.

Pluckythepedanticpenguin

Insertionsort:runningtime

n-1iterationsoftheouterloop

Intheworstcase,aboutniterationsofthisinnerloop

Runningtimeisaboutn2

Insertionsort:correctness

• Maintainaloopinvariant.

• Initialization:theloopinvariantholdsbeforethefirstiteration.• Maintenance:Ifitistruebeforethet’th iteration,itwillbetruebeforethe(t+1)’st iteration• Termination:Itisusefultoknowthattheloopinvariantistrueattheendofthelastiteration.

Aloopinvariantissomethingthatshouldbetrueateveryiteration.

(Thisisproof-of-correctnessbyinduction)

Insertionsort:correctness

• Loopinvariant:Atthestartofthet’th iteration(oftheouterloop),thefirstt elementsofthearrayaresorted.• Initialization:Atthestartofthefirstiteration,thefirstelementofthearrayissorted.✓• Maintenance:Byconstruction,thepointofthet’thiterationistoputthe(t+1)’st thingintherightplace.• Termination:Atthestartofthe(len(A)+1)’stiteration(aka,attheendofthealgorithm),thefirstlen(A)itemsaresorted.✓

Tosummarize

InsertionSort isanalgorithmthatcorrectlysortsanarbitraryn-element

arrayintimeaboutn2.

Canwedobetter?

Canwedobetter?

• MergeSort:adivide-and-conquer approach• Recallfromlasttime:

Bigproblem

Smallerproblem

Smallerproblem

Yetsmallerproblem

Yetsmallerproblem

Yetsmallerproblem

Yetsmallerproblem

Recurse!

DivideandConquer:

Recurse!

1

MergeSort6 4 3 8 1 5 2 7

6 4 3 8 1 5 2 7

3 4 6 8 1 2 5 7

2 3 4 5 6 7 8

Recursivemagic!Recursivemagic!

(MERGE pseudocode+analysisonboard)

MERGE!

(Agoodexercise:howwouldIdothisinplace?)

MergeSort Pseudocode

• n=length(A)• if n≤ 1:• return A

• L=MERGESORT(A[1:n/2])• R=MERGESORT(A[n/2+1:n])• returnMERGE(L,R)

MERGESORT(A):

IfAhaslength1,Itisalreadysorted!

Sorttherighthalf

Sortthelefthalf

Mergethetwohalves

WhatactuallyhappenswhenwerunMergeSort?• (onboard)

Schematicofrecursivecalls

64 3 8 1 5 2 7

1 2 5 73 4 6 8

1 2 3 4 5 6 7 8

Merge!Merge!Merge!Merge!

Merge! Merge!

Merge!

4 3 8 1 5 2 76Originalsequence

Sortedsequence!

Twoquestions

• Doesthiswork?• Isitfast?

ItworksLet’sassumen=2t

• Invariant:“Ineveryrecursivecall,

MERGESORTreturnsasortedarray.”

• n=length(A)• if n≤ 1:

• return A

• L=MERGESORT(A[1:n/2])• R=MERGESORT(A[n/2+1:n])• returnMERGE(L,R)

• Basecase(n=1):a1-elementarrayisalwayssorted.

• Maintenance:SupposethatLandRaresorted.ThenMERGE(L,R)issorted.

• Termination:“Inthetoprecursivecall,MERGESORTreturnsasortedarray.”Themaintenancestepneedsmoredetails!!Whyisthisstatementtrue?

Nottechnicallya“loopinvariant,”buta”recursioninvariant,”thatshouldholdatthebeginningofeveryrecursivecall.

It’sfastLet’skeepassumingn=2t

CLAIM:

MERGESORTrequiresatmost6nlog(n)+6noperationstosortn numbers.

Beforeweseewhy…Howdoesthatcomparetothe≈n2 operationsofINSERTIONSORT?

nlog(n)vsn2

• log(n):howmanytimesdoyouneedtodividenby2inordertogetdownto1?

32168421log(32)=5

6432168421log(64)=6

log(128)=7log(256)=8log(512)=9...log(numberofparticlesintheuniverse)<280

nlog(n)vsn2continued

n nlog(n) n^2

8 24 64

16 64 256

32 160 1024

64 384 4096

128 896 16384

256 2048 65536

512 4608 262144

1024 10240 10485760

200000

400000

600000

800000

1000000

0 200 400 600 800 1000 1200

nlog(n) n^2

It’sfast!

CLAIM:

MERGESORTrequiresatmost6nlog(n)+6noperationstosortn numbers.

Asngrows,that’smuchfasterthanthe≈n2 operationsof

INSERTIONSORT!

Analysis

T(n)=T(n/2)+T(n/2)+T(MERGE)Let’ssay

T(MERGEofsizen/2)≤ 6noperations

Wewillseelaterhowtoanalyzerecurrencerelationsliketheseautomagically…buttodaywe’lldoitfromfirstprinciples.

T(n)=timetorunMERGESORTonalistofsizen

T(MERGEtwolistsofsizen/2)isthetimetodo:

• 3variableassignments(counters←1)• ncomparisons• nmoreassignments• 2ncounterincrements

Sothat’s2T(assign)+nT(compare)+nT(assign)+2nT(increment)

or4n+2operationsPluckythe

pedanticpenguin Luckythelackadaisicallemur

=2T(n/2)+6n

Thisiscalledarecurrencerelation:itdescribestherunningtimeofaproblemofsizenintermsoftherunningtimeofsmallerproblems.

…

Recursiontree

Sizen

n/2n/2

n/4

(Size1)

…

n/4n/4n/4

n/2tn/2tn/2tn/2tn/2tn/2t

…

Level

Amountofworkatthislevel

(justMERGEing)

0

#problems

12

t

log(n)

1

2

4

2t

n

Sizeofeach

problem

n

n/2

n/4

n/2t

1

6n

6n

6n

6n

6n

Amountofworkatalevel:(numberofproblems)× 6× (sizeofproblem)

(explanationonboard)

…

Totalruntime…

• 6nstepsperlevel,ateverylevel

• log(n)+1levels

•6nlog(n)+6nstepstotal

Thatwastheclaim!

Afewreasonstobegrumpy

• Sorting

shouldtakezerosteps…• What’swiththisT(MERGE)<6n?• 2+4n<6nisaloosebound.• Differentoperationsdon’ttakethesameamountoftime.

1 2 3 4 5 6 7 8

Today

• Sorting• Returnofdivide-and-conquer with MergeSort• Skills:• Analyzingcorrectnessofiterativeandrecursivealgorithms.• Analyzingrunningtimeofrecursivealgorithms.

• Howdowemeasuretheruntimeofanalgorithm?• Worst-caseanalysis• AsymptoticAnalysis

NOTE!!!WEACTUALLYSTOPPEDHEREINCLASSONWEDNESDAY!

WE’LLPICKUPAFTERTHISNEXTTIME…

Worst-caseanalysis

• Inthisclass,wewillfocusonworst-caseanalysis

• Pros:verystrongguarantee• Cons:verystrongguarantee

1 2 3 4 5 6 7 8

Sortingasortedlistshouldbefast!!

Algorithmdesigner

Algorithm:Do the thingDo the stuffReturn the answer

Here is my algorithm!Here is an input!

Big-Onotation

• Whatdowemeanwhenwemeasureruntime?• Weprobablycareaboutwalltime: howlongdoesittaketosolvetheproblem,insecondsorminutesorhours?

• Thisisheavilydependentontheprogramminglanguage,architecture,etc.• Thesethingsareveryimportant,butarenotthepointofthisclass.• Wewantawaytotalkabouttherunningtimeofanalgorithm,independentoftheseconsiderations.

Howlongdoesanoperationtake?Whyarewebeingsosloppyabout

that“6”?

Rememberthisslide?

n nlog(n) n^2

8 24 64

16 64 256

32 160 1024

64 384 4096

128 896 16384

256 2048 65536

512 4608 262144

1024 10240 1048576 0

200000

400000

600000

800000

1000000

0 200 400 600 800 1000 1200

nlog(n) n^2

Changenlog(n)to5nlog(n)….

n 5nlog(n) n^2

8 120 64

16 320 256

32 800 1024

64 1920 4096

128 4480 16384

256 10240 65536

512 23040 262144

1024 51200 1048576 0

200000

400000

600000

800000

1000000

0 200 400 600 800 1000 1200

5nlog(n) n^2

AsymptoticAnalysisHowdoestherunningtimescaleasngetslarge?

• Abstractsawayfromhardware- andlanguage-specificissues.

• Makesalgorithmanalysismuchmoretractable.

• Onlymakessenseifnislarge(comparedtotheconstantfactors).

Pros: Cons:

Onealgorithmis“faster”thananotherifitsruntimegrowsmore“slowly”asngetslarge.

2100000000000000 nis“better”thann2 ?!?!

O(…)meansanupperbound

• Wesay“T(n)isO(f(n))”iff(n)growsatleastasfastasT(n)asngetslarge.

• Formally,𝑇 𝑛 = 𝑂 𝑓 𝑛

⟺∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,

0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)(Explanationofwhatallthesesymbolsmeanonboard)

Parsingthat…

𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺

∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)

T(n)

f(n)

cf(n)

n0n

Example1

• T(n)=n,f(n)=n2.• T(n)=O(f(n))

T(n)

n

f(n)

c=1

n0 =1

𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺

∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)

Whydoweneedcinthedefinition?

(formalproofonboard)

Example2

• g(n)=2,f(n)=1.• g(n)=O(f(n))(andalsof(n)=O(g(n)))

𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺

∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)

f(n)

n0=1n

g(n)

2.1⋅ f(n)

1

2

2.1

(Needcbutnotreallyn0)

Example3

• f(n)=1,g(n)asbelow.• g(n)=O(f(n))(andalsof(n)=O(g(n)))

𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺

∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)

f(n)

n

g(n)

2.1⋅ f(n)

1

2

2.1

n0

(Needbothcandn0)

Examples4and5

• AlldegreekpolynomialsareO(nk)• Foranyk≥ 1,nk isnot O(nk-1)

(Ontheboard)

Take-awayfromexamples

• ToproveT(n)=O(f(n)),youhavetocomeupwithcandn0sothatthedefinitionissatisfied.

• ToproveT(n)isNOT O(f(n)),onewayisbycontradiction:• Supposethatsomeonegivesyouacandann0 sothatthedefinitionissatisfied.• Showthatthissomeonemustbylyingtoyoubyderivingacontradiction.

Ω(…)meansalowerbound

• Wesay“T(n)isΩ(f(n))”iff(n)growsatmostasfastasT(n)asngetslarge.

• Formally,𝑇 𝑛 = 𝑂 𝑓 𝑛

⟺∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,

0 ≤ 𝑐 ⋅ 𝑓 𝑛 ≤ 𝑇 𝑛

Switchedthese!!

Parsingthat…

𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺

∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑐 ⋅ 𝑓 𝑛 ≤ 𝑇 𝑛

T(n)

cf(n)

f(n)

n0n

Θ(…)meansboth!

•Wesay“T(n)isΘ(f(n))”if:

T(n)=O(f(n))-AND-

T(n)=Ω(f(n))

Yetmoreexamples

• n3 +3n=O(n3 – n2)• n3 +3n=Ω(n3 – n2)• n3 +3n=Θ(n3 – n2)

• 3n isNOTO(2n)• nlog(n)=Ω(n)• log(n)=Θ(2loglog(n) )

Somebrainteasers

• Aretherefunctionsf,gsothatNEITHER f=O(g)norf=Ω(g)?• Aretherenon-decreasing functionsf,gsothattheaboveistrue?• Definethen’th fibonacci numberbyF(0)=1,F(1)=1,F(n)=F(n-1)+F(n-2)forn>2.• 1,1,2,3,5,8,13,21,34,55,…

Trueorfalse:• F(n)=O(2n)• F(n)=Ω(2n)

We’llbeusinglotsofasymptoticnotationfromhereonout

Butweshouldalwaysbecarefulnottoabuseit.

Inthecourse,(almost)everyalgorithmweseewillbeactuallypractical,withoutneedingtotake𝑛 ≥ 𝑛. = 2:........

Recap

• Divide-and-conquer paradigm• Sorting: MergeSort

• Howdowemeasuretheruntimeofanalgorithm?• Worst-caseanalysis• AsymptoticAnalysis

• Nexttime:• IntegermultiplicationpartII• Amoresystematicapproachtosolvingrecurrences