8/14/2019 Intro to algorithms 2
1/23
11-711 Algorithms for NLP
Introduction to Analysis of Algorithms
Reading:Cormen, Leiserson, and Rivest,
Introduction to Algorithms
Chapters 1, 2, 3.1., 3.2.
8/14/2019 Intro to algorithms 2
2/23
Analysis of Algorithms
Analyzing an algorithm: predicting the computational resources that
the algorithm requires
Most often interested in runtime of the algorithm Also memory, communication bandwidth, etc.
Computational model assumed: RAM
Instructions are executed sequentially
Each instruction takes constant time (different constants)
The behavior of an algorithm may be different for different inputs.
We want to summarize the behavior in general, easy-to-understandformulas, such as worst-case, average-case, etc.
1 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
3/23
Analysis of Algorithms
Generally, we are interested in computation time as a function of
input size.
Different measures of input size depending on the problem
Examples:
Problem Input Size
Sorting List of numbers to sort # of numbers to sort
Multiplication Numbers to multiply # of bits
Graph Set of nodes and edges # of nodes + # of edges
Running time of an algorithm: number of primitive steps
executed
Think of each step as a constant number of instructions, each of
which takes constant time.
Example: Insertion-Sort (CLR, p. 8)
2 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
4/23
Insertion Sort (A)
cost times
1. for j
2 to length[A]
n
2. do key
A[j]
n-1
3.
4. i
j - 1
n-1
5. while i 0 and A[i] key
6. do A[i+1]
A[i]
!
#
7. i
i-1
$
!
#
8. A[i+1] key % n-1
3 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
5/23
Insertion-Sort
&
'
: # of times the while loop in line 5 is executed for the value(
)
0
1
2
3
5
6
1
7
5
8
0
1
9
@
2
7
5
A
0
1
9
@
2
7
5
B
C
D
E
8
G
D
7
5
H
C
D
E
8
0
G
D
9
@
2
7
5
I
C
D
E
8
0
G
D
9
@
2
7
5
P
0
1
9
@
2
Q In the best case, when the array is already sorted, the running time is
a linear function ofR
.
)
0
1
2
3
S 1
7
T
Q In the worst case, when the array is opposite of sorted order, the
running time is a quadratic function ofR
.
)
0
1
2
3
S 1
8
7
T 1
7
5
4 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
6/23
Analysis of Algorithms
U We are mostly interested in worst-case performance because
1. It is an upper bound on the running time for any input.
2. For some algorithms it occurs fairly often.
3. The average case is often roughly as bad as the worst case.
U We are interested in the rate of growth of the runtime as the input size
increases. Thus, we only consider the leading term.
Example: Insertion-Sort has worst caseV
W
X
Y
a
U We will normally look for algorithms with the lowest order of growth
U But not always:
b
for instance, a less efficient algorithm may be preferable if its an
anytime algorithm.
5 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
7/23
Analyzing Average Case of Expected Runtime
c Non-experimental method:
1. Assume all inputs of lengthd
are equally likely
2. Analyze runtime for each particular input and calculate theaverage over all the inputs
c Experimental method:
1. Select a sample of test cases (large enough!)
2. Analyze runtime for each test case and calculate the average and
standard deviation over all test cases
6 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
8/23
Recursive Algorithms (Divide and Conquer)
e General structure of recursive algorithms:
1. Divide the problem into smaller subproblems.
2. Solve (Conquer) each subproblem recursively.
3. Merge the solutions of the subproblems into a solution for the full
problem.
e
Analyzing a recursive algorithm usually involves solving arecurrence equation.
e Example: Merge-Sort (CLR, p. 13)
7 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
9/23
Recursive Algorithms
f
g
h
i
p
q time to divide a problem of sizei
f
r
h
i
p
q time to combine solutions of sizei
f Assume we divide a problem intos
subproblems, each tu
the size of
the original problem.
f The resulting recurrence equation is:
v
h
i
p
q
w
h
x
p
i
sv
h
u
p
g
h
i
p
r
h
i
p
8 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
10/23
Merge
: Initialization
times
1.
j
k
l
m
n
o
1
2. k m
l 1
3. create arrays
jo
and
o
1
4. fork
to
j
j
5. do
k
n
o
m
j
6. forz
k
to
m
j
7. do
z
k
l
o
z
m
j
8.
j
o
k {
1
9.
o
k {
1
10.k
1
11.z
k
1
9 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
11/23
Merge|
}
~ ~ ~
(cont.): Actual Merge
times
12. for
to
n
13. do if n
14. then
15.
16. else
17.
10 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
12/23
Merge Sort (A,p,r)
times
1. ifp
r
2. then q
(p+r)/2
3. Merge-Sort(A,p,q)
4. Merge-Sort(A,q+1,r)
5. Merge(A,p,q,r)
11 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
13/23
A note on Implementation
Note that in the pseudocode of the Merge operation, the for-loop is
always executed
times, no matter how many items are in each of
the subarrays.
This can be avoided in practical implementation.
Take-home message: dont just blindly follow an algorithm.
12 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
14/23
Merge-Sort
-
According to the Merge-Sort algorithm, we divide the problem into 2subproblems of size
.
The resulting recurrence equation is:
By solving the recurrence equation we get
,
much better than
for Insertion-Sort.
There are several ways of solving this recurrence equation. The
simplest one uses a recurrence tree, where each node in the tree
indicates what the cost of solving it is. Then inspecting the tree cangive us the solution.
13 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
15/23
In this case, we get
, because we divide the problem in half in
each recursive step.
14 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
16/23
Growth of Functions
We need to define a precise notation and meaning for order of growth
of functions.
We usually dont look at exact running times because they are toocomplex to calculate and not interesting enough.
Instead, we look at inputs of large enough size to make only the order
of growth relevant - asymptotic behavior.
Generally, algorithms that behave better asymptotically will be the
best choice (except possibly for very small inputs).
15 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
17/23
-notation (Theta notation)
Asymptotically tight bound
Idea:
if both functions grow equally fast
Formally:
there exist positive constants
,
, and
such that
for
all
means a constantfunction or a constant
16 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
18/23
-notation (Big-O notation)
Asymptotic upper bound
Idea:
if
grows faster than
, possibly much
faster
Formally:
there exist positive constants
and
such that
for all
Note:
Big-O notation often allows us to describe runtime by just
inspecting the general structure of the algorithm.
Example: The double loop in Insertion-Sort leads us to conclude
that the runtime of the algorithm is
.
We use Big-O analysis for worst-case complexity since that
guarantees an upper bound on any case.
17 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
19/23
-notation (Big-Omega notation)
Asymptotic lower bound
Idea:
if
grows slower than
, possibly
much slower
Formally:
there exist positive constants
and
such that for all
Theorem:
For any two functions
and
,
iff
and
18 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
20/23
-notation (Little-o notation)
Upper bound that is not asymptotically tight
Idea:
if
grows much faster than
Formally:
for any positive constant
, there exists
a constant
such that
for all
We will normally use Big-O notation since we wont want to make
hard claims about tightness.
19 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
21/23
-notation (Little-omega notation)
Lower bound that is not asymptotically tight
Idea:
!
"
#
$
%
!
&
!
"
# #
if&
!
"
#
grows much slower than
!
"
#
Note:
!
"
#
'
%
!
&
!
"
# #
if and only if&
!
"
#
'
(
!
!
"
# #
Formally:
%
!
&
!
"
# #
$
)
!
"
#0
for any positive constant1
2
3
, there exists
a constant"
4
2
3
such that3
5
1
&
!
"
#
6
!
"
#
for all"
7
"
4
8
20 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
22/23
Comparison of Functions
9 The@
,A
,B ,C
, andD
relations are transitive.
Example:
E
F
G
H
I
A
F
P
F
G
H H
Q
P
F
G
H
I
A
F
S
F
G
H H
I
T
E
F
G
H
I
A
F
S
F
G
H H
9 The@
,A
, andB relations are reflexive.
Example:E
F
G
H
I
A
F
E
F
G
H H
9 Note the analogy to the comparison of two real numbers.e.g.
E
F
G
H
I
A
F
P
F
G
H H
U
W
X
Y
e.g.E
F
G
H
I
D
F
P
F
G
H H
U
W
a
Y
21 11-711 Algorithms for NLP
8/14/2019 Intro to algorithms 2
23/23
Comparison of Growth of Functions
b Transitivity: all
b Reflexivity:
c
d
e
f
g
h
d
c
d
e
f f
c
d
e
f
g
i
d
c
d
e
f f
c
d
e
f
g
p
d
c
d
e
f f
b Symmetry:
c
d
e
f
g
h
d
q
d
e
f f
iffq
d
e
f
g
h
d
c
d
e
f f
b Transpose symmetry:
c
d
e
f
g
i
d
q
d
e
f f
iffq
d
e
f
g
p
d
c
d
e
f f
c
d
e
f
g
r
d
q
d
e
f f
iffq
d
e
f
g
s
d
c
d
e
f f
b Lack of Trichotomy:
Somec
d
e
f
are neitheri
d
c
d
e
f f
norp
d
c
d
e
f f
22 11-711 Algorithms for NLP
Top Related