Dynamic Programming- Mirage Group
Transcript of Dynamic Programming- Mirage Group
-
8/6/2019 Dynamic Programming- Mirage Group
1/37
DYNAMIC PROGRAMMINGDYNAMIC PROGRAMMING
SUBMITTED BY:DR. DEBANJANA GHOSHDR. PRABHPREET KAUR DR. PRASHANTH KULALADR. SANDIP KR NEOGI
SHWETNISHA BOSE
-
8/6/2019 Dynamic Programming- Mirage Group
2/37
DYNAMIC PROGRAMMING
Technique that divides the problem to be solved into a numberof sub problems and then solves each sub-problem in such away that the overall solution is optimal to the originalproblem.
Invented by American mathematician Richard Bellman in the1950s to solve optimization problems
Programming here means planning
Dynamic programming is a stage-wise search method suitablefor optimization problems whose solutions may be viewed asthe result of a sequence of decisions.
-
8/6/2019 Dynamic Programming- Mirage Group
3/37
MAIN IDEA
set up a recurrence relating a solution to a largerinstance to solutions of some smaller instancessolve smaller instances oncerecord solutions in a table
extract solution to the initial instance from thattable
-
8/6/2019 Dynamic Programming- Mirage Group
4/37
Example: Fibonacci numbers using recursion
Fibonacci numbers: F (n ) = F (n -1) + F (n -2) F (0) = 0 F (1) = 1
Computing the n th Fibonacci number recursively (top -d own ):
F (n )
F (n-
1) + F
(n-
2)
F (n- 2) + F (n- 3) F (n- 3) + F (n- 4)... Bad algorithm as
numerous repetitive
calculations are there
-
8/6/2019 Dynamic Programming- Mirage Group
5/37
Example Fibonacci Numbersusing Dynamic programming
f (5)
f (4) f (3)
f (3) f (2) f (2) f (1)
f (2) f (1)
C alculation from bottom to top and Values are stored for later use
This reduces repetitive calculation
-
8/6/2019 Dynamic Programming- Mirage Group
6/37
CHARACTERISTICS OFDYNAMIC PROGRAMMING
The problem can be divided into stages with a decisionrequired at each stage.In the capital budgeting problem the stages were the allocations toa single plant. The decision was how much to spend. In the shortest
path problem, they were defined by the structure of the graph. Thedecision was were to go next.Each stage has a number of states associated with it.The states for the capital budgeting problem corresponded to theamount spent at that point in time. The states for the shortest path
problem was the node reached.The decision at one stage transforms one state into a state inthe next stage.The decision of how much to spend gave a total amount spent for the next stage. The decision of where to go next defined where youarrived in the next stage.
-
8/6/2019 Dynamic Programming- Mirage Group
7/37
Given the current state, the optimal decision for each of theremaining states does not depend on the previous states or
decisions .In the budgeting problem, it is not necessary to know how themoney was spent in previous stages, only how much was spent.In the path problem, it was not necessary to know how you got to
a node, only that you did.There exists a recursive relationship that identifies theoptimal decision for stage j , given that stage j +1 has alreadybeen solved.The final stage must be solvable by itself.
The last two properties are tied up in the recursiverelationships given above.
-
8/6/2019 Dynamic Programming- Mirage Group
8/37
STEPS OF DYNAMIC PROGRAMMING
IDENTIFY DECISIONVARIABLES
SPECIFY OBJECTIVE FUNCTION
DECOMPOSE INTOSMALLER PROBLEMS ( STAGE )
-
8/6/2019 Dynamic Programming- Mirage Group
9/37
IDENTIFYSTATE VARIABLES
STATE THE GENERALRECURSIVE RELATIONS. ALSO DECIDE FORW ARD/BACKWARD
CALCULATE REQ UIRED VALUES OFRETURN FUNCTION BY APPR OPRIATE
FORMULAE AT EACH STAGE
REPEAT AND DETERMINE THE
OVERALL OPTIMALPOLICY
-
8/6/2019 Dynamic Programming- Mirage Group
10/37
THE
STAGECOACHPROBLEM
-
8/6/2019 Dynamic Programming- Mirage Group
11/37
A MINIMUM PATH PROBLEM
Given a series of paths from point A to point BA and B are not directly connectedEach path has a value linked to it
Find the best route from A to B
The idea for this problem is that a salesman is traveling from
one town to another town, in the old west. His means of
travel is a stagecoach. Each leg of his trip cost a certainamount and he wants to find the minimum cost of his trip,given multiple paths.
-
8/6/2019 Dynamic Programming- Mirage Group
12/37
A SAMPLE STAGECOACH
PROBLEM
-
8/6/2019 Dynamic Programming- Mirage Group
13/37
W e begin by dividing the problem into stages
-
8/6/2019 Dynamic Programming- Mirage Group
14/37
Suppose we are at node i, we want to find the lowestcost route from i to 10
Start at node 10, and work backwards through thenetwork.
Define variables such that:
cost of travel from node i to node j
node chosen for stage n = 1; 2; 3; 4state
-
8/6/2019 Dynamic Programming- Mirage Group
15/37
START AT STAGE 1 (THE LAST STAGE),THEN
STATE DEC ISION10
BESTDEC ISION
BESTDISTAN C E
8 2+0 10 2
9 4+0 10 4
-
8/6/2019 Dynamic Programming- Mirage Group
16/37
AT STAGE 2
STATE DEC ISION
8
DEC ISION
9
BESTDEC ISION
BESTDISTAN C E
5 1+2=3 4+4=8 8 3
6 6+2=8 3+4=7 9 7
7 3+2=5 3+4=7 8 5
-
8/6/2019 Dynamic Programming- Mirage Group
17/37
APPLICATIONS
This problem can be used inC
omputer Networks
Plane travel
-
8/6/2019 Dynamic Programming- Mirage Group
18/37
BELLMANS PRINCIPLE OF OPTIMALITY
An optimal policy has the property that whatever the initial state and initial decisionsare, the remaining decisions must constitutean optimal policy with regard to the stateresulting from the first decision
-
8/6/2019 Dynamic Programming- Mirage Group
19/37
OPTIMALITY PRINCIPLE
Assume that optimal path is the shortest oneOP indicates that any portion of any optimal path isalso optimal
Set of optimal paths from all sources to a givendestination forms a tree that is routed at the
destination
A B C
D E F
2 3
2 1 2
4 1
Optimal path between A and Fis A-B-C-E-F
Then, the optimal path betweenB and F is B-C-E-F
-
8/6/2019 Dynamic Programming- Mirage Group
20/37
PRINCIPLE OF OPTIMALITY
ac
Assertion : If abe is the optimal path from a to e, then be is
the optimal path from b to e.Proof: Suppose it is not. Then there is another path(note that existence is assumed here) bce whichis optimal from b to e, i.e.
J bce > J be but then J abe = J ab+ J be< J ab+ J bce= J abceThis contradicts the hypothesis that abe is theoptimal path from a to e.
JabJbe
eJbceb
-
8/6/2019 Dynamic Programming- Mirage Group
21/37
PRINCIPLE OF OPTIMALITY
1
2
3
4
5
6
7
8
9
Bellmans Principle of Optimality : At any intermediate state x i in an optimal path from x 0 to x f , the policy from x ito goal x f must itself constitute optimal policy
-
8/6/2019 Dynamic Programming- Mirage Group
22/37
BELLMANS PROGRAMMING
Transforms a complex optimization problem into a
sequence of simpler ones.
Usually begins at the end and works backwards
C an handle a wide range of problems
Relies on recursion, and on the principle of optimality
Developed by Richard Bellman
-
8/6/2019 Dynamic Programming- Mirage Group
23/37
MARKOVIAN REQUIREMENT
An optimal policy starting in a given state depends
only on the state of the process at that stage and not
on the state at preceding stages. The path is of no
consequence, only the present stage and state
The state variables fully describe the state of thesystem at each stage and capture all past decisions
-
8/6/2019 Dynamic Programming- Mirage Group
24/37
Optimal Capacity Expansion: W hat isthe least cost way of building plants?
24
C ost of $15 million in any year in which a plant is built.At most 3 plants a year can be built
-
8/6/2019 Dynamic Programming- Mirage Group
25/37
Forwards recursion
Start with stage 1 and determine values for larger stages using recursion.In forward recursion f(n) is determined from f(k)where k < n.Determine the value of the other stages using thevalue of the first stage with the help of forwardrecursion.
25
-
8/6/2019 Dynamic Programming- Mirage Group
26/37
Backwards Recursion
Boundary conditions: optimum values for states at thelast stage.In backward recursion value of f(k) is determinedfrom f(n) where k
-
8/6/2019 Dynamic Programming- Mirage Group
27/37
Determining state and stage
Stage : year Y
State : number of plants
f(j, Y) = optimum cost of capacity expansion starting
with j plants at the end of year Y.
We want to compute f(0, 2002).
We will first compute what is f(j, 2007) for different j.
27
-
8/6/2019 Dynamic Programming- Mirage Group
28/37
Determining state and stage
From f(j,2007) for each j we will compute f(j, 2006)
for each j.
After that we will complete f(j, 2005) for each j.
Then compute f(j, 2004) for each j using f(j,2005).
From f(j,2004) f(j,2003) for each j will be computed.
Lastly from f(j,2003) f(j,2002)can be calculated.
28
-
8/6/2019 Dynamic Programming- Mirage Group
29/37
TRAVELING SALESMAN
Start 1
East
2
3
4
5
6
7
8
9
End 10
West
-
8/6/2019 Dynamic Programming- Mirage Group
30/37
INSURANCE POLICY COST
T O
2 3 4 5 6 7 8 9 10
F 1 2 4 3 2 7 4 6 5 1 4 8 3
R 3 3 2 4 6 6 3 9 5
O 4 4 1 5 7 3 3
M
-
8/6/2019 Dynamic Programming- Mirage Group
31/37
Minimize the cost for each run
1 2 6 9 10Total cost = 14
However,1 4 6 9 10
Total cost = 12 !
-
8/6/2019 Dynamic Programming- Mirage Group
32/37
CONDITIONS
Those that can satisfy Bellmans principle of optimality:
The globally optimum solution includes no suboptimal local
decision.
or in Bellmans original words from 1957:
An optimal policy has the property that, regardless of the
decisions taken to enter a particular state, the remaining decisions
made for leaving that stage must constitute an optimal policy.
-
8/6/2019 Dynamic Programming- Mirage Group
33/37
APPLICATIONS
In numerous multistage decision problems of optimum fitsuch as:
Error correction coding (Viterbi algorithm).Automatic labelling of speech segments (dynamic time warping).Video coding.Digital watermark detection.C ell library binding (as part of logic optimization).Flight trajectory planning.Genome sequencing (Smith-Waterman and Needleman-Wunschalgorithms).Knapsack problems.Stereo vision.
-
8/6/2019 Dynamic Programming- Mirage Group
34/37
Dynamic Programming v/s LinearProgramming
Dynamic Programming is different from Linear Programmingon two counts.
1). There does not exist a standard mathematical formulation of Dynamic Programming problem. There is no algorithm likeSimplex method , that may be programmed to solve all the
problems. Instead, Dynamic Programming is a technique whichdissects difficult problems into a sequence of sub- problems,which are then evaluated by stages.
2). While Linear Programming is a method which gives singlestage, i.e. one time period solutions. But Dynamic Programmingahs the power to determine the optimal solution e.g. one year time horizon divided into 12 months time horizon. Thus , it is amultistage approach to solve a problem.
-
8/6/2019 Dynamic Programming- Mirage Group
35/37
Solving linear programming by dynamic programming
Eg: Maximize Z= c 1x1+c2x2+ +c nxn
Su bject to the constraints
a ixi+a ix+ ..+ax=0 where j=1,2 n
Consider this LPP as m u ltistage problem with eachactivity j (j = 1,2, ..n) as individ u al stage so this is an
n- stage problem
-
8/6/2019 Dynamic Programming- Mirage Group
36/37
SUMMARYThe Dynamic Programming technique is useful for making a
sequence of inter-related decisions where the objective is tooptimize the overall outcome of the entire sequence of decisionsover a period of time.
This technique decomposes the original problem into asequence of stages, which can then be handled more efficientlyfrom the computational point of view.
In contrast, most linear and nonlinear programming approaches
attempt to solve such problems by considering all the constraintssimultaneously.
The dynamic programming technique is used in various fieldssuch as inventory control, allocation, Bidding problems etc.
-
8/6/2019 Dynamic Programming- Mirage Group
37/37