1 1 © 2003 Thomson /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s...

1© 2003 Thomson© 2003 Thomson/South-Western/South-Western Slide

Slides Prepared bySlides Prepared by

JOHN S. LOUCKSJOHN S. LOUCKSSt. Edward’s UniversitySt. Edward’s University


Chapter 18Chapter 18Dynamic ProgrammingDynamic Programming

Dynamic Programming OverviewDynamic Programming Overview Dynamic Programming NotationDynamic Programming Notation Backwards RecursionBackwards Recursion 3 Applications of Dynamic Programming3 Applications of Dynamic Programming A Production and Inventory Control ProblemA Production and Inventory Control Problem


Dynamic ProgrammingDynamic Programming Dynamic programmingDynamic programming (DP) is an approach to (DP) is an approach to

problem solving which permits decomposing of problem solving which permits decomposing of the original problem into a series of several the original problem into a series of several smaller subproblems.smaller subproblems.

To successfully apply DP, the original problem To successfully apply DP, the original problem must be viewed as a must be viewed as a multistage decision multistage decision problemproblem..

Defining the stages of a DP problem is Defining the stages of a DP problem is sometimes obvious, but at other times this sometimes obvious, but at other times this requires subtle reasoning.requires subtle reasoning.


Dynamic ProgrammingDynamic Programming The power of DP is that one need solve only a The power of DP is that one need solve only a

small portion of all subproblems, due to Bellman's small portion of all subproblems, due to Bellman's principleprinciple ofof optimalityoptimality. .

Bellman’s principle states that regardless of what Bellman’s principle states that regardless of what decisions were made at previous stages, if the decisions were made at previous stages, if the decision to be made at stage decision to be made at stage nn is to be part of an is to be part of an overall optimal solution, then the decision made overall optimal solution, then the decision made at stage at stage nn must be optimal for all remaining must be optimal for all remaining stages.stages.


Dynamic Programming NotationDynamic Programming Notation At each stage, At each stage, nn, of the dynamic program, there is:, of the dynamic program, there is:

•a a state variablestate variable, , xxnn

•an an optimal decision variableoptimal decision variable, , ddnn

For each value of For each value of xxnn and and ddnn at stage at stage nn, there is:, there is:•a a return function valuereturn function value, , rrnn((xxnn,,ddnn))

The output of the process at stage The output of the process at stage nn is: is:•the state variable for stage the state variable for stage n-n-1, 1, xxn-n-11

•xxn-n-11 is calculated by a stage is calculated by a stage transformation transformation functionfunction, , ttnn((xxnn,,ddnn))

The The optimal value functionoptimal value function, , ffnn((xxnn), is the cumulative ), is the cumulative return starting at stage return starting at stage nn in state in state xxnn and proceeding and proceeding to stage 1 under an optimal strategy.to stage 1 under an optimal strategy.


Generally, a dynamic programming problem is Generally, a dynamic programming problem is solved by starting at the final stage and working solved by starting at the final stage and working backwards to the initial stage, a process called backwards to the initial stage, a process called backwards recursionbackwards recursion..

The following recursion relation can be used to The following recursion relation can be used to operationalize the principle of optimality:operationalize the principle of optimality:

ffnn((xxnn) = MAX {) = MAX {rrnn((xxnn,,ddnn) + ) + ffnn -1-1((ttnn((xxnn , , ddnn))}))}

ddnn

A problem is solved beginning at stage 0 with A problem is solved beginning at stage 0 with the boundary condition the boundary condition ff00((xx00) = 0, and working ) = 0, and working backwards to the last stage, backwards to the last stage, NN..

Backwards RecursionBackwards Recursion


Three Applications of Dynamic Three Applications of Dynamic Programming ProblemsProgramming Problems

Shortest Route ProblemShortest Route ProblemIn solving a In solving a shortest route problemshortest route problem using using

dynamic programming, one should consider dynamic programming, one should consider the network as a series of stages with a unique the network as a series of stages with a unique subset of nodes corresponding to each stage. subset of nodes corresponding to each stage. The state variables correspond to the different The state variables correspond to the different nodes at each stage.nodes at each stage.



Knapsack or Cargo Loading ProblemKnapsack or Cargo Loading ProblemThe The knapsack problemknapsack problem seeks to determine seeks to determine

the optimal number of each of the optimal number of each of NN items (which items (which must not be fractional) to select in order to must not be fractional) to select in order to maximize profit subject to an overall capacity maximize profit subject to an overall capacity constraint. In solving a knapsack problem using constraint. In solving a knapsack problem using dynamic programming, the stages correspond to dynamic programming, the stages correspond to the different items being placed into a knapsack. the different items being placed into a knapsack. The state variables correspond to the capacity The state variables correspond to the capacity available at the stage.available at the stage.



Production and Inventory Control ProblemsProduction and Inventory Control ProblemsIn production and inventory control In production and inventory control

problems, the stages correspond to time periods problems, the stages correspond to time periods and the state variables generally will refer to the and the state variables generally will refer to the amounts of inventory on hand at the beginning amounts of inventory on hand at the beginning of each stage.of each stage.


Example: Dicom CorporationExample: Dicom Corporation Production and Inventory Control ProblemProduction and Inventory Control Problem

Dicom Corporation wishes to determine Dicom Corporation wishes to determine a production schedule for its new Model 44/12 a production schedule for its new Model 44/12 robotic welder. Because of differences in parts robotic welder. Because of differences in parts availability and spare production capacity, the availability and spare production capacity, the cost of producing the machines will vary from cost of producing the machines will vary from month to month. month to month.

The holding cost for each unsold welder The holding cost for each unsold welder still in inventory at the end of the month is still in inventory at the end of the month is $500,000. Corporate policy dictates that the $500,000. Corporate policy dictates that the maximum number of welders allowed in maximum number of welders allowed in inventory at the end of any month is 8. inventory at the end of any month is 8.

Additional data appears on the next Additional data appears on the next slide.slide.


Example: Dicom CorporationExample: Dicom Corporation

Cost of MaximumCost of Maximum Production Production SalesProduction Production Sales Per Machine Level Per Machine Level

DemandDemand MonthMonth (in $100,000's)(in $100,000's) for Monthfor Month (in (in

Units)Units) August 32 August 32 4 4 22 September 18 September 18 3 3 11 October 26 October 26 4 4 55 November 45 November 45 5 5 33

Determine an optimal 4-month production Determine an optimal 4-month production schedule for the Model 44/12 welder.schedule for the Model 44/12 welder.


Example: Dicom CorporationExample: Dicom Corporation 4-Stage Dynamic Programming Problem4-Stage Dynamic Programming Problem

Working backwards, let stage 1 correspond to Working backwards, let stage 1 correspond to November, stage 2 to October, etc. The following data November, stage 2 to October, etc. The following data (costs in $100,000's) can be inferred: (costs in $100,000's) can be inferred:

Product. HoldingProduct. Holding Product. Storage Cost CostProduct. Storage Cost Cost Month Dem. Capacity Capacity Per Unit Per unit Month Dem. Capacity Capacity Per Unit Per unit ((nn) ) DDnn PPnn WWnn CCnn HHnn

1 3 5 1 3 5 8 8 45 45 5 5

2 5 4 8 26 52 5 4 8 26 5 3 1 3 3 1 3 8 18 8 18

55 4 2 4 4 2 4 8 32 8 32

55


Example: Dicom CorporationExample: Dicom Corporation State Variable DefinedState Variable Defined xxnn = number of computers in inventory at = number of computers in inventory at

the the beginning of month beginning of month nn. . xx44 = 0 (Since the computer is new, there = 0 (Since the computer is new, there will be no will be no inventory at the start of inventory at the start of August.)August.)

Decision Variable DefinedDecision Variable Defined ddnn = production quantity for month = production quantity for month nn. .


Example: Dicom CorporationExample: Dicom Corporation Stage Transformation FunctionStage Transformation Function Then the stage transformations for months Then the stage transformations for months

0 0 through 4 can be defined by: through 4 can be defined by: (Previous month's inventory)(Previous month's inventory) + (Production this month) + (Production this month) (Demand this month)(Demand this month)

xxn-n-11 = = xxnn + + ddnn - - DDnn


Example: Dicom CorporationExample: Dicom Corporation Return FunctionReturn Function

rrnn((xxnn,,ddnn) = (Production cost) + (Holding cost) ) = (Production cost) + (Holding cost) for month for month nnProduction cost = (Production cost per unit)Production cost = (Production cost per unit)

x (Number of units x (Number of units produced in month produced in month nn))Holding cost = (Holding cost per unit)Holding cost = (Holding cost per unit)

x (Ending inventory for month x (Ending inventory for month nn) )

Hence, Hence, rrnn((xxnn,,ddnn) = ) = CCnn ddnn + + HHnn((xxnn + + ddnn - - DDnn))


Return FunctionReturn Function (continued)(continued)

rrnn((xxnn,,ddnn) = ) = CCnn ddnn + + HHnn((xxnn + + ddnn - - DDnn))

This gives: This gives: rr11((xx11,,dd11) = 50) = 50dd11 + 5 + 5xx11 - 15 - 15 rr22((xx22,,dd22) = 31) = 31dd22 + 5 + 5xx22 - 25 - 25 rr33((xx33,,dd33) = 23) = 23dd33 + 5 + 5xx33 - 5 - 5 rr44((xx44,,dd44) = 37) = 37dd44 + 5 + 5xx44 – 10 – 10




Restrictions on Restrictions on xxnn and and ddnn •Restriction (1)Restriction (1)

Because backordering is not allowed, we must Because backordering is not allowed, we must be able to meet the sales demand in any be able to meet the sales demand in any month. month.

xxnn + + ddnn >> DDnn



Restrictions on Restrictions on xxnn and and ddnn (continued)(continued)•Restriction (2)Restriction (2)

Because there is a maximum storage of Because there is a maximum storage of WWnn at at each stage each stage nn, the total inventory at the end of , the total inventory at the end of any month cannot exceed any month cannot exceed WWnn . .

xxnn + + ddnn - - DDnn << WWnn oror

xxnn + + ddnn << WWnn + + DDnn



Restrictions on Restrictions on xxnn and and ddnn (continued)(continued)•Restriction (3)Restriction (3)

The amount produced in any given month The amount produced in any given month cannot exceed the production capacity for cannot exceed the production capacity for that month.that month.

ddnn << PPnn


Optimal Value FunctionOptimal Value Functionffnn((xxnn) = optimal return (minimal cost) for ) = optimal return (minimal cost) for

stages 1 stages 1 through through nn given one starts given one starts stage stage nn with with xxnn welders in inventory welders in inventory

ffnn((xxnn) = Min {) = Min {rrnn((xxnn , ,ddnn) + ) + ffnn -1-1((xxnn --

11)})} ddnn

where where ddnn is constrained by restrictions (1), (2), is constrained by restrictions (1), (2), and (3).and (3).



Example: Dicom CorporationExample: Dicom Corporation Stage 1 (November)Stage 1 (November)

Because Because ff00((xx00) = 0 is a boundary condition:) = 0 is a boundary condition:

ff11((xx11) = Min ) = Min rr11((xx11, , dd11)) dd11

Using the expression for Using the expression for rr11((xx11, , dd11) and restrictions (1), ) and restrictions (1), (2), (3):(2), (3):

ff11((xx11) = Min 5) = Min 5xx11 + 50 + 50dd11 - 15 - 15 s.t. s.t. xx1 1 + + dd11 >> 3 (1) 3 (1)

xx11 + + dd11 << 11 (2) 11 (2) dd11 << 5 (3) 5 (3)

and, and, dd11 >> 0 0


Example: Dicom CorporationExample: Dicom Corporation Tabulated Values for 50Tabulated Values for 50dd11 +5 +5xx11 –15 –15

dd11 xx11 0 1 2 3 4 5 0 1 2 3 4 5 dd11* * ff11((xx11))

0 135 185 235 3 1350 135 185 235 3 135 1 90 140 190 240 2 901 90 140 190 240 2 90 2 45 95 145 195 245 1 452 45 95 145 195 245 1 45 3 0 50 100 150 200 250 0 03 0 50 100 150 200 250 0 0

Note: Having Note: Having xx11 > 3 would result in having welders in > 3 would result in having welders in inventory at the end of November which is undesirable.inventory at the end of November which is undesirable.


Stage 2 (October)Stage 2 (October)

ff22((xx22) = Min 5) = Min 5xx22 + 31 + 31dd22 -25 + -25 + ff11((xx11)) dd22

Given the restrictions, the subproblem is:Given the restrictions, the subproblem is:

ff22((xx22) = Min 5) = Min 5xx22+ 31+ 31dd22 -25 + -25 + ff11((xx22+ + dd22 - 5)- 5)

s.t. s.t. xx22+ + dd22 >> 5 (1) 5 (1) xx22+ + dd22 << 13 (2) 13 (2) dd22 << 4 (3) 4 (3) and and dd22 >> 0 0



Example: Dicom CorporationExample: Dicom Corporation Tabulated Values for 5Tabulated Values for 5xx22 + 31 + 31dd22 -25 + -25 + ff11((xx11))

dd22

xx22 0 1 2 3 4 0 1 2 3 4 dd22* * ff22((xx22) ) xx22++dd22*-5 = *-5 = xx11

1 1 239 4 239 0 239 4 239 0 2 213 199 4 199 12 213 199 4 199 1 3 187 173 159 4 159 23 187 173 159 4 159 2 4 161 147 133 119 4 119 34 161 147 133 119 4 119 3 5 135 121 107 93 3 93 35 135 121 107 93 3 93 3 6 95 81 67 2 67 36 95 81 67 2 67 3 7 55 41 1 41 37 55 41 1 41 3 8 15 0 15 38 15 0 15 3

Note: Note: xx22= 0 is infeasible.= 0 is infeasible.


Stage 3 (September)Stage 3 (September)

ff33((xx33) = Min 5) = Min 5xx33 + 23 + 23dd33 - 5 + - 5 + ff22((xx22) ) dd33


ff33((xx33) = Min 5) = Min 5xx33 + 23 + 23dd33 - 5 + - 5 + ff22((xx33 + + dd33 - 1) - 1) s.t. s.t. xx33 + + dd33 >> 1 (1) 1 (1) xx33 + + dd33 << 9 (2) 9 (2) dd33 << 3 (3) 3 (3)

and and dd33 >> 0 0



Example: Dicom CorporationExample: Dicom Corporation Tabulated Values for 5Tabulated Values for 5xx33 + 23 + 23dd33 - 5 + - 5 + ff22((xx33 + + dd33 -1) -1)

xx33 0 1 2 3 0 1 2 3 dd33* * ff33((xx33) ) xx33 + + dd33*-1 = *-1 = xx22

0 280 263 3 263 0 280 263 3 263 2 2 1 263 245 228 3 228 1 263 245 228 3 228 3 3 2 244 227 210 193 3 193 2 244 227 210 193 3 193 4 4 3 209 192 175 172 3 172 3 209 192 175 172 3 172 5 5 4 174 157 154 151 3 151 4 174 157 154 151 3 151 6 6 5 139 136 133 130 3 130 5 139 136 133 130 3 130 7 7 6 118 115 112 109 3 109 6 118 115 112 109 3 109 8 8 7 97 94 91 1 91 7 97 94 91 1 91 8 8 8 76 73 1 73 8 76 73 1 73 8 8

dd33


Stage 4 (August)Stage 4 (August) ff44((xx44) = Min 5) = Min 5xx44 + 37 + 37dd44 - 10 + - 10 + ff33((xx33)) dd44


ff44((xx44) = Min 5) = Min 5xx44 + 37 + 37dd44 - 10 + - 10 + ff33((xx44 + + dd44 - 2) - 2)

s.t. s.t. xx44 + + dd44 >> 2 (1) 2 (1) xx44 + + dd44 << 10 (2) 10 (2) dd44 << 4 (3) 4 (3)

and and dd44 >> 0 0



Tabulated Values for 5Tabulated Values for 5xx44 + 37 + 37dd44 - 10 + - 10 + ff33((xx44 + + dd44 - 2)- 2)

dd44 xx44 0 1 2 3 4 0 1 2 3 4 dd44* * ff44 ( (xx44 ) )

xx44++dd44*-2 = *-2 = xx33

0 0 327 329 331 2 327 327 329 331 2 327 0 0

Note: Because August starts with Note: Because August starts with xx44 = 0 = 0 inventory on hand, compute table only for inventory on hand, compute table only for xx44 = 0. = 0.



Example: Dicom CorporationExample: Dicom Corporation Solution SummarySolution Summary

Working backwards through the tables, one can Working backwards through the tables, one can determine the optimal solution that gives the determine the optimal solution that gives the minimum cost of $327 x 100,000 = $32,700,000.minimum cost of $327 x 100,000 = $32,700,000. Inventory On-Inventory On-

Hand BeginningHand Beginning Stage Month Produce (Stage Month Produce (ddnn*) Next *) Next

Month Month xxnn-1-1 4 4 August August 2 2 0 0

3 3 September 3 September 3 2 2 2 2 October October 4 4 1 1 1 1 November 2 November 2 0 0


End of Chapter 18End of Chapter 18

1 1 © 2003 Thomson /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s...

Documents

Transcript of 1 1 © 2003 Thomson /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s...