Attention Allocation to Partially Attention Allocation to Partially Observable Heterogeneous Observable Heterogeneous Customers – with Imperfect Customers – with Imperfect

TreatmentTreatment

Avi LatnerAvi LatnerAdvisor: Prof. Gadi RabinowitzAdvisor: Prof. Gadi Rabinowitz

Possible model implementations:

• Machine maintenance• Allocation of medical staff

(NICU)• Law enforcement response

Resource allocation dilemmaResource allocation dilemma

server server server

stable unstableunstablei=1 i=5

unstablei=3 i=4

stablei=2

A single customer state transition diagramA single customer state transition diagram

stable

unstable

failuretreatmentαi

βi

γi*pi

γi*(1-pi)

∞

decision to treat

decision to

treat

αi – deterioration rateβi – “death” rateγi – treatment rate

Server Server Server

Who to Treat?

Actual state:

unstablei=1

stablei=5

unstablei=3

stablei=4

stablei=2

Perceived state:

unstablei=5

unstablei=1

unstablei=2

stablei=3

stablei=4

unstablei=3

stablei=5

Partial ObservabilityPartial Observability

perceived: stable

perceived: unstable

stab

leun

stab

le

stable stable

failure

treatment

unstable treatmentunstable

perceived: stable

perceived: unstable

stab

leun

stab

le

stable

failure

treatment

unstable treatment

stable

unstable

γi*pi

γi*(1-pi)

θi*(1-qi)

γi*qi

∞

αiαi

βi βi

decision to treat

decision to treat

decision to treat

decision to treat

φi

νi

ωi

ρi

Model dimensionsModel dimensions

customer server×2 ×1

customer server×5 ×2

customer server×20 ×4

8

192

≈5*108

Ω(2n)-O(3n)

Solution method: Solution method: the relaxed modelthe relaxed model

Semi-Markov Decision Process

Continuous Time

Markov Chain

who's next?

Service Rates:

)δi , ψi (i=1…n

The relaxed model:The relaxed model:Continuous Time Markov ChainContinuous Time Markov Chain

• Instead of the decision to treat a treatment rate is set for each customer:δi – treatment rate for “unstable”i – treatment rate for “stable”

• Treatment rate is exponentially distributed

• Full server utilization• The problem is formulized as a nonlinear

planning problem

Steady State EquationsSteady State Equationsiiiiiiiiiiiiiii pq 654321 )()(

iiiiiiii 324 )(

iiiiii 12 )(

)1()1()( 65413 iiiiiiiiiiiii pq

iiiiii 215

iiiiii 436

16

1

jij

And… steady state probabilitiesAnd… steady state probabilities1. π1 = -((γ2 θ2 (α + δ + ν) ω (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) / EXP2); (8)

2. π2 = -((γ2 θ2 φ ω (β2+ β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) / EXP2;3. π3 = (γ θ (α2 (β + δ + ρ) - (-1 + q) (β + δ + ρ) (ν ψ + δ (φ + ψ)) + α (δ2 + β (δ + ν + ψ - q ψ) + ρ (ν + φ + ψ - q ψ) + δ (ν + ρ + φ - p φ + ψ - q ψ)))) / EXP1; 4. π4 = -

(1

*((γ θ ω (-1 - (γ2 θ (α + δ + ν) (θ + ψ) ω (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) / EXP2 + (γ θ φ (α θ (γ + ψ) - γ (δ + θ) ω) (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) / EXP2));5. π 5 = (γ2θ (-(δ φ) - (α + δ + ν) ψ) ω (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) / EXP2;6. π6 = (θ (α2 (β ψ + ρ ψ + δ (ψ + ω)) - (-1 + q) (ν ψ + δ (φ + ψ)) (β ψ + ρ ψ + δ (ψ + ω)) + α (β (δ (φ + ψ) + ψ (ν + ψ - q ψ)) + (δ + ν + φ + ψ - q ψ) (ρ ψ + δ (ψ + ω)))) / EXP1;Where:EXP1 = (γ δ2 θ φ - q γ δ2 θ φ + γ δ θ ρ φ - q γ δ θ ρ φ + γ δ2 θ ψ + p γ δ2 θ ψ - q γ δ2 θ ψ + γ δ θ ν ψ + p γ δ θ ν ψ - q γ δ θ ν ψ + γ δ θ ρ ψ + p γ δ θ ρ ψ - q γ δ θ ρ ψ + γ θ ν ρ ψ + p γ θ ν ρ ψ - q γ θ ν ρ ψ + p γ δ2 φ ψ + p γ δ θ φ ψ + δ2 θ φ ψ - q δ2 θ φ ψ + p γ δ ρ φ ψ + p γ θ ρ φ ψ + δ θ ρ φ ψ - q δ θ ρ φ ψ + p γ δ2 ψ2 + δ2 θ ψ2 - q δ2 θ ψ2 + p γ δ ν ψ2 + δ θ ν ψ2 - q δ θ ν ψ2 + p γ δ ρ ψ2 + δ θ ρ ψ2 - q δ θ ρ ψ2 + p γ ν ρ ψ2 + θ ν ρ ψ2 - q θ ν ρ ψ2 + β2 γ (θ (ν + φ) + ν ψ + δ (θ + φ + ψ)) + p γ δ2 θ ω + p γ δ θ ν ω + p γ δ2 φ ω + γ δ θ φ ω + p γ δ θ φ ω - q γ δ θ φ ω + δ2 θ φ ω - q δ2 θ φ ω + p γ δ2 ψ ω + γ δ θ ψ ω - q γ δ θ ψ ω + δ2 θ ψ ω - q δ2 θ ψ ω + p γ δ ν ψ ω + γ θ ν ψ ω - q γ θ ν ψ ω + δ θ ν ψ ω - q δ θ ν ψ ω + α2 θ (δ ψ + ρ ψ + β (γ + ψ) + δ ω + γ (δ + ρ + ω)) + β ((1 - q) θ ψ (ν ψ + δ (φ + ψ)) + γ (δ2 (θ + φ + ψ) + ν ψ (ρ + p ψ + ω) + θ (φ (ρ + p ψ + ω) + ν (ρ + ψ + p ψ - q ψ + ω)) + δ (ν ψ + p φ ψ + p ψ2 + ρ (φ + ψ) + φ ω + ψ ω + θ (ν + ρ + 2 φ - q φ + ψ + p ψ - q ψ + ω)))) + α (β2 γ (θ + ψ) + θ (δ + ν + φ + ψ - q ψ) (ρ ψ + δ (ψ + ω)) + β (θ (δ (φ + ψ) + ψ (ν + ψ - q ψ)) + γ (δ (2 θ + ψ) + ψ (ρ + p ψ + ω) + θ (ν + ρ + φ + ψ + p ψ - q ψ + ω))) + γ (δ2 θ + p ρ ψ2 + θ (p φ ψ + ρ (φ + ψ + p ψ - q ψ) + φ ω + ψ ω - q ψ ω + ν (ρ + ω)) + δ (p ψ (ψ + ω) + θ (ν + ρ + φ - p φ + ψ + p ψ - q ψ + ω + p ω))))); andEXP2 = ((-α - δ - ν) ((q γ θ ψ - γ θ (α + φ + ψ)) ω (-(θ (β + δ + ρ) (γ + ψ)) - (γ + δ) θ ω) + γ2 θ (θ + ψ) ω (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω)))) - γ θ φ ((-(θ (β + δ + ρ) (γ + ψ)) - (γ + δ) θ ω) (-(α (β + p ψ)) + (q δ + ν) ω) - (α θ (γ + ψ) - γ (δ + θ) ω) (β2 + β (δ + ρ + p ψ + ω) + p (ρ ψ + δ (ψ + ω))));

perceived: stable

perceived: unstable

stable

failure

treatment

unstable treatment

stable

unstable decision to treat

decision to treat

decision

to treat

decis

ion

to tre

at

δi

δii

i

perceived: stable

perceived: unstable

stable treatment

treatment

stable

failure

unstable

Πi4

unstable

Πi3

βi βi

)],(),([, 43 iiiiiiiiiih

perceived: stable

perceived: unstable

failure

unstable

Πi4

stableΠi1

treatment

treatment

stable

Πi2

unstable

Πi3

i

iiii

i

iiiiiiig

)()(),( 4321

δi

i

i

δi

Nonlinear programming problemNonlinear programming problem

),(),(1

i

n

iihhMin

nini

Kgg

TS

i

i

i

n

ii

1010

),(),(

..

1

Solving the nonlinear problemSolving the nonlinear problem• Optimum point is found by Karush Kuhn Tucker

conditions• How much will a small addition of resource to

customer i decrease the failure rate:

• Distribute resources by the rule “who would contribute the most” until all the capacity is used

),(), ii

i

iii

i

i

g

hOR(g

h

-350

-300

-250

-200

-150

-100

-50

0

1

2

3

4

5

-350

-300

-250

-200

-150

-100

-50

0

Solving the nonlinear problem – Cont.Solving the nonlinear problem – Cont.

Solving the nonlinear problem – Cont.Solving the nonlinear problem – Cont.

δi\ii=00<i∞<i∞→

δi=0GROUP A

Immediatesolution

GROUP BOnedimensionalquery

-----

0<δi∞<GROUP C

Onedimensionalquery

GROUP DTwodimensionalquery

GROUP EOnedimensionalquery

δi∞→-----GROUP F

Onedimensionalquery

GROUP GImmediatesolution

Treatment rate for a “stable” customer

Trea

tmen

t rat

e fo

r an

“uns

tabl

e” c

usto

mer

-350

-300

-250

-200

-150

-100

-50

0

1

2

3

4

5

-350

-300

-250

-200

-150

-100

-50

0

GROUP G

GROUP G

GROUP C GROUP C

GROUP DGROUP D

GROUP AGROUP A

λ

Solving the nonlinear problem – Cont.Solving the nonlinear problem – Cont.

2 31 5 64

Sever

δ=0=30δ=240 δ=∞ =∞ δ∞=δ=110=0

FIFO By treatment rate

“Stable”

Unstable”"

Legend:

Service Priority PolicyService Priority Policy

2 31 5 64

שרת

δ∞=∞=δ∞= δ=240 =30δ=110ψ=50 δ∞=∞= δ=240 =30∞=

“Stable”

Unstable”"

Legend:

Service Priority PolicyService Priority Policy

Possible model implementations:

• Machine maintenance• Allocation of medical staff

(NICU)• Law enforcement response