Advanced Pipelining

COMP25212

Advanced Pipelining

Out of Order Processors

Overview and Learning Outcomes

• Find out how modern processors work

• Understand the evolution of processors

• Learn how out-of-order processors can improve processors performance

• Discover architectural solutions to support and improve out-of-order execution

2

Remember from last week

• Classic 5-stage pipeline• Control Hazards• Data Hazards

• Instruction Level Parallelism• Superscalar

• Out of order execution

Classic 5-stage pipeline

Fetch

Logic

DecodeLogic

Exec

Logic

Mem

Logic

Write

Logic

Inst Cache Data Cache

• A single execution flow

Modern Pipelines

Fetch

Decode

Add1

Ld2

Write

Back

Inst Cache

Write

Back

Ld1

Mul3

Write

Back

Mul1

Mul2

Div3

Write

Back

Div1

Div2

• Many execution flows

Functional Units

In ARM Processors

In-order processor

Out of order processor

Out of Order Execution

• The original order in a program is not preserved

• Processors execute instructions as input data becomes available

• Pipeline stalls due to conflicted instructions are avoided by processing instructions which are able to run immediately

• Take advantage of ILP

• Instructions per cycle increases

Conflicted Instructions

• Cache misses: long wait before finishing execution

• Structural Hazard: the required resources are not available

• Data hazard: dependencies between instructions

Structural Hazards

• Functional Units are typically not pipelined

• This means only one instruction can use them at once

• If all suitable Functional Units for executing an instruction are busy, then the instruction can not be executed

Modern Pipelines

Fetch

Decode

Add1

Ld2

Write

Back

Inst Cache

Write

Back

Ld1

Mul3

Write

Back

Mul1

Mul2

Div3

Write

Back

Div1

Div2

• Many execution flows

Functional Units

Data dependencies

• True dependencyr1 <- r2 op r3

r4 <- r1 op r5

• Anti-dependencyr1 <- r2 op r3

r2 <- r4 op r5

• Output dependencyr1 <- r2 op r3

…

r1 <- r4 op r5

Read-after-write

RAW

Write-after-read

WAR

Write-after-write

WAW

• Key Idea: Allow instructions behind stall to proceed. => Instructions executing in parallel. There are multiple execution units, so use them

DIVD F0, F2, F4

ADDD F10, F0, F8

SUBD F12, F8, F14

– Enables out-of-order execution => out-of-order completion

Even though ADDD stalls, theSUBD has no dependencies and can run.

Dynamic Scheduling

Dynamic pipeline scheduling overcomes the limitations of in-order pipelined execution by allowing out-of-order instruction execution

Out of Order Execution with Scoreboard

Scoreboard

• The scoreboard is a centralized hardware mechanism

– Instruction are executed as soon as their operands are available and there are no hazard conditions

• It dynamically constructs the dependency graph by hardware for a window of instructions as they are issued in program order

• The scoreboard is a “data structure” that provides the information necessary for all pieces of the processor to work together

(In Appendix A.8) CDC6600 (1963)

• Out-of-order execution divides ID stage:1.Issue—decode instructions, check for structural

hazards

2.Read operands—wait until no data hazards, then read operands

• Scoreboards allow instruction to execute whenever 1 & 2 hold, not waiting for prior instructions

• We will use In-order issue, out-of-order execution, out of order commit ( also called completion)

The Key idea of Scoreboards

Typical Scoreboard Structure

Issue

WriteBack

ReadOperands

ExecuteInteger

WriteBack

ExecuteFP Multiplication

WriteBack

ExecuteFP Add

WriteBack

ExecuteFP Division

WriteBack

ExecuteFP Multiplication

Stages of a Scoreboard Pipeline

1. Issue —decode instructions & check for structural & WAW hazards (ID)

If a functional unit for the instruction is free (no structural hazards) and no other active instruction has the same destination register (no WAW), the scoreboard issues the instruction to the functional unit and updates its internal data structure.

If a structural or WAW hazard exists, then the instruction issue stalls, and no further instructions will issue until these hazards are cleared.

2. Read operands —wait until no data hazards, then read operands (RO)

A source operand is available if no earlier issued active instruction is going to write it, or if the register containing the operand is being written by a currently active functional unit (no RAW).

When the source operands are available, the scoreboard tells the functional unit to proceed to read the operands from the registers and begin execution. The scoreboard resolves RAW hazards dynamically in this step, and instructions may be sent into execution out of order.

Alwaysdone in programorder

Can bedone out ofprogramorder


3. Execution —operate on operands (EX)

The functional unit begins execution upon receiving operands. When the result is ready, it notifies the scoreboard that it has completed execution.

4. Write result —finish execution (WB)

Once the scoreboard is aware of the fact that the functional unit has completed execution, the scoreboard checks for WAR hazards. If none, it writes results. If WAR, then it stalls the instruction.

Example:

DIVD F0,F2,F4

ADDD F10,F0,F8

SUBD F8,F8,F14

Scoreboard would stall SUBD until ADDD reads operands


1. Instruction status—which of 4 stages the instruction is in

2. Functional unit status—Indicates the state of the functional unit (FU). 9 fields for each functional unit

Busy—Indicates whether the unit is being used or not

Op—Operation to perform in the unit (e.g., + or –)

Fi—Destination register

Fj, Fk—Source-register numbers

Qj, Qk—Functional units producing source registers Fj, Fk

Rj, Rk—Flags indicating when Fj, Fk are ready. Set to No after operands are read.

3. Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions will write that register

Information within the Scoreboard

Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2LD F2 45+ R3MULTDF0 F2 F4SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 NoMult2 NoAdd NoDivide No

Register result status

Clock F0 F2 F4 F6 F8 F10 F12 ... F30FU

Instruction stream

Instruction status:Scoreboard only

records the statusWe will show the

times for each stage, for convenience






FU countdown

Functional Units:1 Integer

2 Multiplication1 Addition1 Division


Source and destination

registers

Which FU willproduce each

operand

OperandsReady?





Clock cycle counter


Which FU will write in each register?

A Scoreboard ExampleThe following code is run on a scoreboard pipeline with:

L.D F6, 34(R2)

L.D F2, 45(R3)

MUL.D F0, F2, F4

SUB.D F8, F6, F2

DIV.D F10, F0, F6

ADD.D F6, F8, F2

Functional Unit (FU) # of FUs EX cycles

Integer Mem 1 1Floating Point Multiply 2 10 Floating Point Add 1 2Floating point Divide 1 40

Functional units are not pipelined

L.D F6, 34(R2)L.D F2, 45(R3)MUL.D F0, F2, F4SUB.D F8, F6, F2DIV.D F10, F0, F6ADD.D F6, F8, F2

123456

L.D F6, 34 (R2)

1

L.D F2, 45 (R3)

2

MUL.D F0, F2, F4

3

DIV.D F10, F0, F6

5

SUB.D F8, F6, F2

4

ADD.D F6, F8, F2

6

Date Dependence:(1, 4) (1, 5) (2, 3) (2, 4) (2, 6) (3, 5) (4, 6)

Output Dependence:(1, 6)

Anti-dependence: (5, 6)

Example Code

Real Data Dependence (RAW)

Anti-dependence (WAR)

Output Dependence (WAW)

Dependency Graph For Example Code

Scoreboard Example Cycle 1

Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1LD F2 45+ R3MULTDF0 F2 F4SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger Yes Load F6 R2 YesMult1 NoMult2 NoAdd NoDivide No


Clock F0 F2 F4 F6 F8 F10 F12 ... F301 FU Integer

Issue LD #1


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2LD F2 45+ R3MULTDF0 F2 F4SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?




LD#1 reads operandsLD #2 can’t issue since integer unit is busyMULT can’t issue because we require in-order issue.Pipeline Stalls

Stall


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3LD F2 45+ R3MULTDF0 F2 F4SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?




LD #1 completes


Instruction status Read ExecutionWriteInstruction j k IssueoperandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3MULTDF0 F2 F4SUBDF8 F6 F2DIVDF10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for jFU for kFj? Fk?

TimeName Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 NoMult2 NoAdd NoDivide No


Clock F0 F2 F4 F6 F8 F10 F12 ... F304 FU

LD #1 writes back and frees Integer FU and

register F6


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5MULTDF0 F2 F4SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?




Issue LD #2 since integer unit is now free.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6MULTDF0 F2 F4 6SUBD F8 F6 F2DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger Yes Load F2 R3 YesMult1 Yes Mult F0 F2 F4 Integer No YesMult2 NoAdd NoDivide No


Clock F0 F2 F4 F6 F8 F10 F12 ... F306 FU Mult1 Integer

Issue MULT.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7MULTDF0 F2 F4 6SUBD F8 F6 F2 7DIVD F10 F0 F6ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger Yes Load F2 R3 YesMult1 Yes Mult F0 F2 F4 Integer No YesMult2 NoAdd Yes Sub F8 F6 F2 Integer Yes NoDivide No


Clock F0 F2 F4 F6 F8 F10 F12 ... F307 FU Mult1 Integer Add

MULT can’t read its operands (F2) because LD #2 hasn’t finished.

SUBD is issued

Scoreboard Example Cycle 8a

Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7MULTDF0 F2 F4 6SUBD F8 F6 F2 7DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger Yes Load F2 R3 YesMult1 Yes Mult F0 F2 F4 Integer No YesMult2 NoAdd Yes Sub F8 F6 F2 Integer Yes NoDivide Yes Div F10 F0 F6 Mult1 No Yes


Clock F0 F2 F4 F6 F8 F10 F12 ... F308 FU Mult1 Integer Add Divide

MULT and SUBD both waiting for F2.

DIVD issues.

Scoreboard Example Cycle 8b

Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6SUBD F8 F6 F2 7DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 Yes Mult F0 F2 F4 Yes YesMult2 NoAdd Yes Sub F8 F6 F2 Yes YesDivide Yes Div F10 F0 F6 Mult1 No Yes


Clock F0 F2 F4 F6 F8 F10 F12 ... F308 FU Mult1 Add Divide

LD #2 writes F2.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger No

10 Mult1 Yes Mult F0 F2 F4 Yes YesMult2 No

2 Add Yes Sub F8 F6 F2 Yes YesDivide Yes Div F10 F0 F6 Mult1 No Yes



Now MULT and SUBD can both read F2.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?






MULT and SUB continue operation

9

1


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?






ADDD can not be issued because add unit is busy.

SUBD completes


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?


7 Mult1 Yes Mult F0 F2 F4 Yes YesMult2 NoAdd NoDivide Yes Div F10 F0 F6 Mult1 No Yes


Clock F0 F2 F4 F6 F8 F10 F12 ... F3012 FU Mult1 Divide

SUBD finishes.

DIVD waits for F0


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?


6 Mult1 Yes Mult F0 F2 F4 Yes YesMult2 NoAdd Yes Add F6 F8 F2 Yes YesDivide Yes Div F10 F0 F6 Mult1 No Yes



ADDD issues.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13 14Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?



2 Add Yes Add F6 F8 F2 Yes YesDivide Yes Div F10 F0 F6 Mult1 No Yes



MULT and ADDDcontinue their operation


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13 14Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?






Nearly there…


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13 14 16Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?






ADDD completes execution







ADDD can’t write because of RAW with DIVDADDD stalls write back







MULT still continuesits execution


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13 14 16Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?





MULT completes execution.


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8ADDDF6 F8 F2 13 14 16Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 NoMult2 NoAdd Yes Add F6 F8 F2 Yes YesDivide Yes Div F10 F0 F6 Yes Yes


Clock F0 F2 F4 F6 F8 F10 F12 ... F3020 FU Add Divide

MULT writes and frees FU and register F0


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8 21ADDDF6 F8 F2 13 14 16Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 NoMult2 NoAdd Yes Add F6 F8 F2 Yes YesDivide Yes Div F10 F0 F6 Yes Yes


Clock F0 F2 F4 F6 F8 F10 F12 ... F3021 FU Add Divide

DIVD can read operands


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8 21ADDDF6 F8 F2 13 14 16 22Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?

Time Name Busy Op Fi Fj Fk Qj Qk Rj RkInteger NoMult1 NoMult2 NoAdd No

40 Divide Yes Div F10 F0 F6 Yes YesRegister result status

Clock F0 F2 F4 F6 F8 F10 F12 ... F3022 FU Divide

Now ADDD can write since WAR removed

ADD FU and register F6 freed

39 cycles later…


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8 21 61ADDDF6 F8 F2 13 14 16 22Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?


0 Divide Yes Div F10 F0 F6 Yes YesRegister result status

Clock F0 F2 F4 F6 F8 F10 F12 ... F3061 FU Divide

DIVD completes execution


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8 21 61 62ADDDF6 F8 F2 13 14 16 22Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?


0 Divide NoRegister result status


DIVD writes back and frees resources

Execution Complete


Instruction status Read ExecutionWriteInstruction j k Issue operandscompleteResultLD F6 34+ R2 1 2 3 4LD F2 45+ R3 5 6 7 8MULTDF0 F2 F4 6 9 19 20SUBD F8 F6 F2 7 9 11 12DIVD F10 F0 F6 8 21 61 62ADDDF6 F8 F2 13 14 16 22Functional unit status dest S1 S2 FU for j FU for k Fj? Fk?


0 Divide NoRegister result status


In-order issueOut-of-order executionOut-of-order completion

• Techniques to deal with data hazards in instruction pipelines by:

– Result forwarding to reduce or eliminate RAW hazards

– Hazard detection hardware to stall the pipeline during hazards

– Compiler-based static scheduling to separate the dependent instructions minimizing actual hazard-prevention stalls in scheduled code (will discuss in detail next week.)

– Uses a hardware-based mechanism to rearrange instruction execution order to reduce stalls dynamically at runtime (dynamic scheduling)

» Better dynamic exploitation of instruction-level parallelism (ILP)

Summary

• The amount of parallelism available among the instructions (chosen from the same basic block)

• The number of score entries (The size of the scoreboard determines the size of the window)

• The number and types of functional units (Structural hazards increase when out of order execution is used)

• The presence of antidependence and output dependences lead to WAR and WAW stalls.

Limitations of Scoreboard

Advanced Pipelining

Documents

Transcript of Advanced Pipelining