18-447
Computer Architecture
Lecture 7: Pipelining
Prof. Onur Mutlu
Carnegie Mellon University
Spring 2015, 1/30/2015
Agenda for Today & Next Few Lectures
Single-cycle Microarchitectures
Multi-cycle and Microprogrammed Microarchitectures
Pipelining
Issues in Pipelining: Control & Data Dependence Handling, State Maintenance and Recovery, …
Out-of-Order Execution
Issues in OoO Execution: Load-Store Handling, …
2
Recap of Last Lecture
Multi-cycle and Microprogrammed Microarchitectures
Benefits vs. Design Principles
When to Generate Control Signals
Microprogrammed Control: uInstruction, uSequencer, Control Store
LC-3b State Machine, Datapath, Control Structure
An Exercise in Microprogramming
Variable Latency Memory, Alignment, Memory Mapped I/O, …
Microprogramming
Power of abstraction (for the HW designer)
Advantages of uProgrammed Control
Update of Machine Behavior
3
Review: A Simple LC-3b Control and Datapath
4
C.4. THE CONTROL STRUCTURE 9
Microinstruction
R
Microsequencer
BEN
x2
Control Store
6
IR[15:11]
6
(J, COND, IRD)
269
35
35
Figure C.4: The control structure of a microprogrammed implementation, overall block
diagram
on the LC-3b instruction being executed during the current instruction cycle. This state
carries out the DECODE phase of the instruction cycle. If the IRD control signal in the
microinstruction corresponding to state 32 is 1, the output MUX of the microsequencer
(Figure C.5) will take its source from the six bits formed by 00 concatenated with the
four opcode bits IR[15:12]. Since IR[15:12] specifies the opcode of the current LC-
3b instruction being processed, the next address of the control store will be one of 16
addresses, corresponding to the 14 opcodes plus the two unused opcodes, IR[15:12] =
1010 and 1011. That is, each of the 16 next states is the first state to be carried out
after the instruction has been decoded in state 32. For example, if the instruction being
processed is ADD, the address of the next state is state 1, whose microinstruction is
stored at location 000001. Recall that IR[15:12] for ADD is 0001.
If, somehow, the instruction inadvertently contained IR[15:12] = 1010 or 1011, the
Simple Design
of the Control Structure
A Simple Datapath
Can Become
Very Powerful
Review: The Power of Abstraction
The concept of a control store of microinstructions enables the hardware designer with a new abstraction: microprogramming
The designer can translate any desired operation to a sequence of microinstructions
All the designer needs to provide is
The sequence of microinstructions needed to implement the desired operation
The ability for the control logic to correctly sequence through the microinstructions
Any additional datapath elements and control signals needed (no need if the operation can be “translated” into existing control signals)
7
Review: Advantages of Microprogrammed Control
Allows a very simple design to do powerful computation by controlling the datapath (using a sequencer)
High-level ISA translated into microcode (sequence of u-instructions)
Microcode (u-code) enables a minimal datapath to emulate an ISA
Microinstructions can be thought of as a user-invisible ISA (u-ISA)
Enables easy extensibility of the ISA
Can support a new instruction by changing the microcode
Can support complex instructions as a sequence of simple microinstructions
Enables update of machine behavior
A buggy implementation of an instruction can be fixed by changing the microcode in the field
8
Wrap Up Microprogrammed Control
Horizontal vs. Vertical Microcode
Nanocode vs. Microcode vs. Millicode
Microprogrammed MIPS: An Example
9
Horizontal Microcode
A single control store provides the control signals
10
Microprogram counter
Address select logic
Adder
1
Input
Datapath
control
outputs
Microcode
storage
Inputs from instruction
register opcode field
Outputs
Sequencing
control
[Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
ALUSrcA
IorD
IRWrite
PCWrite
PCWriteCond
….
n-bit mPC input
k-b
it c
on
tro
l sig
nal
ou
tpu
t
Control Store: 2n k bit (not including sequencing)
MIPS design From P&H, Appendix D
Vertical Microcode Two-level control store: the first specifies abstract operations
11
Microprogram counter
Address select logic
Adder
1
Input
Datapath
control
outputs
Microcode
storage
Inputs from instruction
register opcode field
Outputs
Sequencing
control
“PC PC+4”
“PC ALUOut”
“PC PC[ 31:28 ],IR[ 25:0 ],2’b00”
“IR MEM[ PC ]”
“A RF[ IR[ 25:21 ] ]”
“B RF[ IR[ 20:16 ] ]”
…………. …….
ROM
ALU
SrcA
IorD
IRW
rite
PC
Write
PC
WriteC
on
d
….
[Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
If done right (i.e., m<<n, and m<<k), two ROMs together (2nm+2mk bit) should be smaller than horizontal microcode ROM (2nk bit)
m-bit input
k-bit control signal output
n-bit mPC input
1-bit signal means do this RT (or combination of RTs)
Nanocode and Millicode
Nanocode: a level below traditional microcode
microprogrammed control for sub-systems (e.g., a complicated floating-point module) that acts as a slave in a microcontrolled datapath
Millicode: a level above traditional microcode
ISA-level subroutines that can be called by the microcontroller to handle complicated operations and system functions
E.g., Heller and Farrell, “Millicode in an IBM zSeries processor,” IBM JR&D, May/Jul 2004.
In both cases, we avoid complicating the main u-controller
You can think of these as “microcode” at different levels of abstraction
12
Nanocode Concept Illustrated
13
ROM
mPC
arithmetic datapath
a “mcoded” FPU implementation
ROM
mPC
processor datapath
a “mcoded” processor implementation
We refer to this as “nanocode” when a mcoded subsystem is embedded in a mcoded system
Microcoded Multi-Cycle MIPS Design
Any ISA can be implemented with a microprogrammed microarchitecture
P&H, Appendix D: Microprogrammed MIPS design
We will not cover this in class
However, you can do an extra credit assignment for Lab 2
14
Microcoded Multi-Cycle MIPS Design
15 [Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Control Logic for MIPS FSM
16 [Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Microprogrammed Control for MIPS FSM
17 [Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Multi-Cycle vs. Single-Cycle uArch
Advantages
Disadvantages
You should be very familiar with this right now
18
Microprogrammed vs. Hardwired Control
Advantages
Disadvantages
You should be very familiar with this right now
19
Can We Do Better?
What limitations do you see with the multi-cycle design?
Limited concurrency
Some hardware resources are idle during different phases of instruction processing cycle
“Fetch” logic is idle when an instruction is being “decoded” or “executed”
Most of the datapath is idle when a memory access is happening
20
Can We Use the Idle Hardware to Improve Concurrency?
Goal: More concurrency Higher instruction throughput
(i.e., more “work” completed in one cycle)
Idea: When an instruction is using some resources in its processing phase, process other instructions on idle resources not needed by that instruction
E.g., when an instruction is being decoded, fetch the next instruction
E.g., when an instruction is being executed, decode another instruction
E.g., when an instruction is accessing data memory (ld/st), execute the next instruction
E.g., when an instruction is writing its result into the register file, access data memory for the next instruction
21
Pipelining
22
Pipelining: Basic Idea
More systematically:
Pipeline the execution of multiple instructions
Analogy: “Assembly line processing” of instructions
Idea:
Divide the instruction processing cycle into distinct “stages” of processing
Ensure there are enough hardware resources to process one instruction in each stage
Process a different instruction in each stage
Instructions consecutive in program order are processed in consecutive stages
Benefit: Increases instruction processing throughput (1/CPI)
Downside: Start thinking about this… 23
Example: Execution of Four Independent ADDs
Multi-cycle: 4 cycles per instruction
Pipelined: 4 cycles per 4 instructions (steady state)
24
Time
F D E W
F D E W
F D E W
F D E W
F D E W
F D E W
F D E W
F D E W
Time
Is life always this beautiful?
The Laundry Analogy
“place one dirty load of clothes in the washer”
“when the washer is finished, place the wet load in the dryer”
“when the dryer is finished, take out the dry load and fold”
“when folding is finished, ask your roommate (??) to put the clothes away”
25
- steps to do a load are sequentially dependent
- no dependence between different loads - different steps do not share resources
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Pipelining Multiple Loads of Laundry
26
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
- latency per load is the same
- throughput increased by 4
- 4 loads of laundry in parallel
- no additional resources
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Pipelining Multiple Loads of Laundry: In Practice
27
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
the slowest step decides throughput
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Pipelining Multiple Loads of Laundry: In Practice
28
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
A
B
A
B
throughput restored (2 loads per hour) using 2 dryers
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Time76 PM 8 9 10 11 12 1 2 AM
A
B
C
D
Task
order
Task
order
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
An Ideal Pipeline
Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)
Repetition of identical operations
The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)
Repetition of independent operations
No dependencies between repeated operations
Uniformly partitionable suboperations
Processing can be evenly divided into uniform-latency suboperations (that do not share resources)
Fitting examples: automobile assembly line, doing laundry
What about the instruction processing “cycle”?
29
Ideal Pipelining
30
combinational logic (F,D,E,M,W) T psec
BW=~(1/T)
BW=~(2/T) T/2 ps (F,D,E) T/2 ps (M,W)
BW=~(3/T) T/3 ps (F,D)
T/3 ps (E,M)
T/3 ps (M,W)
More Realistic Pipeline: Throughput
Nonpipelined version with delay T
BW = 1/(T+S) where S = latch delay
k-stage pipelined version
BWk-stage = 1 / (T/k +S )
BWmax = 1 / (1 gate delay + S )
31
T ps
T/k ps
T/k ps
Latch delay reduces throughput
(switching overhead b/w stages)
More Realistic Pipeline: Cost
Nonpipelined version with combinational cost G
Cost = G+L where L = latch cost
k-stage pipelined version
Costk-stage = G + Lk
32
G gates
G/k G/k
Latches increase hardware cost
Pipelining Instruction Processing
33
Remember: The Instruction Processing Cycle
Fetch
Decode
Evaluate Address
Fetch Operands
Execute
Store Result
34
1. Instruction fetch (IF) 2. Instruction decode and register operand fetch (ID/RF) 3. Execute/Evaluate memory address (EX/AG) 4. Memory operand fetch (MEM) 5. Store/writeback result (WB)
Remember the Single-Cycle Uarch
35
Shift left 2
PC
Instruction memory
Read address
Instruction [31– 0]
Data memory
Read data
Write data
RegistersWrite register
Write data
Read data 1
Read data 2
Read register 1
Read register 2
Instruction [15–11]
Instruction [20–16]
Instruction [25–21]
Add
ALU result
Zero
Instruction [5– 0]
MemtoReg
ALUOp
MemWrite
RegWrite
MemRead
Branch
JumpRegDst
ALUSrc
Instruction [31–26]
4
M u x
Instruction [25–0] Jump address [31– 0]
PC+4 [31–28]
Sign extend
16 32Instruction [15–0]
1
M u x
1
0
M u x
0
1
M u x
0
1
ALU control
Control
AddALU
result
M u x
0
1 0
ALU
Shift left 2
26 28
Address
PCSrc2=Br Taken
PCSrc1=Jump
ALU operation
bcond
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
T BW=~(1/T)
Dividing Into Stages
36
200ps
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instruction
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
Address
Data
memory1
ALU result
M u x
ALU
Zero
IF: Instruction fetch ID: Instruction decode/
register file read
EX: Execute/
address calculation
MEM: Memory access WB: Write back
Is this the correct partitioning? Why not 4 or 6 stages? Why not different boundaries?
100ps 200ps 200ps 100ps
RF write
ignore for now
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Instruction Pipeline Throughput
37
Instruction
fetchReg ALU
Data
accessReg
8 nsInstruction
fetchReg ALU
Data
accessReg
8 nsInstruction
fetch
8 ns
Time
lw $1, 100($0)
lw $2, 200($0)
lw $3, 300($0)
2 4 6 8 10 12 14 16 18
2 4 6 8 10 12 14
...
Program
execution
order
(in instructions)
Instruction
fetchReg ALU
Data
accessReg
Time
lw $1, 100($0)
lw $2, 200($0)
lw $3, 300($0)
2 nsInstruction
fetchReg ALU
Data
accessReg
2 nsInstruction
fetchReg ALU
Data
accessReg
2 ns 2 ns 2 ns 2 ns 2 ns
Program
execution
order
(in instructions)
200 400 600 800 1000 1200 1400 1600 1800
200 400 600 800 1000 1200 1400
800ps
800ps
800ps
200ps 200ps 200ps 200ps 200ps
200ps
200ps
5-stage speedup is 4, not 5 as predicted by the ideal model. Why?
Enabling Pipelined Processing: Pipeline Registers
38 T
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instruction
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
Address
Data
memory1
ALU result
M u x
ALU
Zero
IF: Instruction fetch ID: Instruction decode/
register file read
EX: Execute/
address calculation
MEM: Memory access WB: Write back
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Data
memory
Address
No resource is used by more than 1 stage!
IRD
PC
F
PC
D+4
PC
E+4
nP
CM
AE
BE
Imm
E
Ao
ut M
B
M
MD
RW
A
ou
t W
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
T/k ps
T/k ps
Pipelined Operation Example
39
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw
Address
Data
memory
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Instruction decode
lw
Address
Data
memory
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw
Address
Data
memory
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Instruction decode
lw
Address
Data
memory
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Execution
lw
Address
Data
memory
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
Data
memory
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Memory
lw
Address
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write data
Read dataData
memory
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Write back
lw
Write register
Address
97108/Patterson
Figure 06.15
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
Data
memory
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Memory
lw
Address
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write data
Read dataData
memory
1
ALU result
M u x
ALU
Zero
ID/EX MEM/WB
Write back
lw
Write register
Address
97108/Patterson
Figure 06.15
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0
Address
Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
Data
memory
1
ALU result
M u x
ALU
Zero
ID/EX
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
All instruction classes must follow the same path and timing through the pipeline stages.
Any performance impact?
Pipelined Operation Example
40
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction decode
lw $10, 20($1)
Instruction fetch
sub $11, $2, $3
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw $10, 20($1)
Address
Data
memory
Address
Data
memory
Clock 1
Clock 2
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction decode
lw $10, 20($1)
Instruction fetch
sub $11, $2, $3
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw $10, 20($1)
Address
Data
memory
Address
Data
memory
Clock 1
Clock 2
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
3216Sign
extend
Write register
Write data
Memory
lw $10, 20($1)
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
sub $11, $2, $3
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
lw $10, 20($1)
Instruction decode
sub $11, $2, $3
3216Sign
extend
Address
Data
memory
Data
memory
Address
Clock 3
Clock 4
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
3216Sign
extend
Write register
Write data
Memory
lw $10, 20($1)
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
sub $11, $2, $3
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
lw $10, 20($1)
Instruction decode
sub $11, $2, $3
3216Sign
extend
Address
Data
memory
Data
memory
Address
Clock 3
Clock 4
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uction
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
lw $10, 20($1)
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uctio
n
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
sub $11, $2, $3
Memory
sub $11, $2, $3
Address
Data memory
Address
Data
memory
Clock 6
Clock 5
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uction
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
lw $10, 20($1)
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uctio
n
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
sub $11, $2, $3
Memory
sub $11, $2, $3
Address
Data memory
Address
Data
memory
Clock 6
Clock 5
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Is life always this beautiful?
Illustrating Pipeline Operation: Operation View
41
MEM
EX
ID
IF Inst4
WB
IF
MEM
IF
MEM
EX
t0 t1 t2 t3 t4 t5
ID
EX IF ID
IF ID
Inst0 ID
IF Inst1
EX
ID
IF Inst2
MEM
EX
ID
IF Inst3
WB
WB MEM
EX
WB
steady state
(full pipeline)
Illustrating Pipeline Operation: Resource View
42
I0
I0
I1
I0
I1
I2
I0
I1
I2
I3
I0
I1
I2
I3
I4
I1
I2
I3
I4
I5
I2
I3
I4
I5
I6
I3
I4
I5
I6
I7
I4
I5
I6
I7
I8
I5
I6
I7
I8
I9
I6
I7
I8
I9
I10
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
IF
ID
EX
MEM
WB
Control Points in a Pipeline
43
PC
Instruction memory
Address
Instr
uction
Instruction [20– 16]
MemtoReg
ALUOp
Branch
RegDst
ALUSrc
4
16 32
Instruction [15– 0]
0
0Registers
Write register
Write data
Read data 1
Read data 2
Read register 1
Read register 2
Sign extend
M u x
1Write
data
Read
data M u x
1
ALU
control
RegWrite
MemRead
Instruction [15– 11]
6
IF/ID ID/EX EX/MEM MEM/WB
MemWrite
Address
Data memory
PCSrc
Zero
AddAdd
result
Shift
left 2
ALU
result
ALU
Zero
Add
0
1
M u x
0
1
M u x
Identical set of control points as the single-cycle datapath!!
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Control Signals in a Pipeline
For a given instruction
same control signals as single-cycle, but
control signals required at different cycles, depending on stage
Option 1: decode once using the same logic as single-cycle and buffer signals until consumed
Option 2: carry relevant “instruction word/field” down the pipeline and decode locally within each or in a previous stage
Which one is better?
44
Control
EX
M
WB
M
WB
WB
IF/ID ID/EX EX/MEM MEM/WB
Instruction
Pipelined Control Signals
45
PC
Instruction memory
Instr
uctio
n
Add
Instruction [20– 16]
Me
mto
Re
g
ALUOp
Branch
RegDst
ALUSrc
4
16 32Instruction [15– 0]
0
0
M u x
0
1
AddAdd
result
RegistersWrite register
Write data
Read data 1
Read data 2
Read register 1
Read register 2
Sign extend
M u x
1
ALU result
Zero
Write data
Read data
M u x
1
ALU control
Shift left 2
Re
gW
rite
MemRead
Control
ALU
Instruction [15– 11]
6
EX
M
WB
M
WB
WBIF/ID
PCSrc
ID/EX
EX/MEM
MEM/WB
M u x
0
1
Me
mW
rite
Address
Data memory
Address
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
Remember: An Ideal Pipeline
Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)
Repetition of identical operations
The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)
Repetition of independent operations
No dependencies between repeated operations
Uniformly partitionable suboperations
Processing an be evenly divided into uniform-latency suboperations (that do not share resources)
Fitting examples: automobile assembly line, doing laundry
What about the instruction processing “cycle”?
46
Instruction Pipeline: Not An Ideal Pipeline
Identical operations ... NOT!
different instructions not all need the same stages Forcing different instructions to go through the same pipe stages external fragmentation (some pipe stages idle for some instructions)
Uniform suboperations ... NOT!
different pipeline stages not the same latency Need to force each stage to be controlled by the same clock
internal fragmentation (some pipe stages are too fast but all take the same clock cycle time)
Independent operations ... NOT! instructions are not independent of each other
Need to detect and resolve inter-instruction dependencies to ensure the pipeline provides correct results pipeline stalls (pipeline is not always moving)
47
Issues in Pipeline Design
Balancing work in pipeline stages
How many stages and what is done in each stage
Keeping the pipeline correct, moving, and full in the presence of events that disrupt pipeline flow
Handling dependences
Data
Control
Handling resource contention
Handling long-latency (multi-cycle) operations
Handling exceptions, interrupts
Advanced: Improving pipeline throughput
Minimizing stalls
48
Causes of Pipeline Stalls
Stall: A condition when the pipeline stops moving
Resource contention
Dependences (between instructions)
Data
Control
Long-latency (multi-cycle) operations
49
Dependences and Their Types
Also called “dependency” or less desirably “hazard”
Dependences dictate ordering requirements between instructions
Two types
Data dependence
Control dependence
Resource contention is sometimes called resource dependence
However, this is not fundamental to (dictated by) program semantics, so we will treat it separately
50
Handling Resource Contention
Happens when instructions in two pipeline stages need the same resource
Solution 1: Eliminate the cause of contention
Duplicate the resource or increase its throughput
E.g., use separate instruction and data memories (caches)
E.g., use multiple ports for memory structures
Solution 2: Detect the resource contention and stall one of the contending stages
Which stage do you stall?
Example: What if you had a single read and write port for the register file?
51
Data Dependences
Types of data dependences
Flow dependence (true data dependence – read after write)
Output dependence (write after write)
Anti dependence (write after read)
Which ones cause stalls in a pipelined machine?
For all of them, we need to ensure semantics of the program is correct
Flow dependences always need to be obeyed because they constitute true dependence on a value
Anti and output dependences exist due to limited number of architectural registers
They are dependence on a name, not a value
We will later see what we can do about them
52
Data Dependence Types
53
Flow dependence r3 r1 op r2 Read-after-Write r5 r3 op r4 (RAW)
Anti dependence r3 r1 op r2 Write-after-Read r1 r4 op r5 (WAR) Output-dependence r3 r1 op r2 Write-after-Write r5 r3 op r4 (WAW) r3 r6 op r7
Pipelined Operation Example
54
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction decode
lw $10, 20($1)
Instruction fetch
sub $11, $2, $3
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw $10, 20($1)
Address
Data
memory
Address
Data
memory
Clock 1
Clock 2
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction decode
lw $10, 20($1)
Instruction fetch
sub $11, $2, $3
Instruction
memory
Address
4
32
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Instruction fetch
lw $10, 20($1)
Address
Data
memory
Address
Data
memory
Clock 1
Clock 2
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
3216Sign
extend
Write register
Write data
Memory
lw $10, 20($1)
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
sub $11, $2, $3
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
lw $10, 20($1)
Instruction decode
sub $11, $2, $3
3216Sign
extend
Address
Data
memory
Data
memory
Address
Clock 3
Clock 4
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uctio
n
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
3216Sign
extend
Write register
Write data
Memory
lw $10, 20($1)
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
sub $11, $2, $3
Instruction
memory
Address
4
0
AddAdd
result
Shift
left 2
Instr
uction
IF/ID EX/MEM MEM/WB
M u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
Write register
Write data
Read data
1
ALU result
M u x
ALU
Zero
ID/EX
Execution
lw $10, 20($1)
Instruction decode
sub $11, $2, $3
3216Sign
extend
Address
Data
memory
Data
memory
Address
Clock 3
Clock 4
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uction
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
lw $10, 20($1)
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uctio
n
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
sub $11, $2, $3
Memory
sub $11, $2, $3
Address
Data memory
Address
Data
memory
Clock 6
Clock 5
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uction
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
lw $10, 20($1)
Instruction
memory
Address
4
32
0
AddAdd
result
1
ALU result
Zero
Shift
left 2
Instr
uctio
n
IF/ID EX/MEMID/EX MEM/WB
Write backM u x
0
1
Add
PC
0Write data
M u x
1
Registers
Read data 1
Read data 2
Read register 1
Read register 2
16Sign
extend
M u x
ALURead data
Write register
Write data
sub $11, $2, $3
Memory
sub $11, $2, $3
Address
Data memory
Address
Data
memory
Clock 6
Clock 5
Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]
What if the SUB were dependent on LW?
Data Dependence Handling
55
Readings for Next Few Lectures
P&H Chapter 4.9-4.11
Smith and Sohi, “The Microarchitecture of Superscalar Processors,” Proceedings of the IEEE, 1995
More advanced pipelining
Interrupt and exception handling
Out-of-order and superscalar execution concepts
56
How to Handle Data Dependences
Anti and output dependences are easier to handle
write to the destination in one stage and in program order
Flow dependences are more interesting
Five fundamental ways of handling flow dependences
Detect and wait until value is available in register file
Detect and forward/bypass data to dependent instruction
Detect and eliminate the dependence at the software level
No need for the hardware to detect dependence
Predict the needed value(s), execute “speculatively”, and verify
Do something else (fine-grained multithreading)
No need to detect
57
Interlocking
Detection of dependence between instructions in a pipelined processor to guarantee correct execution
Software based interlocking
vs.
Hardware based interlocking
MIPS acronym?
58
Approaches to Dependence Detection (I)
Scoreboarding
Each register in register file has a Valid bit associated with it
An instruction that is writing to the register resets the Valid bit
An instruction in Decode stage checks if all its source and destination registers are Valid
Yes: No need to stall… No dependence
No: Stall the instruction
Advantage:
Simple. 1 bit per register
Disadvantage:
Need to stall for all types of dependences, not only flow dep.
59
Not Stalling on Anti and Output Dependences
What changes would you make to the scoreboard to enable this?
60
Approaches to Dependence Detection (II)
Combinational dependence check logic
Special logic that checks if any instruction in later stages is supposed to write to any source register of the instruction that is being decoded
Yes: stall the instruction/pipeline
No: no need to stall… no flow dependence
Advantage:
No need to stall on anti and output dependences
Disadvantage:
Logic is more complex than a scoreboard
Logic becomes more complex as we make the pipeline deeper and wider (flash-forward: think superscalar execution)
61
Once You Detect the Dependence in Hardware
What do you do afterwards?
Observation: Dependence between two instructions is detected before the communicated data value becomes available
Option 1: Stall the dependent instruction right away
Option 2: Stall the dependent instruction only when necessary data forwarding/bypassing
Option 3: …
62
We did not cover the following slides in lecture.
These are for your preparation for the next lecture.
Data Forwarding/Bypassing
Problem: A consumer (dependent) instruction has to wait in decode stage until the producer instruction writes its value in the register file
Goal: We do not want to stall the pipeline unnecessarily
Observation: The data value needed by the consumer instruction can be supplied directly from a later stage in the pipeline (instead of only from the register file)
Idea: Add additional dependence check logic and data forwarding paths (buses) to supply the producer’s value to the consumer right after the value is available
Benefit: Consumer can move in the pipeline until the point the value can be supplied less stalling
64
A Special Case of Data Dependence
Control dependence
Data dependence on the Instruction Pointer / Program Counter
65
Control Dependence
Question: What should the fetch PC be in the next cycle?
Answer: The address of the next instruction
All instructions are control dependent on previous ones. Why?
If the fetched instruction is a non-control-flow instruction:
Next Fetch PC is the address of the next-sequential instruction
Easy to determine if we know the size of the fetched instruction
If the instruction that is fetched is a control-flow instruction:
How do we determine the next Fetch PC?
In fact, how do we know whether or not the fetched instruction is a control-flow instruction?
66
Top Related