Buffer-to-Buffer Credits, Exchanges, and Urban Legendsand ......Throughout the session we debunk the...

Buffer-to-Buffer Credits, Exchanges, and Urban Legendsand Urban Legends

Howard L Johnson BrocadeHoward L. Johnson, Brocade

8 August 2012 (8:00am – 9:00am)Session 12076

Elite 2 (Anaheim Marriott Hotel)

L l St ffLegal Stuff

• Notice• IBM may have patents or pending patent applications covering subject matter

described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

• Any references in this information to non-IBM Web sites are provied for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

• Trademarks• The following terms are trademarks of the International Business MachinesThe following terms are trademarks of the International Business Machines

Corporation in the United States, other countries, or both: FICON® IBM® RedbooksTM System z10TM z/OS® zSeries® z10TM

• Other Company, product, or service names may be trademarks or service marks of othersothers.

Ab t tAbstract

• Performance in a FICON network is influenced by the underlying flow y y gcontrol mechanisms of Fibre Channel. In this session, we examine how Buffer-to-Buffer credits flow from the channel to the control unit. We also look at how exchanges are used in FICON applications and how they change with the introduction of zHPF. During both examinations, we explore the role of the FICON Director in managing Buffer-to-Buffer credits and exchanges over a cascaded network. Throughout the session we debunk the various FICON “UrbanThroughout the session, we debunk the various FICON “Urban Legends” featuring credits and exchanges. Take the opportunity to learn from two of the FICON industry’s leading experts in channel and fabric development and join our sessionfabric development and join our session.

A dAgenda

• Buffer CreditsWhat are the and ho do the ork?• What are they and how do they work?

• How do you fill the pipe?• What if you can’t fill the pipe?• What’s wrong with multiple senders and one receiver?• What happens when the pipes are different sizes?

What’s it like in the real world?• What s it like in the real world?• How do cascades Directors work?

• Exchanges• What are they and how do they work?

• What’s an Exchange?g• How many exchanges are needed?• Can they be "reused?”• Can you have too many exchanges?

• Error Sensitivity• Is FICON more sensitive to errors than FCP?• Is FICON more sensitive to errors than FCP?

• How sensitive are FICON frames to loss or corruption?• What recovery actions are taken?• What are the differences with FCP?

BUFFER CREDITSWhat are they and how do they work?

Wh t i B ff t B ff C dit?What is Buffer-to-Buffer Credit?

• The greater the BB Credit….The greater the BB Credit….• A. The faster frames can be sent• B. The farther apart the two ports can be• C. The larger the frames can be• D. None of the above

Wh t i B ff t B ff C dit?What is Buffer-to-Buffer Credit?

• The greater the BB Credit….The greater the BB Credit….• A.• B. The farther apart the two ports can be• C.• D.

Fl C t lFlow Control

• Related to the devices’ ability to receive and process frames• Manages when frames are coming faster than they can be processed• Manages when frames are coming faster than they can be processed• Dropped frames occur when frames are arriving too fast to be processed

• Frames can only be transmitted when the receiver is ready• Credit establishment communicates the number of frames a device can receive at a timeCredit establishment communicates the number of frames a device can receive at a time• The credit value is exchanged at login• Transmission stops when credit runs out• The receiver indicates when it is ready to receive more frames

Credit exchange at Fabric Login

Host says, “I can receive 40 frames.”

Storage says, “I can receive 16 frames.”

Switch says, “I can receive 8 frames.”

B ff C ditBuffer Credit

• At initialization, the two ports establish creditEach b ffer credit corresponds to a frame (regardless of si e)• Each buffer credit corresponds to a frame (regardless of size)

• Each side can support different values• Credit Count

• If a port doesn’t have credit, it can’t send a frameCredit Count has reached zero• Credit Count has reached zero

• Mechanism limits frame drops

Host thinks, “Good I can send S it h thi kGood, I can send 8 frames without thinking about it.”

Storage thinks, “Good, I can send 8 frames without thinking about it.”

Switch thinks, “OK, I can send 40 frames that way and 16 frames this way, but I have to think about it.”

Credit Count88

Credit Count8

Credit Count40

Credit Count16

Credit accounting after Fabric Login

R i R d (R RDY)Receiver Ready (R_RDY)

• R_RDYU d f li k l l fl t l

• Frame transmissionBB C dit i d t d• Used for link level flow control

• Called buffer-to-buffer credit (BB Credit)

• R_RDY is not a frame• It is a “primitive” so it doesn’t consume

• BB Credit is decremented• Once for each frame transmitted

• When BB Credit = 0• Transmission stops

• Frame received• It is a primitive so it doesn t consume a buffer • R_RDY is sent

• Causes transmitter to increment BB Credit

Credit Accounting during Transmission

Storage says “ICredit Count

1

Host says, “I can send 8 frames without stopping.”

Storage says, I can send 8 frames without stopping.”

1Credit Count

40

Credit Count5

Credit Count16R_RDY

stopping.Switch says, “I can 40 frames that way and 16 frames the other way without stopping.”

R_RDY

Urban Legend:B ff C dit t Z P blBuffer Credits at Zero are a Problem

• Buffer credit determines DISTANCETh di t t d b t d till i t i f ll li k f t• The distance two nodes can be apart and still maintain full link frame rate

• Buffer credit is the number of FRAME buffers• A port provides for it’s NEAREST neighbor for RECEIVING framesp p g• Does NOT have to be symmetrical

• Buffer credit is a FRAME count• Not a data SIZE• Not a data SIZE• A 1 byte frame consumes 1 buffer credit• A 2K byte frame consumes 1 buffer credit

• Number of credits needed is determined by:• Raw Link Speed• Speed of light thru a fiber• Distance between two adjacent nodesDistance between two adjacent nodes

BUFFER CREDITSExample: A full pipe

Initial Conditions

Channel “Perfect” Switch Control Unit

20

Number of B-B Credits the switch advertised to the channel during link init

NOTE: In these animations, both the frames and R_RDY’s are numbered. This is for illustrative purposes only. In reality, neither the frames nor the R_RDY’s are numbered. The arrival of an R_RDY only informs the receiver that A frame has been forwarded, now WHICH frame has been forwarded.

19

11

18

12 12

17

3 123 12

16

3 14 23 14 2

15

5 3 14 25 3 14 2

14

5 3 16 4 25 3 16 4 2

13

7 5 3 16 4 27 5 3 16 4 2

12

78 5 3 16 4 278 5 3 16 4 2

11

9 78 5 36 4 2 19 78 5 36 4 2 1

1

9 78 5 36 410

10

129 78 5 36 410 12

1 2

9

9 78 56 41011 3 129 78 56 41011 3 12

31 2

8

9 78 56101112 3 14 29 78 56101112 3 14 2

1 2 3 4

7

9 78 610111213 5 3 14 29 78 610111213 5 3 14 2

1 2 3 4 5

6

9 781011121314 5 3 16 4 29 781011121314 5 3 16 4 2

1 2 3 4 5 6

5

9 8101112131415 7 5 3 16 4 29 8101112131415 7 5 3 16 4 2

1 2 3 4 5 6 7

4

910111213141516 78 5 3 16 4 2910111213141516 78 5 3 16 4 2

1 2 3 4 5 6 7 8

4

1011121314151617 9 78 5 3 16 4 21011121314151617 9 78 5 3 16 4 2

2 3 4 5 6 7 8 9

4

1112131415161718 9 78 5 3 16 4 2101112131415161718 9 78 5 3 16 4 210

3 4 5 6 7 8 9 10

4

1213141516171819 9 78 5 36 4 210111213141516171819 9 78 5 36 4 21011

4 5 6 7 8 9 1011 1

THIS PAGE INTENTIONALLY LEFT BLANK

BUFFER CREDITSExample: A not so full pipe

Suppose the switch is too far away from the channel for the B-B credit it pp yadvertised to the channel

20

19

11

18

12 12

17

3 123 12

16

3 14 23 14 2

15

5 3 14 25 3 14 2

14

5 3 16 4 25 3 16 4 2

13

7 5 3 16 4 27 5 3 16 4 2

12

78 5 3 16 4 278 5 3 16 4 2

11

78 5 3 16 4 29 78 5 3 16 4 29

10

78 5 3 16 4 2910 78 5 3 16 4 2910

9

78 5 3 16 4 291011 78 5 3 16 4 291011

8

78 5 3 16 4 29101211 78 5 3 16 4 29101211

7

78 5 3 16 4 2910111213 78 5 3 16 4 2910111213

6

78 5 3 16 4 291011121314 78 5 3 16 4 291011121314

5

78 5 3 16 4 29101112131415 78 5 3 16 4 29101112131415

4

78 5 3 16 4 2910111213141516 78 5 3 16 4 2910111213141516

1

3

78 5 3 16 4 291011121314151617 78 5 3 16 4 291011121314151617

1 2

2

78 5 3 16 4 29101112131415161718 78 5 3 16 4 29101112131415161718

1 2 3

1

78 5 3 16 4 2910111213141516171819 78 5 3 16 4 2910111213141516171819

1 2 3 4

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5 6

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5 6 7

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5 6 7 8

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5 6 7 8 9

0

78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920

1 2 3 4 5 6 7 8 9 10

0

78 5 36 4 291011121314151617181920 78 5 36 4 291011121314151617181920

1 2 3 4 5 6 7 8 9 1011 1

0

78 5 36 491011121314151617181920 78 5 36 491011121314151617181920

1 2 3 4 5 6 7 8 9 101112 1 2

0

78 56 491011121314151617181920 78 56 491011121314151617181920

1 2 3 4 5 6 7 8 9 10111213 1 2 3

0

78 5691011121314151617181920 78 5691011121314151617181920

1 2 3 4 5 6 7 8 9 1011121314 1 2 3 4

0

78 691011121314151617181920 78 691011121314151617181920

1 2 3 4 5 6 7 8 9 101112131415 1 2 3 4 5

0

789101112131415161718192021 7891011121314151617181920

2 3 4 5 6 7 8 9 101112

21

13141516 1 2 3 4 5 6

0

8910111213141516171819202122 891011121314151617181920

3 4 5 6 7 8 9 101112

2122

1314151617 1 2 3 4 5 6 7

0

91011121314151617181920212322 91011121314151617181920212322

4 5 6 7 8 9 101112131415161718 1 2 3 4 5 6 7 8

0

1011121314151617181920212324 22 1011121314151617181920212324 22

5 6 7 8 9 10111213141516171819 1 2 3 4 5 6 7 8 9

0

212324 2225 1112131415161718192021

1 2 3 4 5 6 7 8 9 10

2324 2225

6 7 8 9 1011121314151617181920

11121314151617181920

0

212324 222526 12131415161718192021

2 3 4 5 6 7 8 9 10

2324 222526

7 8 9 1011121314151617181920

121314151617181920

11

0

212324 22252627 131415161718192021

3 4 5 6 7 8 9 10

2324 22252627

8 9 1011121314151617181920

1314151617181920

1112

0

212324 2225262728 1415161718192021

4 5 6 7 8 9 10

2324 2225262728

9 1011121314151617181920

14151617181920

111213

0

212324 222526272829 151617181920212324 222526272829

1011121314151617181920

151617181920

5 6 7 8 9 10 11121314

0

212324 22252627282930 1617181920212324 22252627282930

11121314151617181920

1617181920

6 7 8 9 10 1112131415

0

212324 2225262728293031 17181920212324 2225262728293031

121314151617181920

17181920

7 8 9 10 111213141516

0

212324 222526272829303132 181920212324 222526272829303132

1314151617181920

181920

8 9 10 11121314151617

0

212324 22252627282930313233 1920212324 22252627282930313233

14151617181920

1920

9 10 1112131415161718

0

212324 2225262728293031323334 20212324 2225262728293031323334

151617181920

20

10 111213 151617181914

0

212324 222526272829303132333435 212324 222526272829303132333435

1617181920 11121314151617181920

0

212324 22252627282930313233343536 212324 22252627282930313233343536

181920 2117 121314151617181920

0

212324 2225262728293031323334353637 212324 22252627282930313233343536

181920 13141516171819202122

37

0

212324 222526272829303132333435363738 212324 22252627282930313233343536

1920 14151617181920212223

3738

0

212324 22252627282930313233343536373839 212324 22252627282930313233343536

20 15161718192021222324

373839

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

1718192021 25222324

37383940

16

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

17181920212223242526

37383940

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

181920222324252627

37383940

21

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

192022232425262728

37383940

21

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

202223242526272829

37383940

21

0

212324 2225262728293031323334353637383940 212324 22252627282930313233343536

222324252627282930

37383940

21

0

2324 2225262728293031323334353637383940 2324 22252627282930313233343536

222324252627282930

37383940

21 31 21

0

232425262728293031323334353637383940 2324252627282930313233343536

3132

37383940

22232425262728293021 2221

BUFFER CREDITSExample: Multiple Senders and One Receiver

Suppose there are multiple senders to one receiver

20

Suppose there are multiple senders to one receiver

Each sender attempts to send at 100% link speed

20

20

1919

1

1

19

1818

12

12

18

1717

3 12

3 12

17

1616

3 14 2

3 14 2

16

1515

5 3 14 2

5 3 14 2

15

1414

5 3 16 4 2

5 3 16 4 2

14

1313

7 5 3 16 4 2

7 5 3 16 4 2

13

1212

78 5 3 16 4 2

78 5 3 16 4 2

12

1111

78 5 36 4 29

1

1

78 5 3

1

6 4 29

11

1010

78 5 36 49102

1

78 5 36 4910

11

21

101

99

78 56 4910113

1 2

78 56 491011

112

32

19

31

88

78 56910121134

1 2

78 569101211

1122

341 2

841 2

7745

78 6910111213

1 2 3

11223

34

78 6910111213

1 27

45

1 2

66456

7891011121314

1 2 3

11223

45

3

7891011121314

1 2 3

656

1 2 3

55567

89101112131415

1 2 3 4

1122334

45

89101112131415

1 2 3

5567

1 2 3

4 845678

910111213141516

1 2 3 4

11223344

56

910111213141516

1 2 3 4

4678

1 2 3 4

4 946789

1011121314151617

52 3 4

112233445

56

1011121314151617

1 2 3 4

36789

1 2 3 4

3 910

36789

1112131415161718

52 3 4

1122334455

67

1112131415161718

52 3 4

378910

52 3 4

3 1011

3789

10

1213141516171819

5 63 4

1223344556

67

1213141516171819

52 3 4

27891011

52 3 4

2 101112

2789

10

1314151617181920

5 63 4

2233445566

78

1314151617181920

5 63 4

289101112

5 63 4

2 111213

289

10

1415161718192021

5 6 74

2334455667

1415161718192021

5 63 478

15 63 4 8

91011121313

1 11121314

189

10

1516171819202122

5 6 74

3344556677

5 6 7489

1516171819202122

15 6 74 9

101112131414

1 12131415

19

1011

1617181920212223

5 6 7 8

3445566778

89

1617181920212223

5 6 74

091011121314

5 6 74

1415

0 1213141516

09

1011

1718192021222324

5 6 7 8

4455667788

910

1718192021222324

5 6 7 8

0101112131415

5 6 7 8

1516

0 1314151617

0101112

1819202122232425

96 7 8

4556677889

910

18192021222324

5 6 7 8

0101112131415

5 6 7 8

151617

18

0 131415161718

0101112

19202122232425

96 7 8

5566778899

1011

19202122232425

96 7 8

0111213141516

96 7 8

161718

19

0 141516171819

0111213

20212223242526

9 107 8

56677889910

202122232425

101196 7 8

0111213141516

96 7 8

19

161718

1920

0 141516171819

0111213

212223242526

9 107 8

667788991010

2122232425

11129 107 8

26

0121314151617

9 107 8

19

1718

20

2021

0 151617181920

0121314

222324252627

9 10 118

6778899101011

1112

2223242526

9 107 8

0121314151617

9 107 8

19

1718

2021

202122

0 151617181920

0121314

2324252627

9 10 118

77889910101111

12

23242526

9 10 118

27

0 131415161718

9 10 118

1918

202122

212223

0 161718192021

0131415

2425262728

9 10 11 12

788991010111112

12

24252627

9 10 118

0 131415161718

9 10 118

1918

2021222323

21222324

0 161718192021

0131415

25262728

9 10 11 12

8899101011111212

1314

25262728

9 10 11 12

0

19

1415161718

9 10 11 12

192021222324

22232425

0 171819202122

0141516

26272829

1310 11 12

89910101111121213

1314

25262728

9 10 11 12

0

19

1415161718

9 10 11 12

192021222324

22232425

0 171819202122

0

26272829

1310 11 12

141516

991010111112121313

141

26272829

1310 11 12

0

19

15161718

20

1310 11 12

202122232425

23242526

0 181920212223

0151617

27282930

13 1011 12

9101011111212131314

26272829

1310 11 121410

1310 11 12

19

15161718

20202122232425

BUFFER CREDITSExample: Different sized pipes

F t PiFat Pipe / Skinny Pipe

20

8 Gig Link 4 Gig Link

18

12

17

3 12

16

3 14 2

15

5 3 14 2

14

5 3 16 4 2

13

7 5 3 16 4 2

12

78 5 3 16 4 2

11

9 78 5 36 4 2 1

102

9 78 5 36 410 1

1

93

19 78 56 41011

1

2

843

129 78 56101112

1 2

74543

1239 78 610111213

1 2

6456

1239 781011121314

1 2 3

54565

7

12349 8101112131415

1 2 3

844565

78

1234910111213141516

1 2 3 4

9356789

123451011121314151617

1 2 3 4

910

356789

123451112131415161718

52 3 4

1011

278910

6 123451213141516171819

52 3 4

101112

278910

6 234520 13141516171819

5 63 4

111213

1891011

6 234572021 141516171819

53 4 6

11121314

1891011

6 3457202122 1516171819

5 74 6

12131415

09101112

6 3457820212223 16171819

5 74 6

1213141516

09101112

6 457820212223 171819

5 7 86

24

1314151617

010111213

9 6 457820212223 181924

5 7 86

131415161718

010111213

9 6 57823 20212223 1924

97 86

19

141516171819

011121314

10 9 6 57823 2021222324

97 86

1920

141516171819

011121314

10 9 6782325 21222324

97 8 10

2021

151617181920

012131415

11 10 9 6782325 222324

97 8 10

202122

151617181920

012131415

11 10 9 7826 2325 2324

9 118 10

2223

161718192021

013141516

12 11 10 9 7826 2325 24

9 118 10

222324

161718192021

013141516

12 11 10 9 826 232527

9 11 1210

2324

171819202122

014151617

13 12 11 10 9 826 232527

9 11 1210

232423

171819202122

014151617

13 12 11 10 92628 2527

1311 1210

232423

181920212223

015161718

2628 2527

1311 1210

14 13 12 11 10 9

23242325

181920212223

015161718

14 13 12 11 1025 2628 27

1311 12 14

BUFFER CREDITSExample: Real Life?

More real life example: Two senders sending at 30% 50% link rate to one receiver

20

More real life example: Two senders sending at 30% - 50% link rate to one receiver

20

20

1919

1

1

19

1818

12

12

18

1717

3 12

12

18

1616

3 14 2

12

18

1616

3 14 2

3 12

17

1616

3 14 2

3 14 2

16

1515

3 14 25

3 14 2

16

1515

34 25

1

34 2

1

1

16

1515

3452

1

34

11

2

5

115

1

14143

2

456

1

112

2

3456

114

1

1414

5634

21

56

1122

1 2

34

141 2

14144

56

21 3

11223

3

456

1 214

1 2

14144

56

21 3

11223

4

3

31 2

56

1431 2

1414 4

56

21 3

1122334

4

56

31 2

7

1331 2

14145

6

421 3

11223344

4

567

31 213

431 2

1313 5

6

421 3

7

112233445

567

431 213

431 2

1414

67

542 3

112233445

567

431 213

431 2

1313

678

542 3

12233445

67

5

543214

5432

1313 6

78

543

9

223344556

6

7

5432

8

135432

1212 6543

78910

23344556

543

78

6

614

543 6

1313

78910

654

33445566

78

543 614

543 6

1212

7891011

654

3445566

78

54 615

54 6

1313

7891011

65

445566

8

7

754 615

754 6

1313

7891011

65

45566

8

7

75 6

9

1575 6

1414

891011

76

5566

8

77

75 6

9

1575 6

1313

891011

76

12

56677

8

75 6

9

1575 6

1313

91011

876

12

56677

8

8

76

9

1676

14149

1011

87

12

667788

876

9

16876

BUFFER CREDITSExample: Cascaded Directors

7

1

820

1

6

12

820

12

5

3 12

820

3 12

4

3 14 2

820

3 14 2

3 819

15 34 2 1

1

5 34 2

2 818

125 36 4 12

1 2

5 36 4

1 817

3 127 56 4

31 2

3 127 56 4

0 816

3 14 278 56

1 2 3 4

3 14 278 56

0 815

5 3 14 29 78 6

2 3 4 5

5 3 14 29 78 6

0 814

5 3 16 4 278910 5 3 16 4 278910

3 4 5 6

0 813

7 5 3 16 4 2891011 7 5 3 16 4 2891011

4 5 6 7

0 812

78 5 3 16 4 29101112 78 5 3 16 4 29101112

5 6 7 8

0 811

9 78 5 3 16 4 210111213 9 78 5 3 16 4 210111213

6 7 8 9

0 810

9 78 5 3 16 4 21011121314 9 78 5 3 16 4 21011121314

7 8 9 10

0 79

12131415 9 78 5 36 4 21011 112131415

8 9 1011

9 78 5 36 4 21011

1

1

0 68

78 5 36 49101112 1213141516 78 5 36 49101112

1 2

1213141516

9 101112

0 57

14151617 3 1278 56 491011121314151617

10 111213

3 1278 56 4910111213

1 2 3

0 46

78 569101112131415161718 3 14 278 5691011121314

1 2 3 4

15161718

11121314

3 14 2

0 35

5 34 278 6910111213141516171819 5 34 278 69101112131415

1 2 3 4 5

16171819

12131415 1

0 24

5 36 47891011121314151617181920 5 36 478910111213141516

1 2 3 4 5 6

17181920

13141516 1 2

0 13

7 56 489101112131415161718192021 7 56 4891011121314151617

1 2 3 4 5 6 7

18192021

14151617 1 2 3

0 02

78 56910111213141516171819202122 78 569101112131415161718

1 2 3 4 5 6 7 8

19202122

15161718 1 2 3 4

0 01

78 691011121314151617181920212322 78 6910111213141516171819

1 2 3 4 5 6 7 8 9

20212322

16171819 2 3 4 5

0 00

7891011121314151617181920212324 22 78910

1 2 3 4 5 6 7 8 9 10

11121314151617181920212324 22

17181920 3 4 5 6

0 00

891011121314151617181920212324 2225 891011

2 3 4 5 6 7 8 9 10

121314151617181920

11

212324 2225

181920 21 4 5 6 7

BUFFER CREDITSOther Neat Stuff

H h dit d I d?How much credit do I need?

• Good “Rule of thumb”

Number of credits needed = 1 + Link speed in Gb/s * Distance in kmFrame Size in KBFrame Size in KB

Example: 20 km at 1 Gb/s 110

4Gb32 Credits

Example: 20 km at 1 Gb/s1 + 1 * 20 = 11

25060708090

100en

t Dat

a R

ate

Example: 10 km at 4 Gb/s1 + 4 * 10 = 21

24 20 36 52 68 84 100 116 132 148

Distance (Km)

10203040

Per

ce

2 Distance (Km)

H “l ” i f ?How “long” is a frame?

• Traveling at the speed of light, a frame can be very long• At 1G, the length of a frame is about 4-kilometers.• At 2G, the length of a frame is about 2-kilometers.• At 4G, the length of a frame is about 1-kilometer.• At 8G, the length of a frame is about 500-meters.• At 16G, the length of a frame is about 200-meters.

H “f t” i f ?How “fast” is a frame?

• Speed of light in fibreSpeed of light in fibre• 200,000 km/second• 5-microseconds/km

• Transmission Rates in fibre• At 1G, a frame is sent in about 20-microseconds.• At 2G, a frame is sent in about 10-microseconds.t G, a a e s se t about 0 c oseco ds• At 4G, a frame is sent in about 5-microseconds.• At 8G, a frame is sent in about 2.5-microseconds.• At 16G a frame is sent in about 1-microsecond• At 16G, a frame is sent in about 1-microsecond.

EXCHANGES

Th t f t i iThe parts of a transmission

• Frame• Building block of an Fibre Channel connection• Contains the information to be transmitted• The address of the source and destination• Control information

• Sequence• A set of one or more related FramesA set of one or more related Frames• Transmitted unidirectionally from one port to an other

• Exchange• An Exchange is one or more nonconcurrent sequences• An Exchange is one or more nonconcurrent sequences• A single operation• May be unidirectional or bidirectional

Fib Ch l FFibre Channel Frame

The basic building block is the FRAME

SOF Header Payload CRC EOF

4 24 0 – 2048(2112)

4 4

Source/Destination link addressesExchange ID

Protects Frame

Identifies end of a frame

Identifies beginning of a frame

Protocol TypeStart or End of SequenceStart or End of Exchange

Identifies end of a frame

g g

Fi IU E lFicon IU Examples

SOF Header Ficon CRC EOF

1 Frame IU to transfer a Read CCW

SOF Header Header CRC EOF

SOF Header Ficon Header

CCW Data (2016 bytes) CRC EOF

3 Frame IU to transfer 4K of data

Header

SOF Header CCW Data (2048 bytes) CRC EOF

SOF Header CCW Data (32 bytes)

Ficon CRC CRC EOF

Urban Legend:FICON f E h Th FCPFICON uses fewer Exchanges Than FCP

• Fibre Channel Architecture defines an Exchange asg• “A mechanism for identifying and managing an operation between

two ports“

• All IUs (a.k.a. Sequences) that make up a single I/O operation are part of an Exchange

• In Ficon, each concurrent I/O operation uses two Exchanges• One unidirectional Exchange for IUs from the Channel to the CU• A different unidirectional Exchange for IUs from the CU to the• A different unidirectional Exchange for IUs from the CU to the

Channel

• The PAIR is commonly know as a “Ficon Exchange”• The PAIR is commonly know as a Ficon Exchange

S d IUSequences and IUs

• Each Upper Layer Protocol (ULP) defines the contents and format of pp y ( )it’s own Information Units (IUs)• Commands• Data• Status• Control• Etc

• Ficon IUs can be up to 8K (8192) in size• 8160 (8K-32) bytes of data

• 32 bytes contain Ficon Header informationy

• 4 frames are needed for the largest IU• The collection of frame(s) that make up a IU are called a Sequence

• A Sequence may be as small as a single Frame• A Sequence may be as small as a single Frame

H E h d I d?How many Exchanges do I need?

• Little’s Law states:• The number of “things” in a system can be determined by multiplying the

average arrival rate of those “things” by the average time each “thing” stays in the system.

• Applied to Ficon:• The average number of Exchanges active at any given time =

A erage I/O rate * A erage response timeAverage I/O rate * Average response time

• Example: 5000 Ficon I/Os / Second on a given channel with .4ms service time1 needs 2 Active Exchanges (pairs) at any given timeservice time needs 2 Active Exchanges (pairs) at any given time

1 The amount of time the I/O is active in the channel

ERROR SENSITIVITY

Urban Legend:FICON i iti th FCPFICON is more sensitive than FCP

• Is Ficon More Sensitive to Errors than FCP?• Reasons for Link Errors are the same

• Is a Ficon frame more likely to get lost, damaged or corruptedIs a Ficon frame more likely to get lost, damaged or corrupted than FCP?• The probability is the same• Frames are framesFrames are frames

• When a Ficon frame gets lost, damaged or corrupted, is the recovery action different from FCP?recovery action different from FCP?• Both protocols retry when errors occur

• FCP by the Device Driver• Ficon by IOS/ERPy

S Wh t th Diff ?So What are the Differences?

• z Operating Systemsz Operating Systems tend to provide more detailed messagesg

• Ficon does provide additional debug data and actions• RNID

Source: FC-FS-3 INCITS/T11 Draft Standard v0 92• Link Error Status Blocks• Extensive State Change

Processing

Source: FC FS 3 INCITS/T11 Draft Standard v0.92

See www.t11.org

Processing

SUMMARYThank you

SSummary

• Buffer Credits• Distance• Flow Control

• Exchanges• Unidirectional• Bidirectional

• Error Sensitivity• Error Sensitivity• Recovery• Reporting

SHARE A h i A t 2012SHARE, Anaheim, August 2012Buffer-to-Buffer Credits, Exchanges, and Urban LegendsSession 12076

THANK YOU!

Session 12076Howard L. Johnson, Brocade

REFERENCES

S k Bi hSpeaker Biography

• Howard L. JohnsonHoward L. Johnson• BROCADE• Technology Architect, FICON

28 t h i l d l t d t• 28 years technical development and management

• Contact Information• howard johnson@brocade [email protected]


• Lou RicciLou Ricci• IBM

• 32-years• 24 years in channel development• 24-years in channel development• An inventor of FICON

• FICON Firmware Team Leader• Contact Information

• [email protected]


• Patty DrieverPatty Driever• IBM• System z I/O and Networking Technologist

• Contact Information• [email protected]

BONUS SLIDES

S d f Li htSpeed of Light

• 299 792 458 meters per second• In a vacuum• In a vacuum• National Institute of Standards and Technology

• http://www.nist.gov/pml/wmd/metric/length.cfm• 300 000 000 meters per second (300 000 kilometers per second)

• Common approximation• Wikipedia

• http://en.wikipedia.org/wiki/Speed_of_light• 1.44 – 1.46

• Refractive index of 1300 nanometer fiber• Encyclopedia of Laser Physics and Technologyy p y gy

• http://www.rp-photonics.com/refractive_index.html• 1.5

• Common approximation• Wikipedia

• http://en wikipedia org/wiki/Optical fiber#Materialshttp://en.wikipedia.org/wiki/Optical_fiber#Materials• 200 000 000 meters per second (200 000 kilometers per second)

• Calculated speed of light in single-mode fiber• V = C/N• 200 000 000 = 300 000 000 / 1.5

C l t d fi iti i b d li ht i fib t li 1 kil t i 5 i d• Common latency definition is based on light in fiber traveling 1-kilometer in 5-microseconds

C l l ti B ff C ditCalculating Buffer Credits• Axioms

• Fibre Channel Frames are held in BuffersFibre Channel Frames are held in Buffers• Receiver memory is measured in frame buffers• Transmitter counts frames buffers used by counting frames sent

• Objective is to maintain full link bandwidth• Enough Receiver memory is required to receive all of the frames that can be transmitted in the time it takes to return

the “Receiver Ready” acknowledgement• Includes Frames “in flight”

• Frames transmitted but not yet received (i.e. frames on the link)• Includes time to respond to frames received

• R_RDY acknowledgement• Increments Transmitter frame count

• Definitions• Buffers are a function of distance

• Distance == Rate * Time• D == RT

• Buffers == f(Distance)• B == f(D)• B == f(RT)

• Rate is the Average Frame Transmission RateR• R

• Time is a function of the speed of light in a fiber and distance• Time == Distance / Rate

• Rate is the speed of light in a fiber• T == D / 200,000,000 meters/second• B == f(R * (D / 2 * 108 meters/second))

A F T i i R tAverage Frame Transmission Rate• Frame Transmission Rate

• Number of frames transmitted per secondNumber of frames transmitted per second• Next frame is transmitted as soon as the previous frame completes

• While transmission count is non-zero• Number of frames transmitted depends on the amount of data per frame

• Average Frame Transmission Rate• Effective Data Transmission Rate / Average Frame Density• R == eX / aF

• Effective Data Transmission Rate• Scale * Baud Rate * Encoding Factor• eX == S * bR * E• eX == S * (1.0625 * 108 bytes/second)

Average Frame Density• Average Frame Density• Frame Header is 36-bytes• Frame Payload is 0 – 2112-bytes• aF == 36-bytes + Average Payload bytes• aF == 36 + A bytes/frame

• Average Frame Transmission RateAverage Frame Transmission Rate• R == eX / aF• R == (S * (1.0625 * 108 bytes/second)) / ((36 + A) bytes/frame)

• R is expressed in frames / second

Eff ti D t T i i R tEffective Data Transmission Rate• Effective Data Transmission Rate

• Scale * Baud Rate * Encoding FactorScale Baud Rate Encoding Factor• eX == S * bR * E

• Scale• Fibre Channel baud rate scalars

• 1, 2, 4, 8 Gbs• Baud Rate

• Fibre Channel baud rate• The baud rate is 1.0625 * 109

• bR == S * (1.0625 * 109 bits/second)• Encoding Factor

• Fibre Channel encoding factors• For 1, 2, 4, and 8 Gbs, , ,

• 8-data bits / 10-transmission bits• 8b10b

• Data bits per second• eX == S * (1.0625 * 109 bits/second) * 8-data bits/10-transmission bits• eX == S * (8.5 * 108 bits/second)

• Data bytes per second• Data bytes per second• eX == S * (8.5 * 108 bits/second) * 1-byte/8-bits• eX == S * (1.0625 * 108 bytes/second)

• Arithmetic• eX == S * (1.0625 * 109 bits/second) * (8-bits/10-bits) * (1-byte/8-bits)• eX == S * (1 0625 * 108 bytes/second)eX S (1.0625 10 bytes/second)

B ff C dit C l l tiBuffer Credit Calculation

• Function• B == f(R * T)• B == f(R T)• B == R frames/second * T seconds

• Rate• Rate == Effective Data Transmission Rate / Average Frame Density• R == eX / aF• R == (S * (1.0625 * 108 bytes/second)) / ((36 + A) bytes/frame)

• If the latency of the fibre is known• B == (S * (1.0625 * 108 bytes/second)) / ((36 + A) bytes/frame) * T seconds

• B is the number of buffers expressed as frames• S is the scale (1, 2, 4, or 8)( , , , )• A is the average bytes per frame• T round-trip time

• If the distance of the fibre is known• T == D / 200,000,000 meters/second• B == R frames/second * (D / 200 000 000 meters/second)B R frames/second (D / 200,000,000 meters/second)• B == (S * (1.0625 * 108 bytes/second)) / ((36 + A) bytes/frame) * (D / (2 * 108 meters/second)))

• B is the number of buffers expressed as frames• S is the scale (1, 2, 4, or 8)• A is the average bytes per frame• D round-trip distance in metersp

E d t E d C ditEnd to End Credit

• Device to Device Flow ControlDevice to Device Flow Control• Between source and destination

• Not the linksSi il t b ff t b ff fl t l• Similar to buffer-to-buffer flow control• At N_Port Login• Report available receive buffers (EE_Credit)• Transmitter counts buffers transmitted (EE_Credit_CNT)• Receiver acknowledges frame (ACK)

• ACK 1 (a single data frame in a sequence) – most common• ACK n (several (N) consecutive data frames in a sequence)• ACK 0 (all data frames in a sequence) – not used

Vi t l Ch lVirtual Channels

• Technology to allocate BB Credits to particular data flowsTechnology to allocate BB_Credits to particular data flows• Class F traffic has one data flow• Assigned with Zoning by using special Zone names

THIS SLIDE INTENTIONALLY LEFT BLANK

Buffer-to-Buffer Credits, Exchanges, and Urban Legendsand ......Throughout the session we debunk the...

Documents

Transcript of Buffer-to-Buffer Credits, Exchanges, and Urban Legendsand ......Throughout the session we debunk the...