MSR Command Cell Facility
description
Transcript of MSR Command Cell Facility
![Page 1: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/1.jpg)
WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
MSR Command Cell Facility
Fred Kuhns
Applied Research Laboratory
![Page 2: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/2.jpg)
2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Highlights• Overview
• High level description - Application Layer
• MSR Command Interface Overview
• Cell format and field definitions
• Example
![Page 3: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/3.jpg)
3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Definitions• Session: Open connection between the CP
and a specific SPC. Intended to represent open connections and command state
• Transaction: Represent a complete command. A transaction terminates with either an EOF is received by the CP or and error occurs.
• EOF: End of File is returned to CP with the last bit of command data is returned or in response to a Cancel message (or an error occurs)
![Page 4: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/4.jpg)
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Overview - Cmd Interface on CP• Synchronous Request/Response protocol
• Timeout can be specified as well as the number of retries - Per session option– Essentially provides a reliable service– Issue: if no reply, cmd/reply msg lost in port,
channel or CP. Retries may be a bad thing.
• Address - MSR Port and Command – <MSR_Port, MSR_Command>
• Message destination - Callback function within the Port’s kernel (implements command)
![Page 5: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/5.jpg)
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Command Interface on CP• Types of messages:
– New Command, Get Next set of reply data Command, Cancel Command
– Error Reply, EOF Reply, Continued Reply
• Message Identifiers - Only requires a sequence number initialized to 0 for each New Command:– One sending entity on CP, – One outstanding command for each port,– Ports send exactly one reply msg per command msg,– Command must fit within one cell,– Replies may span multiple cells.
![Page 6: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/6.jpg)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Command Interface on Port• Callback function registered with MSR kernel
and called under 3 cases:– New Command
• Flags = Command; Sequence = 0; Length = valid bytes in buffer; Buffer = application data
– Next Command • Flags = Command | Next; Sequence = previous+1; Length
= valid bytes in buffer; Buffer = application data
– Cancel Command • Flags = Command | Cancel; Sequence = previous+1;
Length = 0; Buffer contains no valid data
![Page 7: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/7.jpg)
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Command Interface on Port• Callback function must:
– Read from/Write to supplied buffer– Set length = Bytes written to buffer (in/out param)– Indicate if an error occurred (return -1)– Whether more data exists (return 0 => EOF, return > 0
=> Not EOF, return < 0 => ERROR | EOF)
• Framework: – generates reply message using same Command value
and Sequence number.– sets flags indicating status (EOF, Error etc)
![Page 8: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/8.jpg)
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Failure Modes• Library support for lost messages:
– if (timeout > 0, Replies > 0), then CP API library will re-send with RETRY flag set.
– if (timeout > 0, Replies = 0 or all replies failed), then API library returns error to application
– If (timeout = 0 - No Timeout), then send operation blocks indefinitely.
• Lost Command message - – if (timeout > 0 and retries > 0), CP resends command;
same sequence number but RETRY flag set. Command buffer and flags passed to callback fn.
![Page 9: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/9.jpg)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Failure Modes• Lost Reply message,
– if no retries, Any issues?– if retries then CP resends
• New Command - Port knows this is a duplicate command (RETRY flag). Application responsible for handling retries. If an issue can use unique message ids. Extreme case use a history (last reply message).
• Next Command - Port receives Command w/Sequence > 0, w/RETRY flag. Passed to application which chooses the correct course of action. The intent is to ensure there are no holes in the reply data received by the CP.
• Cancel message - same as Next command.
![Page 10: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/10.jpg)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Possible Enhancements• Support asynchronous messaging:
– Multiple outstanding commands per port– Asynchronous I/O on CP– Speed up boot process and dynamic configuration– Facilitates implementing port monitoring (ping or
heartbeat) for fault detection and recovery. – two methods for reporting results:
• upcall - function registered by application is called when results arrive
• poll - application periodically polls library for results.
• Support Broadcast and/or Multicast
![Page 11: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/11.jpg)
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Layer• Simple messaging facility optimized for MSR.
• Command message (CP sends):– Sent by CP to a specific MSR port (unicast)– Must fit within one AAL0 cell.– Message header, includes:
• protocol version
• Command
• Sequence number
• flags
– Application data follows header– Library implements Request/Reply protocol.
![Page 12: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/12.jpg)
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Layer• Reply Message (Port sends):
– Port must send reply message in response to a Command message.
– Reply message Header:• version and sequence number: same as command msg.
• Includes application data and flags indicating if command was successful and if more data exists (EOF).
– Application registers command specific callback function at port.
– Callback function must conform to specified interface.
![Page 13: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/13.jpg)
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Overvew
• Command Protocol description– Control Processor sends command messages to a
specific port and expects to receive a reply message indicating either Success or Failure. This is termed a Command Cycle.
– There is the notion of a Command Transaction which may include one or more command cycles. A command transaction is terminated when the target (port) responds with a reply msg containing an EOF
![Page 14: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/14.jpg)
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Overvew• Command Protocol description, continued
– CP processing of Reply msg, depends on EOF flag:• If EOF is set then no further reply data is available and
the command transaction is closed.
• If EOF is not set then there is remaining data and the command transaction is still open.
– If remaining data (Not EOF), then CP must follow with a either a Next or Cancel command message.
• Sequence number indicates the “chunk” of data to be returned.
• Command indicates the message’s destination
• sequence number = previous + 1
![Page 15: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/15.jpg)
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
APIC Cell FormatM
SR C
omm
and
Mes
sage
81624 0cidldccpoutx xxpin
clgfc vpi vci pti
x x x x x x x x x x x x x
ver length command/status sequence number flags
• Cell payload contains the MSR Command• Command header is 4 Bytes, leaving 44 Bytes
for sub-commands and data.
![Page 16: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/16.jpg)
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
ATM/APIC Header
• pin (Ports-In) - Port cell arrived– Tx not used (set to 000b)– Rx: port cell arrived on (is the below correct?)
• 001 Port 0, 010 Port 1, 100 Port 2, etc.
• pout (Ports-Out) - Set of output ports.– Tx: Command library sets:
• 001 Fiber/Link, 010 Ribbon/Switch, 011 Both
• 101 Loopback MV0, 110 Loopback MV1
– Rx: Set by VCXT, see pin above.
cidldccpoutx xxpin
clgfc vpi vci pti
x x x x x x x x x x x x x
![Page 17: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/17.jpg)
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
ATM/APIC Cell Format
• cc (Control Cell Indicator) - Not used, set to 0b
• ld (Low Delay) - Not used, set to 0b. – Should we use low delay?
• cid (Connection Identifier) - set to vci value.
• gfc (Generic Flow Control) - set to 0000b.
cidldccpoutx xxpin
clgfc vpi vci pti
x x x x x x x x x x x x x
![Page 18: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/18.jpg)
18WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
ATM/APIC Cell Format
cidldccpoutx xxpin
clgfc vpi vci pti
x x x x x x x x x x x x x
• vpi (Virtual Path Identifier) - Set to 0x0.
• vci (Virtual Circuit Identifier) - Equal to cid. – See presentation on MSR configurations for a
complete list of VCI assignments.
• pti (Payload Type) - Set to 000b (data cell)
• cl (Cell Loss Priority) - Set to 0b (High Priority)
![Page 19: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/19.jpg)
19WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Header
• Version (2 bits) - Protocol version. Allows for at most 4 versions. Current version set to 0.– field width was a trade off with the length field.
• Length (6 bits) - Number of valid data bytes.– 0 <= Length <= 44, so 6 bits sufficient.– This field is indirectly set by the application or
command implementation. The CP library and kernel interfaces allow for applications to pass a buffer pointer and indicate the number of valid data bytes.
ver length command/status sequence number flags
![Page 20: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/20.jpg)
20WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Header
• Command/Status (8 Bits) CP inserts command value, SPC/port inserts status information.– Valid Commands are listed in $SYS/msr/msr_ctl.h,
also see $MSR/utils/command/*.{c,h}– Library API on CP accepts Command as argument.
implementation in kernel - array of function pointers, uses Command as index
– Reply msg Status indicating success or an error code (Upcall, ATM, Cmd Invalid, Cmd Not Implemented, or Other Cmd Error).
ver length command/status sequence number flags
![Page 21: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/21.jpg)
21WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Current Commands• Current set of commands:
– set_debug/get_debug - set debug flags/masks. – port_init - set local port number.– policy - Manage MSR policy object. Currently
sets the Policy control flags (Gate 0 enable)– dq - Preshanth?– fipl -Ed?– rp_pcu - extensive set of sub-commands.– rp_inst - send message to plugin instance– rp_class - send message to plugin base class
![Page 22: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/22.jpg)
22WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR DEBUG Command• Dynamic setting of debug mask - affects
what messages are sent to syslogd
• set_debug or get_debug command– Valid debug categories/modules
• apic, ipfwd, iprx, iptx, mem, dq, stats, ctl, conf, kern, natm, pcu, plugin, gate
– Valid debug levels - 0 - 255• predefined: verbose, warning, error, critical
• Interface in MSR kernelMSR_DEBUG((MSR_DEBUG_<category>|MSR_DEBUG_LEVEL_<level>, “format”, args));
![Page 23: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/23.jpg)
23WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR RP_PCU Command• rp_pcu Command sub-commands
– addfltr/remfltr - add/remove pkt filter at gate x– flist - port prints current gate x filter list– bind/unbind - bind/unbind instance to fltr/gate combo– create - create plugin instance– free - remove plugin instance– clist - port prints current plugin class list– ilist - port prints current instance list– load - load plugin - not implemented – null - no-op. Can be used as a ping operation
![Page 24: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/24.jpg)
24WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Header
• Sequence Number (8 Bits) - Is of primary use by the applications. – When command message first sent, sequence = 0.– If the reply does not include an EOF flag, then CP
increments sequence by one for each subsequent command message.
– When EOF is received the Command Transaction is complete and the sequence number is reset to 0.
ver length command/status sequence number flags
![Page 25: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/25.jpg)
25WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Command Header
• Flags (8 bits) - Bit field, valid flags are:– Invalid - flag = 0, should not occur– CMD - cell contains a valid command from CP– REPLY - cell contains reply from Port– ERROR - Reply only, error processing on Port – EOF - No reply data remains, end of cmd transaction– NEXT - get next set of reply data– CANCEL - cancel current cmd transaction– RETRY - set if cp resend a command after it was lost
ver length command/status sequence number flags
![Page 26: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/26.jpg)
26WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
CP Library API• Library API for application on CP,
– int sendcmd(int sid, int cmd, char *data, int flags, int *dlen)
• sid = session id, • cmd - Command to execute on port• data = buffer pointer, • flags =
– RETRY (reply timeout),
– CANCEL (cancel current command),
– Next (get next set of reply data)
![Page 27: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/27.jpg)
27WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
MSR Kernel API• MSR kernel interface $SYS/msr/msr_ctl.{h,c}• Callback function signature:
– msr_ctl_<cmd> (void *buf, u_int8_t flags, u_int8_t seq, u_int8_t *dlen)
– buf = command buffer w/application data,
– flags = • CMD,
• NEXT,
• RETRY or
• CANCEL,
– seq = sequence number indicating reply data set, and
– dlen is input/output parameter, data length in bytes.
![Page 28: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/28.jpg)
28WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Kernel State Diagram
Command
Closed
Next Retry
Idle
Retry
Com
man
d
Proto
Error
CommandEO
F
Proto Error
Cance
l
![Page 29: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/29.jpg)
29WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
CP Library State Diagram
Wait (for reply)
Closed
Next Retry
Idle
Proto
col E
rror Command
EOF
Protocol Error
Open Session
Result of a timeout
![Page 30: MSR Command Cell Facility](https://reader035.fdocuments.in/reader035/viewer/2022062301/56815911550346895dc63d7f/html5/thumbnails/30.jpg)
30WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 1/9/01
Example Sending Cmd to Port
CP
Next/PrevHop
Next/PrevHop
Next/PrevHop
Next/PrevHop
Next/PrevHop
Next/PrevHop
Next/PrevHop
wugsP0
P1
P2
P3
P4
P5
P6
P7
192.168.200.X
192.168.201.X
192.168.202.X
192.168.203.X
192.168.204.X
192.168.205.X
192.168.206.X
192.168.207.XSPC/FPX
SPC/FPX
SPC/FPX
SPC/FPX
SPC/FPX
SPC/FPX
SPC/FPX
SPC/FPX
DQ
DQ
DQ
DQ DQ
DQ
DQ
DQ
192.168.203.2
192.168.202.2
sendcmd();create plugin instance:
port id = 0,PluginID = 200
cmddata
cell hdr
msr_ctl
reply();plugin instance created:
Status,Instance ID
Report command completion status
to application.
Lookup sub-commandperform function call
then report results