1

October, 2007

PICMG Hardware Platform Management

2

Introduction to Pigeon Point Systems

• Delivering world-class management components for open modular platforms– Focused on delivering outstanding PICMG management

controllers– Dependable for specification compliance, interoperability

and responsive support– Proven in dozens of companies and with years of

interoperability experience• Founded in 1997, headquartered in Scotts Valley, California• Please visit www.pigeonpoint.com for more details

• See www.pigeonpoint.com/library.html#presentations for the most recent update of this presentation

3

Broad Range of Tier 1 Customers for Pigeon Point™ Platform Management

• Notable examples in key market segments:– Telecom Equipment Manufacturers: Alcatel-Lucent, Nokia,

Siemens– Integrated platform providers: Motorola ECC, Sun

Microsystems– ATCA/AMC board developers: above plus Adlink Technology,

Advantech, Bittware, CES, Diversified Technology, Emerson NP Embedded Computing, Interphase, McAfee, NMS Communications, Qualcomm, Portwell, Znyx Networks

– Electronic packaging companies (shelf vendors): Pentair/Schroff/Electronic Solutions, Rittal/Kaparel

• See www.pigeonpoint.com/customers.html for public list

Pigeon Point, µShelf Manager™ and µCarrier Manager™ are trademarks of Pigeon Point Systems

4

Pigeon Point Leadership Cited by Venture Development Corporation (VDC)

• May, 2007 VDC (www.vdc-corp.com) study of ATCA/µTCA market assessed “build vs. buy” choices for management subsystems

• Key VDC conclusions include:– Most Shelf Managers shipped with ATCA platforms are

based on designs from Pigeon Point– Among IPM Controllers not designed in-house, nearly all

are based on Pigeon Point’s off-the-shelf designs– Most µTCA platform vendors use or plan to use shelf

management technology from their primary supplier for ATCA shelf management technology, Pigeon Point Systems

5

The PICMG “TCA” Landscape

• Six years of development for 4 major spec families; goal to maximize sharing of platform management facilities

• Specification development complemented and strengthened by regular interoperability testing

• Result: sophisticated, but complex, platform management architecture based on IPMI, tuned to PICMG needs

6

Overall Purpose of PICMG Hardware Platform Management

• Monitor & control low-level aspects of boards, modules and other Field Replaceable Units within a shelf

• Watch over basic health of the shelf and its constituent FRUs, report anomalies, take corrective action when needed

• Provide accessibility for inventory information & sensor readings

• Archive (and forward, as appropriate) event reports and failure notifications from boards, modules and other intelligent FRUs

• Manage power, cooling & interconnect resources in the shelf• Enable visibility into a shelf for a logical System Manager—

some mix of software + “swivel chair folk”

7

Overall Approach to PICMG Hardware Platform Management

• Focus on low level h/w management– Mandatory facilities on all boards, modules and shelves

• Adopt Intelligent Platform Management Interface (IPMI) 1.5 Revision 1.1 as foundation– IPMI widely used in PC and Server industry

• Emphasize interoperability among independently implemented components

• Encourage and leverage ecosystem of “off-the-shelf”management components– Re-using such components can be a win for product

developers • Allowing them to focus on their value adds• Ensuring excellent interoperability when mature,

compliant components are chosen– Pigeon Point components used as examples here

8

PICMGHardware Platform Management

Part 1

9

Example ATCA/AMC Shelf, with Dedicated Shelf Management Controllers (ShMCs)

10

AdvancedTCA Shelf Manager: Key Services

• Provide access to inventory information for all FRUs• Manage power consumption and backplane interconnects

– Using self-describing requirements in FRU info• Manage cooling resources for the shelf, responding to FRU-

configured temperature threshold events• Manage distributed collection of sensors

– Self-describing in Sensor Data Records (SDRs)• Collect events in a persistent store and optionally perform

configurable actions in response; IPMI facilities include:– Non-volatile System Event Log that stores N event

records intended for interpretation via SDRs– Platform Event Filtering (PEF) that provides mechanism

for configurable actions on events (such as pages or SNMP traps)

• Provide visibility to System Manager on all the above as desired

11

AdvancedTCA Shelf Manager: Key Services (Cont.)

• System Manager Interface: logical connection to shelf-external management– Specification-required for interoperability:

• IPMI LAN Interface, including Remote Management Control Protocol (RMCP)

– Likely Shelf Manager-specific additions: command line & SNMP interfaces

• Optionally dual redundant Shelf Manager– Assumed from same vendor; coordination protocols not

covered in specs• Specification allows broad implementation freedom for

logical Shelf Manager function

12

One Way to Build Dedicated ShMCs: Use Off-the-shelf Mezzanine Modules

• SO-DIMM-sized Pigeon Point ShMM-500R + shelf-specific carrier yields dedicated ShMC– Avoids massive “from-scratch” development– Small size of mezzanine makes for great physical

flexibility for dedicated ShMC placement in the shelf• See www.pigeonpoint.com/library.html#userdocs for Pigeon

Point Shelf Manager user documentation

13

ShMC Redundancy and Full Connectivity to System Manager

• Spec provides for dual ShMC redundancy w/ auto failover of ShMC IP addresses– Details of coordination are

left to implementation– ShMM-500R details

shown• R2.0 ECN-001 (ShMC cross-

connects) allows both ShMCs to be connected with both hubs for better robustness– Mandated by CP-TA ICD– ECN development led by

Pigeon Point

14

System Manager Interface

• Multiple Sys Man interfaces on most ATCA Shelf Managers– IPMI LAN Interface (RMCP) the

only ATCA-required interface– Typical ShMCs have at least

SNMP & command line, also• Problem: only RMCP has spec-

defined framework and semantics– RMCP is quite low level– SNMP is widely interesting, but:

• Management Info Base (MIB) is necessarily Shelf-Manager-specific

15

SA Forum’s Hardware Platform Interface (HPI) Provides a Solution

• HPI, layered above RMCP, can provide specification-defined:– Platform management API– SNMP MIB implementation

• Portable System Manager uses RMCP or either HPI interface– Preserves System Manager

investments• SAF’s HPI-to-ATCA Mapping

specification provides crucial compatibility assurance

16

OpenHPI: an Effective HPI Enabler

• Provides highly adaptable foundation with plug-ins for particular platform types (see www.openhpi.org)

• Already used for wide range of platforms, including ATCA & IBM BladeCenter

• Remote Procedure Call (RPC) enables HPI clients that are remote from OpenHPI daemon

• Pigeon Point OpenHPI has plug-in that delivers SAF mapping spec compliance w/ Pigeon Point Shelf Manager– Communicates via RMCP– On-/off-ShMM HPI service– Pigeon Point User Guide posted:

www.pigeonpoint.com/library.html#userdocs

17

AdvancedTCA IPM Controller (IPMC): Key Facilities

• IPMI-1.5-specified commands and FRU Information• Key AdvancedTCA extensions:

– Dual redundant IPMB-0 connection to Shelf Manager– Hot swap state management for FRUs (including

represented FRUs)– Electronic keying (commands + FRU Info) for point-to-

point & bused backplane interconnects– LED management, including color, lamp test– Fan control for interoperable fan trays– Payload power control & negotiation w/ shelf

18

IPM Controller Interfaces and Physical Size

19

Overall Approach to Layering AdvancedMC Facilities Above ATCA

• Fit smoothly into established PICMG 3.0 conventions• Avoid impacting PICMG 3.0 R1.0 Shelf Managers w/ AMC.0• Reduce requirements on Module Management Controller

(MMC) to limit its cost and footprint• Require Carrier IPM Controller (IPMC) to represent MMC as

a full-fledged ATCA FRU to ShMC; Carrier IPMC:– Does power negotiation on AMC’s behalf– Does AMC E-Keying similarly to ATCA, but without

involvement of Shelf Manager– Makes Module SDRs visible to Shelf Manager

• Preserve IPMI foundation in MMC– Full IPMI sensor infrastructure optionally available for

MMC sensors

20

IPMC Build Option: Acquire Mature Solution(Pigeon Point Example Here)

• Integration-ready schematics

• Bench top HW for ramp up• Renesas H8S/216x and

Atmel AVR variants• Minimal board footprint

• Firmware source code• Common firmware base

for H8S and AVR• GNU tool chain supplied• Designed for easy

configuration

21

Renesas H8S-based ATCA IPMC and AMC Carrier IPMC; Atmel AVR-based MMC

• Single H8S for IPMC or Carrier IPMC– H8S/216x family used pervasively

in IPMI management applications– Excellent headroom for future

growth:• 33 Mhz 16-bit CPU• Up to 512 KB Flash• 40 KB SRAM• 6 I2C ports

– Allows advanced features:• LPC as payload interface• Serial over LAN with Intel

82571 NICs• Atmel AVR excellent for MMC role• Additional logic block per AMC site

22

Recent/Upcoming Developments with ATCA/AMC

• AMC.0 R2.0 adopted in November, 2006– Many clarifications and other improvements for hardware

platform management– New clock E-Keying facility

• PICMG HPM.1, the IPM Controller Firmware Upgrade spec, adopted in May, 2007– Spec development led by Pigeon Point; see Pigeon

Point article below for introduction and background– “Interoperable Firmware Upgrades for PICMG

Management Controllers”, RTC Magazine, August 2007• PICMG 3.0 R3.0 now under development in PICMG

23

HPM.1 Based Firmware Upgrades Address Key Need in PICMG Platform Management

HPM.1:• Defines interfaces for Upgrade Agents & IPM Controllers• Defines format for upgrade images w/ opaque upgrade data• Allows agent to upgrade firmware in arbitrary controllers

24

PICMG 3.0 R3.0 Progress Report

• Key focus: substantial requirements engineering edit– All requirements numbered for precise references:

• Within PICMG 3.0 and from other PICMG specs• Within profiles defined by related organizations such

as SCOPE or CP-TA• Within ATCA customer/vendor specifications

– Requirements restructured, if necessary, for that goal• Additional substantial effort on clarifications, accounting for

many of 1,700+ CRs (Change Requests) beyond enumeration-related changes– ~500 of those CRs in hardware platform management

• Important functional enhancements/extensions, also• Earliest adoption date: January, 2008

25

Other Functional Enhancements in Preliminary PICMG 3.0 R3.0

• Defined IPMI interface for telco alarm monitoring/control– Can be represented either by Shelf Manager or in-shelf

IPM Controller– Benefits generic System Manager applications

• Improved LED services, including identification of textual and graphic legends associated with each LED– Enables more complete/explicit operator interfaces in

generic System Manager applications• New “Set FRU Extracted” command provides standardized

mechanism for System Manager to communicate operator knowledge of physical FRU removal

• Additional option for IPMB-0 buffers on intelligent FRUs– Simplified buffer can allow improved noise margins

26

PICMGHardware Platform Management

Part 2

27

Hardware Platform Management Elements for µTCA

28

Overall Approach to Hardware Platform Management in µTCA

• Support AMC.0 AMC modules without change– Provide “carrier” environment that is equivalent to an

ATCA carrier in terms of interfaces with the AMCs• Reuse as much as possible of existing PICMG management

infrastructure– Including concepts, terminology, possibly

implementations– Minimize barriers to reuse of ATCA System Manager

applications with µTCA• Choose architectural approaches that facilitate the cost and

modularity goals of the µTCA initiative

29

µTCA Shelf Manager (ShM): Local or Remote to Carrier Manager

• ShM co-located w/ Carrier Manager– Especially relevant for a low-

end, one-carrier shelf– Interface between Shelf &

Carrier Managers is private• ShM located remotely from

Carrier Manager– Shelf-Carrier Manager

Interface mandated as IP– Certainly the choice for a

multi-carrier shelf– Many options for “off-MCH”

Shelf Managers

30

µTCA Shelf Manager Responsibilities

• Represent the entire shelf (including up to 16 MicroTCA carriers, each represented by a Carrier Manager)– Preserve representation offered by an ATCA Shelf

Manager as much as possible• Each carrier appears as an IPMC-managed “board”

with managed FRUs (all the modules in that carrier)– Maintain System Event Log and Sensor Data Record

Repository• Requires tracking shelf-wide FRU population

– Remote Management Control Protocol interface to System Manager mandated, as with ATCA

• Manage cooling for the shelf– Cooling Units in one carrier may cool other carrier(s)– Shelf Manager has data structure that establishes

relationships between “cooler(s)” and “coolee(s)”

31

Non-responsibilities for µTCA Shelf Manager, Compared to ATCA

• No power management; done by Carrier Manager, instead– Tight coordination with in-carrier Power Modules

necessary• No E-Keying; done by Carrier Manager, instead

– Specification addresses only intra-carrier E-Keying– Inter-carrier fabric connections are out of scope

• No use of IPMB-0 to communicate with “boards” (actually, carriers) in the shelf– Shelf-Carrier Manager Interface replaces IPMB-0 as

shelf-wide management link with carriers, via their Carrier Managers

32

Carrier Manager and MCMC(s) Provide Intra-Carrier Management

• Logical Carrier Manager represents entire carrier to Shelf Manager– Uses bused IPMB-0 to

connect with EMMCs– Uses radial IPMB-Ls to

connect with MMCs– Uses I2C to access Carrier

FRU Info Device– Can be implemented on dual

redundant basis• MicroTCA Carrier Management

Controller (MCMC) provides local management for MCH– Sensors for MCH and E-

Keying for MCH fabrics

33

Carrier Manager “Owns” All Resources of its MicroTCA Carrier

• FRUs (including AMCs, Power Modules, Cooling Units, possibly OEM Modules) presented in fixed ranges of FRU IDs for each type– Example: Shelf Manager needs to change fan speeds

on a Cooling Unit• Issues fan speed command to Carrier Manager w/

appropriate FRU ID• Carrier Manager forwards to appropriate CU

• Carrier Manager handles power budgeting, allocation, monitoring for all Power Modules in the carrier– Similar to AMC.0 Carrier IPMC responsibilities, but using

independently implemented modular Power Modules• Carrier-wide Device Sensor Data Repository merges sensor

data from all modules in the carrier– Sensor data for a given module is added/subtracted from

carrier-wide repository when module inserted/extracted

34

MCMC Handles Local Management of its MCH Module

• Implementations may combine the MCMC function with the Carrier Manager function on one MCH-based microcontroller

• Nevertheless, these are distinct logical functions, which is especially important in dual MCH architectures:– One logical Carrier Manager for entire carrier executes

on one of the two MCHs, can switch over if necessary– Two MCMCs are always active, since they represent the

local resources of each MCH, including sensors & fabrics

• Carrier FRU Info stored in I2C SEEPROM allows MCH (inc. Carrier Manager, MCMC) to automatically adapt to carrier specifics

35

µTCA Power Modules (PMs)

• Need to both:– Preserve power-related

interfaces to AMCs– Support modular

implementations with optional redundancy

• PMs provide:– Radial power channel to

each module site– Presence-detect & enabling

for module (E)MMCs– Autonomous power on for

selected FRUs at startup– Optional redundant PM with

hardware takeover for one failed peer

• PMs rely on Carrier Manager for policy decisions

36

µTCA Cooling Units

• Support fan management commands originally designed for ATCA

• Scope of cooling effects may cross MicroTCA Carrier boundaries

• Fan geography data structures in Shelf FRU Info define fan <-> FRU relationships

• Like PMs, managed by Enhanced MMCs (EMMCs)– Equivalent to MMCs,

but with dual IPMB-0

37

AMC.0-compliant AMCs Operate in Either µTCA or ATCA Carriers

• Interfaces presented by the carrier are functionally identical– Redundancy of fabric

and IPMB-L connections may need special care

• Principles that motivated AMC.0 architecture benefit MicroTCA.0, as well

38

Key µTCA Choices for Hardware Platform Management with AMC Modules

• Fully preserve ATCA carrier interface for hardware platform management

• Require carrier controller to represent MMC as a full-fledged FRU to µTCA Shelf Manager; Carrier Manager:– Does power budgeting/allocation on AMC’s behalf,

directly and without involvement of Shelf Manager– Does AMC E-Keying similarly to ATCA, but without

involvement of Shelf Manager– Makes SDRs for all modules in the µTCA carrier visible

to Shelf Manager as if owned by the Carrier Manager

39

Recent/Upcoming Developments with µTCA/AMC

• AMC.0 R2.0 adopted in November, 2006– Many clarifications/improvements for hardware platform

management– New clock E-Keying facility

• PICMG HPM.1, the IPM Controller Firmware Upgrade spec, adopted in May, 2007– Applies to MTCA.0, AMC.0 and PICMG 3.0 controllers– Spec development led by Pigeon Point

• Some revision areas for MTCA.0 already queued, but no active revision effort for base spec under way– Main queued revision area: synching w/ AMC.0 R2.0– Also under way: Rugged µTCA, with a first “dot spec”

(MTCA.1) focusing on air-cooled contexts

40

Rugged µTCA: Hardware Platform Management Needs

• Few areas of necessary extension identified thus far– One key need: a way for system components to self-

identify temperature ranges and ruggedness levels

41

µTCA Management Controller Build Option: Acquire Existing Solution (e.g. Pigeon Point)

• Integration-ready schematics• Bench top HW for ramp up• MMC w/ Atmel AVR• MCMC, EMMC w/ Renesas H8S

• Firmware source code• GNU tool chain supplied• Simple and flexible

configuration

42

Pigeon Point MCH ManagementSolution Widely Used

• Covers MCMC, µCarrier Manager, µShelf Manager• Excellent interoperability with other compliant µTCA

components demonstrated in PICMG (µ)TCA-IWs• Used in MCHs from Emerson, Motorola and Advantech (see

above) with others in development

43

• 15 TCA interop workshops, 14 w/ management testing– ~15-25 participating companies in each event

• ~40 test plans for different functional areas– Developed by participants; PPS provided major subset

• Workshops add emphasis on new PICMG architectures as testable products become available– Primary focus of main workshops has been ATCA/AMC– µTCA emphasis now ramping up

• In addition, µTCA-IWs have provided additional focus on µTCA– 9 µTCA-IWs thus far, including #9 in Germany, 9/07

• Communications Platform Trade Association (CP-TA) aims to augment PICMG Interops with formal testing– Initial focus is ATCA components– Expected to add formal testing for AMC & µTCA, also

PICMG TCA-IWs Validate Specs and Products

44

Implementing PICMG Management Controllers Takes Serious Engineering

• Strongly consider using existing components

• Pick validated, interop-tested components

• Participate in TCA-IWs with your own products

• Six years of PICMG h/wplatform management spec work has yielded:– PICMG 3.0: 187 pages– AMC.0: 98 pages– MTCA.0: 110 pages– HPM.1: 66 pages

45

Summary: ATCA/µTCA/AMC Hardware Platform Management…

• Represents a significant PICMG effort to: – Define interoperable extensions to IPMI– Maintain and enhance these extensions through

interoperability testing and further specification work– Maintain maximum consistency among different platform

architectures such as ATCA, AMC and µTCA• Constitutes a serious engineering project at any of the shelf,

board or module levels– Even if development starts from existing PICMG 2.9 or

conventional IPMI components• Can significantly benefit from the use of off-the-shelf

components that are validated and interoperability-tested

46

Mark Overgaard

mark@pigeonpoint.com

Thank you!!

47

Speaker Background

Mark Overgaard founded Pigeon Point Systems in 1997 to focus on products and services supporting the adoption of open modular platforms to replace proprietary architectures in the telecommunications market. He is a leader in the technical subcommittees of PICMG, including the management aspects of AdvancedTCA, AdvancedMC and MicroTCA. The Pigeon Point platform management components are widely used for shelf, board and module level hardware platform management for all three of the major PICMG platform architectures. Previously Mark was VP, Engineering at Lynx Real-Time Systems (a Unix-compatible RTOS supplier) and TeleSoft (a major supplier of embedded development solutions for Ada).

mark@pigeonpoint.com