Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.
-
Upload
maud-little -
Category
Documents
-
view
213 -
download
1
Transcript of Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.
![Page 1: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/1.jpg)
Scalable name and address resolution infrastructure-- Ira Weiny/John Fleck
#OFADevWorkshop
![Page 2: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/2.jpg)
2
SA interaction difficulties
• SA MAD formats, RMPP, libibumad “quirkiness”• Application shortcuts
– Hard coded PR data– Ignoring parts of queried PR data– Only work on a limited set of clusters or cluster types
• Direct access to libibumad and the SA are vectors for security breaches
March 30 – April 2, 2014 #OFADevWorkshop
![Page 3: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/3.jpg)
3
Current stack is not scalable
• Nodes access the same SA services multiple times from ibacm, kernel, libibumad…– PR queries– Notice/multicast registrations
• Name resolution through standard DNS requires an ARP from IP to GID
March 30 – April 2, 2014 #OFADevWorkshop
![Page 4: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/4.jpg)
4
ibacm name resolution
• Relies on IPoIB (DNS, ARP, etc)• Names map to <GID, Pkey> “end point”
– User’s often don’t care about the partition they are running on.
– “cross” partition names can’t be resolved• Local apps need knowledge of a common partition prior to
name resolution.
– Some work done in this area via ibacm_hosts.data
• Current name resolution requires source “end point” to be specified
March 30 – April 2, 2014 #OFADevWorkshop
![Page 5: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/5.jpg)
5
ibacm as a SA proxy
• ibacm provides a good starting point for addressing some of these concerns…
March 30 – April 2, 2014 #OFADevWorkshop
![Page 6: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/6.jpg)
6
Goals
• Provide controlled and consistent access to user space name and PR resolution services (AKA SA access)– SA access control– Accuracy– Ease of use– Portability– Enable all consumers to access ibssa
• Provide caching and other ibacm services to kernel users
March 30 – April 2, 2014 #OFADevWorkshop
![Page 7: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/7.jpg)
7
ibacm enhancements
• Applications query ibacm as a local “SA proxy”– All SA interactions done through ibacm– Additional name services provided– ibacm can control access to SA and libibumad
• ibacm is backed by “providers”– ibssa– Current features as default– Enhancements for name services are planned
March 30 – April 2, 2014 #OFADevWorkshop
![Page 8: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/8.jpg)
8
ibacm enhancements
• Name resolution services– “DNS” for direct name resolution
• Name to PR (or GID, <GID, PR>, IP, <IP, PR>)
• ibacm provides service to the kernel– Uses netlink– Leverages the same infrastructure for all users
March 30 – April 2, 2014 #OFADevWorkshop
![Page 9: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/9.jpg)
9
Arch
March 30 – April 2, 2014 #OFADevWorkshop
![Page 10: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/10.jpg)
10
Architecture (non-eye chart)
March 30 – April 2, 2014 #OFADevWorkshop
![Page 11: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/11.jpg)
11
Implementation plans
• Separate out ibacm into “core” and default provider
• Core handles– Provider loading and assignment to ports/End points– Steering client requests to correct provider– Port/device Events– Netlink requests and events– Administration like config file parsing, log file, etc
• Default provider handles– Same functionality as current resolve functions
March 30 – April 2, 2014 #OFADevWorkshop
![Page 12: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/12.jpg)
12
Initial data flow details
March 30 – April 2, 2014 #OFADevWorkshop
![Page 13: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/13.jpg)
13
Provider API’s
• Prototype code being worked– Collaborating with OFI WG and rdmacm– submission to the list imminent– “prov” branch in ibacm’s git tree
• The API will evolve, collaborating with ibssa• Main API calls will include
– Path Record resolution– Name to GID mapping helper
March 30 – April 2, 2014 #OFADevWorkshop
![Page 14: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/14.jpg)
14
Expand *_getaddrinfo
• Use ibacm first to resolve a Name prior to calling getaddrinfo (DNS)– Call can provide Path Record hints through the
normal “hints” parameter• For example Service ID, Pkey etc.
• Need both librdmacm and ibacm changes• Only single local end point can be supported
now– Future local end point resolution can be determined
by GID returned from provider name -> GID map
March 30 – April 2, 2014 #OFADevWorkshop
![Page 15: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/15.jpg)
15
Kernel ibacm access
• SRP, IPoIB, and rdma_cm kernel modules use ib_sa to query for Path Records
• Extend ibacm PR resolution/caching to kernel modules
• Use netlink messages to communicate with ibacm• Expand existing RDMA netlink interface• Currently connecting with ib_sa using ibacm
messages– Exploring the use of ib_mad and using MAD formatted
messages
March 30 – April 2, 2014 #OFADevWorkshop
![Page 16: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/16.jpg)
16
Overview
March 30 – April 2, 2014 #OFADevWorkshop
![Page 17: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/17.jpg)
17
A little more detail…
March 30 – April 2, 2014 #OFADevWorkshop
![Page 18: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/18.jpg)
18
Current Netlink / ibacm Message Format
March 30 – April 2, 2014 #OFADevWorkshop
VersionOpcodeStatus
Length
Length
Flags Type
Sequence
Pid
0123
Path Record (64 Bytes)
Type
Flags
Netlink Header
ACM Header
Byte Order
Path Data
Tid
Data[0]
Data[2] Data[1]
Reserved
(16 bytes)
(72 bytes)
(16 bytes)
![Page 19: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/19.jpg)
19
Kernel still uses SA when ibacm not available
March 30 – April 2, 2014 #OFADevWorkshop
![Page 20: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/20.jpg)
20
Future work
• SA event registration and reporting– Notice
• Multicast• IP to GID mapping
– IPoIB netlink to ibacm?– arpd extentions?
March 30 – April 2, 2014 #OFADevWorkshop
![Page 21: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/21.jpg)
21
Thank you
• Hal Rosenstock• Kaike Wan• Sean Hefty
March 30 – April 2, 2014 #OFADevWorkshop
![Page 22: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/22.jpg)
#OFADevWorkshop
Thank You
![Page 23: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/23.jpg)
23
Current SA interactions
• Applications– Direct SA
• Libibumad• UD QP
– Librdmacm• Ibacm• Dns/arp
• Kernel– Direct SA access only
March 30 – April 2, 2014 #OFADevWorkshop
![Page 24: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/24.jpg)
24
Name service requirements
• Generic interface to request remote node by name through “DNS like” resolution– Mapping provided by providers based on cluster
configuration, node configuration, and/or provider/SA communication.
March 30 – April 2, 2014 #OFADevWorkshop
![Page 25: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/25.jpg)
25
librdmacm example
• New librdmacm example app
$ resolve_name -h
usage: resolve_name <name>
[-h]
[-s <service id>] Specify a service ID in PR 'hints'
March 30 – April 2, 2014 #OFADevWorkshop
![Page 26: Scalable name and address resolution infrastructure -- Ira Weiny/John Fleck #OFADevWorkshop.](https://reader030.fdocuments.in/reader030/viewer/2022032607/56649ec05503460f94bcacd7/html5/thumbnails/26.jpg)
26
librdmacm example
$ resolve_name priv03
ai_family 0
ai_route : 0x1ff15a0
Path information
service_id: 0x0
dgid: fe80::11:7500:79:1815
sgid: fe80::11:7500:79:1763
dlid: 2
slid: 1
…
$ resolve_name google.com
ai_family 2
dest (null) 173.194.33.133
March 30 – April 2, 2014 #OFADevWorkshop