Cisco Application Centric Infrastructure with Splunk ... · Cisco Application Centric...
Transcript of Cisco Application Centric Infrastructure with Splunk ... · Cisco Application Centric...
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 57
Cisco Validated Design
Cisco Application Centric Infrastructure with Splunk
Enterprise Solution
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 57
Contents
Audience ................................................................................................................................................................... 4
Introduction .............................................................................................................................................................. 4
Features and Benefits of Cisco ACI for Splunk Enterprise .................................................................................. 4
Business Value ........................................................................................................................................................ 5
About This Cisco Validated Design........................................................................................................................ 6
Architecture Overview ............................................................................................................................................. 6 Cisco Application Centric Infrastructure ................................................................................................................ 6 Cisco Application Policy Infrastructure Controller ................................................................................................. 7 Cisco ACI Features ............................................................................................................................................... 8
Cisco APIC Appliance Features ..................................................................................................................... 11 Cisco Leaf Switch Connection Features ......................................................................................................... 11 Cisco Spine Switch Connection Features ...................................................................................................... 12
Splunk Enterprise ................................................................................................................................................ 12
Solution Overview.................................................................................................................................................. 13
Solution Details ...................................................................................................................................................... 13 Installing Splunk Enterprise 6.4.4 ........................................................................................................................ 14 Starting Splunk Web Server Setup ..................................................................................................................... 15 Installing Your Splunk License ............................................................................................................................ 16 Installing Cisco ACI App for Splunk Enterprise ................................................................................................... 17 Installing Cisco ACI Add-on for Splunk Enterprise .............................................................................................. 20
Cisco ACI App for Splunk Enterprise Operation ................................................................................................. 22 General Use ........................................................................................................................................................ 22
Navigation ...................................................................................................................................................... 22 Within a Dashboard ........................................................................................................................................ 22 Visualization Behaviour .................................................................................................................................. 22 Time Picker .................................................................................................................................................... 23 APIC Host ...................................................................................................................................................... 23 Additional Filters ............................................................................................................................................. 23
Home Dashboard ................................................................................................................................................ 23 APICs Table ................................................................................................................................................... 24 Fabric Health: History Chart ........................................................................................................................... 24 Home Dashboard Single-Value Visualizations ............................................................................................... 24
Help Desk Dashboards ....................................................................................................................................... 24 Help Desk: System Faults ................................................................................................................................... 24
Help Desk: System Faults Dashboard Single-Value Visualizations................................................................ 25 Faults by Node ............................................................................................................................................... 26 Faults by Tenant ............................................................................................................................................. 26 Faults by Severity ........................................................................................................................................... 27 Faults by Domain ........................................................................................................................................... 27 Faults by Severity over Time .......................................................................................................................... 27 Faults by Type ................................................................................................................................................ 27 Top Faults by Rule ......................................................................................................................................... 28 Top Faults by Cause ...................................................................................................................................... 28 Latest Affected Objects .................................................................................................................................. 28
Help Desk: Atomic Counters ............................................................................................................................... 29 Help Desk: Path Degradation .............................................................................................................................. 29 Help Desk: System Threshold ............................................................................................................................. 30 Fabric Dashboards .............................................................................................................................................. 31 Fabric: Fabric Details .......................................................................................................................................... 31
Top Affected Leafs ......................................................................................................................................... 31 Top Affected Spines ....................................................................................................................................... 32
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 57
Health/Fault Details: Leafs ............................................................................................................................. 32 Health/Fault Details: Spines ........................................................................................................................... 32 TCAM Percentage Threshold Statistics .......................................................................................................... 32 Top TCAM Usage by Node ............................................................................................................................ 32 Leafs – Port Utilization and Thresholds .......................................................................................................... 33 Spines – Port Utilization and Thresholds ........................................................................................................ 33 Change Threshold (for Leaf and Spine Utilization) ......................................................................................... 33
Fabric: Authentication ......................................................................................................................................... 33 Authentication Dashboard Single-Value Visualizations .................................................................................. 34 Authentication by Admin ................................................................................................................................. 35 Authentication Failed by User ........................................................................................................................ 35 Authentication Success by User ..................................................................................................................... 35
Fabric: Multi Pod ................................................................................................................................................. 35 APICs ............................................................................................................................................................. 36 Fabric Health – History ................................................................................................................................... 36 Leafs .............................................................................................................................................................. 36 Spines ............................................................................................................................................................ 37 Critical Faults.................................................................................................................................................. 37 EPGs .............................................................................................................................................................. 37
Tenants Dashboards ........................................................................................................................................... 39 Tenants: Tenant Details ...................................................................................................................................... 39
Top 10 Affected Tenants’ Health .................................................................................................................... 40 Top 10 Affected Tenants’ Faults .................................................................................................................... 40
Tenants: Tenant Utilization ................................................................................................................................. 41 <tenant>-Ingress and <tenant>-Egress Utilization Statistics in Bytes ............................................................ 41
Tenants: Microsegmentation ............................................................................................................................... 41 No. of EPGs Microsegmented per Tenant ...................................................................................................... 42 Network-Based Attributes ............................................................................................................................... 43 VM-Based Attributes ...................................................................................................................................... 43
VM Manager Dashboards ................................................................................................................................... 43 VM Manager: VMware ........................................................................................................................................ 43 Search ................................................................................................................................................................ 44 Setup Guide ........................................................................................................................................................ 44
Creating Custom Dashboards .............................................................................................................................. 45 Data Indexed by Cisco ACI App for Splunk Enterprise ....................................................................................... 45
cisco:apic:stats ............................................................................................................................................... 45 cisco:apic:class .............................................................................................................................................. 45 cisco:apic:health ............................................................................................................................................. 45 cisco:apic:authentication ................................................................................................................................ 45 apicsyslog ...................................................................................................................................................... 45
Building a Custom Dashboard ............................................................................................................................ 45 Custom Dashboard: Single-Value Visualization ............................................................................................. 45 Custom Dashboard: Column Chart Visualization ........................................................................................... 47 Custom Dashboard: Table Visualization ........................................................................................................ 48 Accessing Your Custom Dashboard ............................................................................................................... 50
Tuning the Cisco ACI App ..................................................................................................................................... 51
Conclusion ............................................................................................................................................................. 52
Appendix ................................................................................................................................................................ 53 Solution Design and Specifications ..................................................................................................................... 53
Cabinet Configuration ..................................................................................................................................... 55 Detailed Connection Diagram ........................................................................................................................ 56
Bill of Materials .................................................................................................................................................... 56
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 57
Audience
The intended audience for this document includes sales engineers, field consultants, professional services
developers, IT managers, partner engineers, and customers who want to combine the benefits of Splunk
Enterprise with the Cisco® Application Centric Infrastructure (Cisco ACI
™) solution.
Introduction
Managing and monitoring IT infrastructure is more complex and difficult than ever before. The rapid rate of change
and nearly endless streams of data create new challenges. Today, when problems arise, gaining visibility across
your entire infrastructure and finding the root cause quickly is almost impossible. Virtualized and cloud-based
infrastructures also add to the support and management challenges.
Cisco Application Centric Infrastructure and Splunk provide the solution. Splunk Enterprise is the market leader in
the collection and indexing of machine data from physical, virtual, and cloud-based environments. Splunk in
combination with the Cisco ACI solution gives you exceptional access to network and application insights. With
built-in dashboards that you can customize to see meaningful data at-a-glance and the capability to see a myriad
commonly used metrics and application details, the Cisco ACI App for Splunk Enterprise offers you a robust tool for
administering your entire Cisco ACI environment.
The Cisco Validated Design for Cisco ACI with Splunk Enterprise describes the deployment of Cisco ACI in a
single-pod environment and how to set upSplunk. It demonstrates how to install the Cisco ACI Add-on for Splunk
Enterprise and describes the main features and customization capabilities when running Cisco ACI.
Features and Benefits of Cisco ACI for Splunk Enterprise
Cisco ACI for Splunk Enterprise offers these main features and benefits:
● Reduced resolution time with accelerated root-cause analysis
◦ Centrally view the operational health of your entire Cisco ACI environment and underlying entities,
including Cisco Application Policy Infrastructure Controller (APIC) devices, fabric, tenants, and
applications.
◦ In multitenant environments, accelerate root-cause investigation and quickly navigate to the source of
application problems using flexible per-role visibility into Cisco ACI performance.
● Central proactive monitoring of Cisco ACI
◦ Get real-time proactive notification of any Cisco ACI faults including the location and affected objects,
physical components, logical and virtual components, fabrics, tenants, applications, virtual machines, leaf
nodes, and ports.
● Operation analytics
◦ Optimize your network capacity and prevent service deterioration with detailed visibility into fabric-path
degradation.
◦ Meet compliance and security requirements with user analytics, including authentication tracking reports.
◦ Correlate data from Cisco ACI with data from storage resources, operating systems, applications, and
virtual and physical infrastructure for visibility across your entire enterprise.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 57
● Cisco ACI health and user reports
◦ Gain visibility into Cisco ACI health and key performance indicators (KPIs) with dashboards that include:
◦ At-a-glance view of all APIC devices with their uptime, history of overall fabric health scores over five
days, summary of physical inventory including spine and leaf elements, and summary of logical and
virtual inventory including tenants, applications, and virtual machines.
◦ Help desk dashboard with context-specific faults grouped by acknowledgment status, time, severity, type,
rule, cause, and affected objects.
◦ Tenant dashboard with reports highlighting tenant health scores, affected tenants, and application and
endpoint group (EPG) health score details with visibility into the endpoint with which degradation
occurred.
◦ Innovative Cisco ACI fabric architecture that offers flexible multipath capabilities including network
telemetry with atomic counters to avoid network outages; view fabric path degradation with insight into
actual packet loss across any path, without the need to deploy network sniffers to understand the optimal
fabric trajectory.
◦ Authentication tracking with eight prebuilt reports, including reports of successful and failed logins, active
and inactive users, and user audit and event logs.
For more information, see http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-
centric-infrastructure/solution-overview-c22-731967.html.
Business Value
Cisco ACI with Splunk Enterprise offers exceptional business value:
● Unified and centralized visibility across your IT infrastructure
◦ Cisco ACI with Splunk Enterprise allows far-reaching visibility across your IT infrastructure. With the
capability to unify machine data from physical and virtual servers, storage, and application environments
as well as throughout the underlying Cisco ACI fabric and extended network, customers can see their
entire system with a “big picture” view previously unavailable.
◦ Related dashboards: All.
● Holistic health
◦ Environmental health information is central to Cisco ACI functions. Cisco ACI tracks, monitors, and trends
the operational health of all components that run through and comprise the fabric. The health of tenants,
applications, fabric hardware, and endpoints (both virtual and physical) is interwoven throughout the
Cisco ACI for Splunk Enterprise solution.
◦ Related dashboards: Home, Fabric Details, Multi Pod, and Tenant Details.
● Expedited resolution and root-cause analysis
◦ Quickly identifying faults and determining the root cause is always a challenge. With information from
Cisco ACI and from storage resources, operating systems, applications, security devices, and endpoints
correlated and then visualized through Splunk dashboards, you gain new insight. Problems previously
difficult to identify can now be understood instantly, down to the fault-level component, application,
policy, interface, etc. Deployment of this solution can reduce the mean time needed to investigate and
resolve problems by up to 70percent1.
1 See https://blogs.cisco.com/datacenter/aci-for-splunk-enterprise-enabling-comprehensive-application-health
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 57
◦ Related dashboards: System Faults, Atomic Counters, Path Degradation, Tenant Utilization, and System
Threshold.
● Compliance
◦ Establishing an effective compliance and ethics program is now a necessity in nearly all organizations.
The Cisco ACI App for Splunk Enterprise provides readily available compliance and security information
with user analytics, including authentication and Cisco ACI environmental audit reporting capabilities.
◦ Related dashboard: Authentication.
● Real virtual insight
◦ With the deep integration between Cisco ACI and VMware and visualization with Splunk, understanding
your virtualized environment has never been easier. Every element, from the originating VMware vCenter
application, host, virtual machine name, connected interface, associated EPG, etc, contributes to a
meaningful view of your virtualized environment.
◦ Related dashboard: VMware.
● Actionable security information
◦ Today you must be ready to respond when—not if—a security breach occurs. Natively, Cisco ACI
supports microsegmentation, which allows organizations to reduce the potential for lateral movement in
the event of a security breach. Now, with literally two mouse clicks, all your microsegmented details can
be viewed in one place. Problems can be identified and acted on in minutes, not hours.
◦ Related dashboards: Microsegmentation and System Faults.
About This Cisco Validated Design
The Cisco ACI App for Splunk Enterprise solution has been validated using single-pod and multipod Cisco ACI
deployments. The remainder of this document details the deployment of Cisco ACI in a single-pod environment
with Splunk Enterprise.
Architecture Overview
This section provides an overview of the Cisco ACI and Splunk Enterprise architectures.
Cisco Application Centric Infrastructure
Cisco ACI is an innovative architecture that radically simplifies, optimizes, and accelerates the entire application
deployment lifecycle. It uses a holistic systems-based approach, with tight integration between physical and virtual
elements, an open ecosystem model, and innovation-spanning application-specific integrated circuits (ASICs),
hardware, and software. This unique approach uses a common policy-based operating model across a network
that supports Cisco ACI along with security elements (and computing and storage in the future), eliminating IT silos
and drastically reducing cost and complexity.
The main benefits of Cisco ACI include:
● Simplified automation with an application-based policy model
● Common platform for managing physical, virtual, and cloud-based environments
● Centralized visibility with real-time application health monitoring
● Operation simplicity, with common policy, management, and operation models across application, network,
and security resources (and computing and storage resources in the future)
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 57
● Open software flexibility for DevOps teams and for ecosystem partner integration
● Scalable performance and secure multitenancy
Cisco ACI consists of (Figure 1):
● Cisco Application Policy Infrastructure Controller, or APIC
● Cisco Nexus® 9000 Series Switches (Cisco ACI spine and leaf switches)
● Cisco ACI ecosystem
Figure 1. Cisco ACI Architecture
Cisco Application Policy Infrastructure Controller
The infrastructure controller is the main architectural component of the Cisco ACI solution. It is the unified point of
automation and management for the Cisco ACI fabric, policy enforcement, and health monitoring. The APIC
appliance is a centralized, clustered controller that optimizes performance and unifies the operation of the physical
and virtual environments. The controller manages and operates a scalable multitenant Cisco ACI fabric.
The main features of the controller include:
● Application-centric network policies
● Data-model-based declarative provisioning
● Application and topology monitoring and troubleshooting
● Third-party integration (Layer 4 through Layer 7 [L4-L7]) services and VMware vCenter and vShield)
● Image management (spine and leaf)
● Cisco ACI inventory and configuration
● Implementation on a distributed framework across a cluster of appliances
● Health scores for critical managed objects (tenants, application profiles, switches, etc.)
● Fault, event, and performance management
● Cisco Application Virtual Switch (AVS), which can be used as a virtual leaf switch
The controller framework enables broad ecosystem and industry interoperability with Cisco ACI. It enables
interoperability between a Cisco ACI environment and management, orchestration, virtualization, and L4-L7
services from a broad range of vendors.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 57
Cisco ACI Features
The Cisco ACI mode fabric software is an optimized version of the Cisco NX-OS Software operating system that
provides a foundation for building a programmable network infrastructure. NX-OS has been rewritten as a fully
object-based switch operating system for Cisco ACI. The object model enables fluid programmability and full
access to the underlying components of the infrastructure using representational state transfer (REST) APIs. This
approach provides a framework for network control and programmability with a degree of openness that is not
found in other systems.
The infrastructure controller provides centralized access to Cisco ACI through an object-oriented REST API
framework with XML and JavaScript Object Notation (JSON) binding. It also supports a modernized, user-
extensible command-line interface (CLI) and GUI. APIs have full read and write access to Cisco ACI, providing
tenant- and application-aware programmability, automation, and system access.
Table 1 summarizes some of the Cisco ACI main features. For more information about additional features or the
availability of these features by release, please refer to:
● Cisco ACI data sheet: http://www.cisco.com/c/en/us/products/collateral/cloud-systems-
management/application-policy-infrastructure-controller-apic/datasheet-c78-732414.html
● Release notes for Cisco ACI and APIC: http://www.cisco.com/c/en/us/support/cloud-systems-
management/application-policy-infrastructure-controller-apic/tsd-products-support-general-information.html
● Release notes for Cisco Nexus 9000 SeriesSwitches:
http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-release-notes-list.html
Table 1. Cisco ACI Main Features
Feature Description
Integrated overlay over nonblocking 40/100 Gigabit Ethernet IP fabric
● Pv4 unicast and IPv4 multicast at line rate
● Penalty-free application and tenant mobility
● Full host mobility
Cisco ACI multipod solution ● Multipod solution allows 1 APIC cluster to manage multiple Cisco ACI fabrics, in which each fabric is a pod. The multipod can consist of different floors or buildings within a campus or a local metropolitan region. Each pod is a localized fault domain
Cisco ACI fabric extension, WAN connectivity, Border Gateway Protocol (BGP) Ethernet Virtual Private Network (EVPN) and external connectivity
● Cisco ACI fabric as a transit domain: The fabric enables border routers to perform bidirectional route distribution with other routing domains, including route peering with service appliances
● WAN connectivity automation: Cisco ACI fabric and Cisco ASR 9000 Series Aggregation Services Routers and Cisco Nexus 7000 Series Switches data center interconnect (DCI) connectivity is automatically discovered and provisioned based on the BGP-EVPN control plane and Virtual Extensible LAN (VXLAN) overlay dataplane for IPv4/IPv6
● Routing protocols
◦ IPv6 data plane provides support for tenant addressing, contracts, shared services, and routing
◦ Open Shortest Path First (OSPF), Enhanced Interior Gateway Routing Protocol (EIGRP), external BGP (eBGP), internal BGP (iBGP), shared tenant Common Layer 3 outside (L3Out) interface, route leaking from tenant Virtual Routing and Forwarding (VRF) instances, and static routes are supported
● Virtual port channel (vPC): Straight-through mode to end hosts and servers is used
Systemwide application visibility and troubleshooting
● Cisco Switched Port Analyzer (SPAN) and Encapsulated Remote SPAN (ERSPAN) support
● Atomic counters
● Application and tenant health scores
Application network profiles ● Logical representation of all components of the application and its interdependencies on the application fabric
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 57
Feature Description
Policy ● Fabricwide policy enforcement regardless of endpoint location
● Policy enforcement between EPGs
Cisco ACI availability ● 3 APIC node clusters
● APIC cluster software rolling upgrade and downgrade
● Less than 1 second for fabric convergence after node or link failure detection (with spine redundancy and vPC)
● Hot-swappable field-replaceable units (FRUs; except Gigabit Ethernet module [GEM]) for top-of-rack (ToR) per-port VLAN
● Configuration of the same VLAN ID across different EPGs (in different bridge domains) on different ports on the same leaf switch
● Stretched fabric with 10-ms round-trip time (RTT) with Multiprotocol Label Switching (MPLS) pseudowire, dark fiber, and dense wavelength-division multiplexing (DWDM)
Security ● Permit, deny, and taboo list (blacklist), and application-centric whitelist policy model for securing both physical and virtual applications
● EPG policy filtering (source EPG, destination EPG, and Layer 4 ports)
● Microsegmentation (virtual machine attribute–based segmentation) and distributed firewall with the AVS
● Microsegmentation (virtual machine attribute–based segmentation) with Microsoft Hyper-V and System Center Virtual Machine Manager (SCVMM)
● Secure multitenancy at scale built into Cisco ACI fabric
● Built-in distributed Layer 4 security integrated into Cisco ACI fabric to secure east-west traffic
● Role-based access control (RBAC), authenticated access based on certificate authentication, Cisco Secure Access Control System (ACS), and local authentication
● Authentication, authorization, and accounting(AAA)and RBAC integration
● Auditing of all user access and changes
Centralized fabric management ● Automatic fabric discovery
● Single pane across network, hypervisors, and L4-L7 services
● Intuitive GUI, extensible CLI, and REST APIs
● NX-OS style of CLI on the APIC and access to all switches through the controller
Management upgrades, versioning, and scaling ● Switch and APIC upgrades across the fabric
● Support for multiple software versions for leaf and spine switches per APIC domain
● Touchless ToR addition to fabric (zero-touch plug and play)
Troubleshooting GUI ● Troubleshooting wizard
● Capacity dashboard
● Heat map
Secure user authentication ● TACACS+, RADIUS, and Lightweight Directory Access Protocol (LDAP)
● Local authentication with password and RBAC
Monitoring ● Virtual network interface cards (vNICs; VMware only)
◦ Received and transmitted ingress and egress packets
◦ Broadcast, multicast, and dropped packets
● NX-OS and APIC processes and system
◦ Per leaf, spine, and APIC
◦ CPU utilization per process and overall
◦ Memory utilization per process and overall
● Protocol statistics (available on iShell)
◦ Intermediate System–to–Intermediate System (IS-IS) Protocol and iBGP global statistics
◦ Per logical interface and per adjacency for protocol statistics
● Service insertion
◦ Packets and bytes
◦ VLAN and bridge domain statistics
● Cisco ACI contract support for a new action called copy service, which allows traffic flows to be copied between 2 EPGs or through L4-L7 devices and sent to 1 or N destinations simultaneously
● Health scores
◦ 0 to 100 with ±1 granularity
◦ Historical records of health scores
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 57
Feature Description
◦ AVS health status, events, and faults reported to APIC
● Fabric
◦ Spine, leaf, fabric extender (host interfaces [HIFs] and network interfaces [NIFs]), and vPC
◦ Ingress and egress counters
◦ Unicast, multicast, flood, and drop
● EPG (VLAN and VXLAN): aggregated
◦ Ingress only, unicast, and multicast
◦ Flood, VXLAN-only drop (bytes), and egress only for VLAN encapsulated traffic
◦ Per-ingress EPG
◦ Per flow only (drill-down only)
◦ Endpoints (vNIC only and VMware only): drill-down and on demand
L4-L7 services integration ● L4-L7 service policy automation (scripting interface) and data-path integration
● Service chaining; forwarding based (no policy redirection)
● Policy-based redirect allows redirection of traffic based on a classifier match in a service graph
● Symmetric policy-based routing
● Service policy automation through REST API with JSON and XML
● Automated service node insertion and provisioning
● Health score for service and clustering degradation (through scripting interface)
● Support for transparent and routed firewall modes (traditional mode)
For more information, view the latest Cisco ACI L4-L7 compatibility list solution overview.
Virtualization integration ● VMware ESXi, vSphere, and vShield
● VMware vSphere Distributed Switch (VDS) support with automated port-group creation for VLAN and VXLAN mapped to EPG
● VMware vMotion for multiple VMware vCenters
● VMware vMotion movement between the fabric-connected hosts
● VMware vRealize support for AVS workflows such as virtual machine manager (VMM) domain creation and distributed firewall policy
● VMware vCenter Plug-inuser interface that integrates with the vSphere web client to manage and troubleshoot the Cisco ACI fabric, allowing the vSphere web client to become a single management pane for configuring both vCenter and the Cisco ACI fabric
● AVS for Cisco ACI fabric (VMware)
For more information, view the latest Cisco ACI virtualization compatibility list solution overview.
Figure 2 shows the Cisco ACI hardware components.
Figure 2. Cisco ACI Hardware Components
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 57
Cisco APIC Appliance Features
The APIC appliance has two form factors: for medium and for large configurations. Medium configurations have a
medium-size CPU and hard drive and memory for up to 1000 edge ports. Large configurations have a large-size
CPU and hard drive and memory for more than 1000 edge ports. The reference architecture discussed in this
document deploys a medium-size appliance.
The APIC appliance uses a purpose-built Cisco UCS®C220 M4 Rack Server manufactured with an image secured
with a Trusted Platform Module (TPM), certificates, and an APIC product ID. To order the appliance clusters and
additional Cisco ACI components, refer to the bill of materials (BOM) at the end of this document.
Figure 3 shows the APIC connection features.
Figure 3. Connection Features on a Second-Generation APIC Appliance
Cisco Leaf Switch Connection Features
This section identifies the connection features that you use when connecting the Cisco Nexus 9396PX Switch to
the Cisco ACI fabric as a leaf switch (Figure 4).
Figure 4. Connection Features on a Cisco Nexus 9396PX ACI Leaf Switch
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 57
Cisco Spine Switch Connection Features
Figure 5 identifies the connection features that you use when connecting the Cisco Nexus 9336PQSwitch to the
Cisco ACI fabric as a spine switch.
Figure 5. Connection Features on a Cisco Nexus 9336PQ ACI Spine Switch
Splunk Enterprise
Splunk Enterprise provides a holistic way of organizing and extracting real-time insights from massive amounts of
machine data, making it an excellent tool to pair with Cisco ACI. Because Cisco ACI has a single store of
information (the APIC) and that data is indexed through Splunk, you can visualize the entire fabric as well as other
parts of the IT infrastructure. Figure 6 shows the Splunk architecture.
Figure 6. Splunk Architecture
The Splunk server software is written in C/C++ and Python and is provided in an all-in-one distribution. Although
Splunk has several roles that can be configured (search head, indexer, forward, etc.), the design discussed here
deploys all these roles in a single virtual machine. After Splunk is installed, two service processes will be running
on your Linux system: splunkd and splunkweb.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 57
● splunkd is a distributed C/C++ server that accesses, processes, and indexes streaming IT data and also
handles search requests. The splunkd processes and indexes your data by streaming it through a series of
pipelines, each made up of a series of processors. Pipelines are single threads inside the splunkd process,
each configured with a single snippet of XML. Processors are individual, reusable C/C++ or Python
functions that act on the stream of IT data passing through a pipeline. Pipelines can pass data to one
another through queues. splunkd supports a CLI for searching and viewing results.
● splunkweb is a Python-based application server that provides the Splunk web user interface. It allows users
to search and navigate IT data stored by Splunk servers and to manage the Splunk deployment through the
browser interface. splunkweb communicates with your web browser through REST and communicates with
splunkd through Simple Object Access Protocol (SOAP).
Solution Overview
The integrated solution of Splunk and Cisco ACI with the APIC at its core provides exceptional visibility and
reduced time to troubleshoot through the use of comprehensive dashboards and unified views across all your IT
infrastructure (Figure 7). Key health, performance, user, policy, tenant, and configuration data are all available in a
centralized and easy-to-consume way using Splunk visualization features. For additional information, refer to the
Cisco ACI and Splunk solutions brief at http://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/solution-overview-c22-731967.html.
Figure 7. Cisco ACI with Splunk Integrated Solution
Solution Details
The Cisco ACI environment and Splunk Enterprise should be deployed in accordance with the reference
architecture information included at the end of this document. For detailed information about implementation of
your Cisco ACI environment and for configuration and programming guides, consult the following link:
http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-
apic/tsd-products-support-series-home.html?
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 57
Installing Splunk Enterprise 6.4.4
Note: Although Splunk can be run on a virtual machine managed by a Cisco ACI VMM, for the deployment
described here, the Splunk server was installed on a standalone virtual machine with connectivity outside the
Cisco ACI fabric path to the APIC devices. Whether your Splunk server is deployed on bare-metal servers or in
a virtualized environment, the only requirement for this server is that it must have network connectivity to the
Cisco ACI APIC devices in order to pull information from them. No specific Cisco ACI configuration is necessary
to support the Splunk server as deployed in this reference architecture.
Splunk Enterprise software runs on several supported platforms, including Microsoft Windows and several varieties
of Unix and Linux. This document describes the installation steps for a deployment using 64-bit Ubuntu Linux 4.4.0-
31-generic.
1. Navigate to the preferred download location on your Linux server. Enter the following command to download
the Splunk installation file (Figure 8):
wget -O splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz
https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.4
.4&product=splunk&filename=splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz&wget=true
Figure 8. Download Splunk Enterprise 6.4.4
2. Enter the following command to unpack and install Splunk:
tar xvzf splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz -C /opt
Note: To enter commands to unpack, install, start, stop, or restart Splunk, you may need to use a higher
privilege level. If you encounter an error with these actions, precede the command with sudo and then enter
the root user password if prompted.
3. Export the variable for the splunk directory:
export SPLUNKHOME=/opt/splunk
Note: This reference architecture uses the /opt directory to install Splunk. If you installed Splunk in a
different directory, be sure to replace /opt with the path for your installation directory.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 57
4. Navigate to the /$SPLUNKHOME/bin directory:
cd /$SPLUNKHOME/bin
5. Start Splunk and accept the user license (Figure 9):
sudo ./splunk start - accept-license
Figure 9. Accept Splunk License
Starting Splunk Web Server Setup
When you start Splunk, a web service will run. To access this service, navigate in a web browser to
http://your_server_name:8000 (Figure 10).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 57
Figure 10. Splunk Enterprise Home Screen
Installing Your Splunk License
Install your Splunk license as shown in Figures 11a, 11b, and 11c.
Figure 11a. Adding Splunk License
Figure 11b. Adding Splunk License
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 57
Figure 11c. Adding Splunk License
Installing Cisco ACI App for Splunk Enterprise
Follow these steps to install the Cisco ACI App for Splunk Enterprise:
1. Download the Cisco ACI App for Splunk Enterprise from https://splunkbase.splunk.com/app/1896/ (Figure 12).
Figure 12. Splunkbase: Cisco ACI App for Splunk Enterprise
2. Download the Cisco ACI Add-on for Splunk Enterprise from https://splunkbase.splunk.com/app/1897/
(Figure 13).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 57
Figure 13. Splunkbase: Cisco ACI Add-on for Splunk Enterprise
3. Accept the license agreements and agree to download (Figure 14).
Figure 14. Accept License Agreements
4. Copy the files to the Splunk server (Figures 15 and 16).
Figure 15. File Copy from Personal Computer
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 57
Figure 16. File Copy to Linux Server
5. Install the Cisco ACI App for Splunk Enterprise with the following command:
sudo tar xvzf cisco-aci-app-for-splunk-enterprise_22.tgz –C /$SPLUNKHOME/etc/apps/
6. Restart Splunk:
cd /$SPLUNKHOME/bin
sudo ./splunk restart
7. Verify the installation by navigating to http://your_server_name:8000 (Figure 17).
Figure 17. Splunk Home Screen with Cisco ACI App for Splunk Enterprise
8. Update the application by navigating to http://your_server_name:8000/en-us/_bump and clicking “Bump
version” (Figure 18).
Figure 18. Updating the Bump Version
9. Restart Splunk:
cd /$SPLUNKHOME/bin
sudo ./splunk restart
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 57
Installing Cisco ACI Add-on for Splunk Enterprise
Follow these steps to install the Cisco ACI Add-on for Spunk Enterprise:
1. Install the Cisco ACI Add-on for Splunk Enterprise:
sudo tar xvzf cisco-aci-add-on-for-splunk-enterprise_22.tgz-C /$SPLUNKHOME/etc/apps/
2. Restart Splunk:
cd /$SPLUNKHOME/bin
sudo ./splunk restart
3. From the Splunk home screen, click the gear icon next to Apps (Figure 19).
Figure 19. App Settings
4. On the line for Cisco ACI Add-on for Splunk Enterprise, click “Set up” (Figure 20).
Figure 20. App Configuration
5. Provide the credentials for your APIC (Figure 21).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 57
Figure 21. Cisco APIC Credentials
6. Go to Settings (Figure 22) and under Data click “Data inputs” (Figure 23).
Figure 22. Splunk Settings
Figure 23. Data Inputs
7. In the App column, enable all scripts associated with TA_cisco-ACI (Figure 24).
Figure 24. Scripts
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 57
Note: If you are not using SSL certificates to access your Cisco ACI instance, an additional configuration
change is required. To disable SSL connections to Cisco ACI from the Splunk application, from the Splunk
server navigate to the folder as shown here and update the config.ini file:
cd /$SPLUNKHOME/splunk/etc/apps/TA_cisco-ACI/bin
Change the configuration from ENABLE_SSL = True to ENABLE_SSL = False.
8. Restart Splunk:
cd /$SPLUNKHOME/splunk/bin
sudo ./splunk restart
9. Allow up to 15 minutes to populate the data.
Cisco ACI App for Splunk Enterprise Operation
To launch the application, from the main Splunk screen after login click Cisco ACI App for Splunk Enterprise
(Figure 25).
Figure 25. Launch Cisco ACI App for Splunk Enterprise
General Use
This section describes features for the general operation of the Cisco ACI App for Splunk Enterprise.
Navigation
Application dashboards are accessible by navigating across the green ribbon. The dashboard categories are
Home, Help Desk, Fabric, Tenants, VM Manager, Search, and Setup Guide.
Within a Dashboard
There are several dashboards with readings, metrics, and other useful visualizations related to your Cisco ACI
environment. Typically, you can interact with these items to drill down into details, or to further expand information
you want to see.
Visualization Behaviour
Visualization options include the following:
● Bar graph, column graph, and pie chart visualizations: When you interact with bar graphs, column graphs,
or pie charts, an in-page drill-down feature will appear below the bar graph, column graph, or pie chart.
● Table visualizations: Table visualizations are a final level of drill-down feature. If you want to see additional
information, click the magnifying glass icon while hovering over the visualization to bring up the Splunk
search that was used to produce the table.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 57
● Single-value visualizations: When you click a single-value visualization, a new tab with an expanded
dashboard or table related to the single-value visualization is displayed.
● Timeline graph visualizations: No further drill-down interactions are available when you interact with timeline
graphs.
● All visualization behavior: Each visualization has a hover bar below it that contains links as described in
Figure 26.
Figure 26. Splunk App Hover Bar
Time Picker
Just as in a standard search in Splunk, many of the dashboards contain a time picker to help narrow the range
related to information in the dashboard.
APIC Host
The APIC host picker appears on each screen. If you have connected more than one APIC fabric, you can use this
drop-down menu to filter by the specific fabric for which you want to view details.
Additional Filters
Certain dashboards have additional filters such as health score, severity, user, source node, destination node, pod
name, tenants, applications, EPGs, VMware ESXi hosts, and virtual machines (VMs).
Home Dashboard
The Home dashboard is your starting reference with a high-level overall view of your Cisco ACI fabric (Figure 27).
Figure 27. Splunk App Home Dashboard
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 57
APICs Table
The APICs table provides information related to the hardware components and base-level configuration (such as IP
address) that make up your APIC cluster.
Fabric Health: History Chart
Fabric health over time is depicted as a line graph. Because the data is indexed in Splunk, users can access a
longer history than is available in the APIC advanced GUI.
Home Dashboard Single-Value Visualizations
Table 2 lists each single-value visualization and the corresponding dashboard to which it relates. Each dashboard
defined in this table is discussed in more detail later in this document.
Table 2. Visualization-to-Dashboard Mapping
Visualization Dashboard
Tenants Tenant Details
Applications Application Details
VMs VMware
Leafs Fabric Details
Spines Fabric Details
Critical Faults Help Desk
EPGs EPG Details
Bridge Domains Bridge Domain Details
Filters Filters Details
Contracts Contracts Details
L3OUT Networks L3OUT Networks
Help Desk Dashboards
The Help Desk dashboards consist of System Faults, Atomic Counters, Path Degradation, and System Threshold
(Figure 28).
Figure 28. Splunk App Helpdesk Dashboards
Help Desk: System Faults
The Help Desk: System Faults dashboard details APIC system faults visualized in several ways (Figure 29).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 57
Figure 29. Splunk App System Faults Dashboard
Help Desk: System Faults Dashboard Single-Value Visualizations
New-tab tables are associated with each single-value visualization in the Help Desk dashboard single-value
visualizations.
Faults
Faults is a total count of faults, both Acknowledged and Unacknowledged (Figure 30).
Figure 30. Splunk App System Fault Details
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 57
Acknowledged Faults
Acknowledged Faults is a subset of faults that contains only faults that have been acknowledged (Figure 31).
Figure 31. Splunk App System Fault Detail: Acknowledged Faults
Unacknowledged Faults
Similar to Acknowledged Faults, Unacknowledged Faults is a subset of faults that contains only faults that have not
been acknowledged (Figure 32).
Figure 32. Splunk App System Fault Detail: Unacknowledged Faults
Faults by Node
Faults by Node is a pie chart depicting system faults by fabric node. Interacting with a slice will open a detail table
below the pie chart containing all instances of faults for that particular fabric node (Figure 33).
Figure 33. Splunk App Faults by Node Detail
Faults by Tenant
Faults by Tenant is a pie chart depicting system faults by tenant. Interacting with a slice will open a detail table
below the pie chart containing all instances of faults for that particular tenant (Figure 34).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 57
Figure 34. Splunk App Faults by Tenant Detail
Faults by Severity
Faults by Severity is a pie chart depicting system faults by level of severity. Interacting with a slice will open a detail
table below the pie chart containing all instances of faults with that particular severity level (Figure 35).
Figure 35. Splunk App Faults by Severity Detail
Faults by Domain
Faults by Domain is a pie chart depicting system faults by ACI domain. Interacting with a slice will open a detail
table below the pie chart containing all instances of faults with that particular domain (Figure 36).
Figure 36. Splunk App Faults by Domain Detail
Faults by Severity over Time
Faults by Severity over Time is a timeline graph depicting system faults by severity over time.
Faults by Type
Faults by Type is a bar graph depicting system faults by the type of fault. Interacting with a bar in the graph will
open a detail table below the bar graph containing all instances of faults of that particular type (Figure 37).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 57
Figure 37. Splunk App Faults by Type Detail
Top Faults by Rule
Top Faults by Rule is a pie chart depicting system faults sliced by a rule. Interacting with a slice will open a detail
table below the pie chart containing all instances of faults with that particular rule (Figure 38).
Figure 38. Splunk App Faults by Rule Detail
Top Faults by Cause
Top Faults by Cause is a pie chart depicting system faults sliced by cause. Interacting with a slice will open a detail
table below the pie chart containing all instances of faults with that particular cause (Figure 39).
Figure 39. Splunk App Faults by Cause Detail
Latest Affected Objects
Latest Affected Objects is a table displaying the fabric objects most recently affected (Figure 40).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 57
Figure 40. Splunk App Latest Affected Objects
Help Desk: Atomic Counters
The Atomic Counters dashboard (Figure 41) contains two table elements that display information when you use
Cisco ACI to troubleshoot with atomic counters: Endpoint to Endpoint (EP to EP) and Endpoint Group to Endpoint
Group (EPG to EPG). If you have not used atomic counters to troubleshoot EP to EP or EPG to EPG, no results
will be displayed.
Figure 41. Splunk App Atomic Counters Dashboard
Help Desk: Path Degradation
The Path Degradation dashboard (Figure 42) contains a table that displays information when you use Cisco ACI to
troubleshoot intrafabric traffic using atomic counters. If you have not used atomic counters to troubleshoot
intrafabric traffic, no results will be displayed.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 57
Figure 42. Splunk App Path Degradation Dashboard
Help Desk: System Threshold
The System Threshold dashboard provides easy-to-view user-definable fabric thresholds. Among them are Tenant,
EPG, Contracts, Filters, Bridge Domains, and L3OUT Networks, all depicted as easy-to-read gauges (Figure 43).
All these visualizations have an in-window Change Threshold link that opens a new tab and allows you to make
changes to the thresholds set.
Figure 43. Splunk App System Threshold Dashboard
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 57
Fabric Dashboards
The Fabric menu on the green navigation bar consists of three dashboards accessible from the drop-down menu.
These dashboards are Fabric Details, Authentication, and Multi Pod (Figure 44).
Figure 44. Splunk App Fabric Dashboards
Fabric: Fabric Details
The Fabric Details dashboard displays health statistics for various nodes in your Cisco ACI fabric (Figure 45).
Figure 45. Splunk App Fabric Details Dashboard
Top Affected Leafs
Top Affected Leafs visualizes health scores in a colored column graph for each leaf node in your Cisco ACI fabric.
Interacting with a column in the graph will open seven tables below the graph containing hardware, health,
utilization, and fault details related to that particular leaf node (Figure 46).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 57
Figure 46. Splunk App Leaf Hardware, Health, and Utilization Visualizations
Top Affected Spines
In the same way as Top Affected Leafs, Top Affected Spines visualizes node health as a colored column graph for
each spine in your Cisco ACI fabric. The same seven tables will appear below the column graph when you interact
with a specific column in the Top Affected Spines visualization (Figure 47).
Figure 47. Splunk App Spine Hardware, Health, and Utilization Visualizations
Health/Fault Details: Leafs
Health/Fault Details: Leafs is a table listing health and fault information for leaf switches over a period of time
specified in the time picker.
Health/Fault Details: Spines
Health/Fault Details: Spines, just like the table for leaf switches, visualizes health and fault information over a
specified period of time.
TCAM Percentage Threshold Statistics
TCAM Percentage Threshold Statistics is a simple table showing current settings for Warning Threshold, Critical
Threshold, and Max Threshold percentages.
Top TCAM Usage by Node
Top TCAM Usage by Node is a statistics table showing colored bars in a graph for each fabric node (Figure 48).
The Change Threshold link in the Top TCAM Usage by Node window will open a new tab and allow you to adjust
the TCAM percentage threshold values. Interacting with a bar on the chart will open two additional tables beneath
the TCAM Percentage Threshold Statistics bar chart.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 57
Figure 48. Splunk App Top TCAM Usage by Node
Leafs – Port Utilization and Thresholds
The Leafs – Port Utilization and Thresholds table presents summarized egress and ingress information along with
threshold levels for each leaf switch (Figure 49).
Figure 49. Splunk App Summarized Leaf Port Utilization
Spines – Port Utilization and Thresholds
The Spines – Port Utilization and Thresholds table presents summarized egress and ingress information along with
threshold levels for each spine switch (Figure 50).
Figure 50. Splunk App Summarized Spine Port Utilization
Change Threshold (for Leaf and Spine Utilization)
The Change Threshold link opens a new tab on which you can change values for Warning and Critical thresholds
related to port utilization on Cisco ACI fabric leaf and spine switches (Figure 51).
Figure 51. Splunk App Change Link Utilization Threshold Tab
Fabric: Authentication
The Authentication dashboard displays information about users, authentication attempts, and audit information
(Figure 52).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 57
Figure 52. Splunk App Authentication Dashboard
Authentication Dashboard Single-Value Visualizations
New-tab tables are associated with each single-value visualization on the Authentication dashboard:
● All Users (Figure 53)
Figure 53. Splunk App All Users Table
● Local Users (Figure 54)
Figure 54. Splunk App Local Users Table
● Remote Users (Figure 55)
Figure 55. Splunk App Remote Users Table
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 35 of 57
Authentication by Admin
Authentication by Admin is a pie chart depicting successful authentications by the admin user by IP address.
Clicking the chart will open a table below the main visualizations window with historical data related to the pie slice
selected (Figure 56).
Figure 56. Splunk App Authentication by Admin Table
Authentication Failed by User
Authentication Failed by User is a column chart depicting failed authentications by user. Clicking an individual
column will open a table below the main visualizations window with historical data related to that specific user
(Figure 57).
Figure 57. Splunk App Failed Authentication by User Table
Authentication Success by User
Authentication Success by User is a column chart depicting successful authentications by user. Clicking an
individual column will open a table below the main visualizations window with historical data related to that specific
user (Figure 58).
Figure 58. Splunk App Successful Login by User Table
Fabric: Multi Pod
Multi Pod setup and configuration are outside the scope of this document. However, a customer who deploys the
Cisco ACI App for Splunk Enterprise will have access to the Multi Pod dashboard (Figure 59). The Multi Pod
dashboard provides an overall view of each pod in a multipod environment. In addition to the time picker filter,
users can filter by health score and pod name.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 36 of 57
Figure 59. Splunk App Multi Pod Dashboard
APICs
The APICs table has important details related to your APIC cluster, such as name, management IP address, and
pod membership.
Fabric Health – History
Fabric Health – History depicts the history of the fabric health for each pod of your multipod deployment as a health
trend over time.
Leafs
Leafs provides a count of total leaf switches categorized by pod and represented by a column graph (Figure 60).
When you interact with a column on the graph, an additional visualization will open below the column chart with
specific health information for each individual leaf switch.
Figure 60. Splunk App Affected Leafs Visualization
Affected Leafs of pod-#
You can drill down further by interacting with a specific leaf switch in the column chart. Doing so will open six tables
with hardware-specific information for that leaf as shown in Figure 61.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 37 of 57
Figure 61. Splunk App Affected Leaf Hardware Tables
Spines
Spines displays a count of total spine switches categorized by pod and represented by a column graph (Figure 62).
When you interact with the column on the graph, an additional visualization will open below the column chart with
specific health information for each individual spine switch.
Figure 62. Splunk App Affected Spines Visualization
Affected Spines of pod-#
You can drill down further by interacting with a specific spine switch in the column chart. Doing so will open six
tables with hardware-specific information for that spine switch as shown in Figure 63.
Figure 63. Splunk App Affected Spine Hardware Tables
Critical Faults
Critical Faults is a pie chart depicting pods in your multipod environment. When you select a slice, a new
visualization appears below the Critical Faults pie chart.
Time Chart: Critical Fault (30-day period) for pod-x
The Critical Fault chart depicts critical faults over a 30-day period for the selected pod.
EPGs
EPGs are represented as a pie chart of the pods of your multipod environment. Interacting with a slice will open
two new visualizations below the EPGs pie chart.
EPGs with Static Ports for pod-x
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 38 of 57
EPGs with Static Ports for pod-x displays, by tenant, a count of EPGs with port assignments (Figure 64).
Interacting with a particular column will open two additional tables below the column graph with static port
information and EPG health for the selected tenant.
Figure 64. Splunk App EPGs with Static Ports Visualization
EPG Static Port Details for Tenant: tenant
EPG Static Port Details for Tenant: tenant displays information about the port and EPG assignments for the
selected tenant (Figure 65).
Figure 65. Splunk App EPG Static Port Details for Tenant Table
EPG Health Details for Tenant: tenant
EPG Health Details for Tenant: tenant displays information about EPG health for the selected Tenant. (Figure 66).
Figure 66. Splunk App EPG Health Details for Tenant Table
EPGs Unassigned to Any Pod
If EPGs are created but are not assigned to ports in your Cisco ACI fabric, they will be depicted in this column
graph (Figure 67). Interacting with columns among the tenants listed in the column graph will open a table below it.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 39 of 57
Figure 67. Splunk App EPG Unassigned to Any Pod Table
EPG Health Details for Tenant: tenant
EPG Health Details for Tenant: tenant displays information about EPG health for the selected Tenant. This
information is displayed when selecting a tenant from among the columns of tenants in the EPGs Unassigned to
any Pod column graph (Figure 68).
Figure 68. Splunk App EPG Health Details for Tenant Table
Tenants Dashboards
The Tenants menu on the green navigation bar consists of three dashboards accessible from the drop-down menu.
These dashboards are Tenant Details, Tenant Utilization, and Micro segmentation (Figure 69).
Figure 69. Splunk App Tenant Dashboards
Tenants: Tenant Details
The Tenant Details dashboard displays basic health details by tenant (Figure 70).
Figure 70. Splunk App Tenant Details Dashboard
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 40 of 57
Top 10 Affected Tenants’ Health
Top 10 Affected Tenants’ Health is a bar chart that shows colored health scores by tenant. Interacting with a bar in
the visualization will open additional on-screen panels beneath the bar chart with details related to the selected
tenant.
Application Health for Tenant: tenant
The Application Health for Tenant: tenant table shows health scores by application for the selected tenant
(Figure 71).
Figure 71. Splunk App Application Health for Tenant Table
End Point Group Health for Tenant: tenant
The End Point Group for Tenant: tenant table shows health scores by EPG and related applications for the
selected tenant (Figure 72).
Figure 72. Splunk App End Point Group Health for Tenant Table
Application Statistics
The Application Statistics table shows utilization statistics for each application of the selected tenant (Figure 73).
Figure 73. Splunk App Application Statistics Table
Client End Point Details
The Client End Point Details table lists endpoint information for the selected tenant (Figure 74).
Figure 74. Splunk App Client End Point Details Table
Top 10 Affected Tenants’ Faults
Top 10 Affected Tenants’ Faults is a pie chart depicting fault count by tenant. Interacting with a particular slice will
open a table below the pie chart with additional information.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 41 of 57
<tenant> Tenant Fault Details
The Tenant Fault Details table shows related faults for the tenant selected (Figure 75).
Figure 75. Splunk App Tenant Fault Details Table
Tenants: Tenant Utilization
The Tenant Utilization dashboard displays packet information categorized by tenant (Figure 76). Interacting with
either the Ingress or Egress Utilization column charts will open two tables beneath the column charts with
additional information.
Figure 76. Splunk App Tenant Utilization Dashboard
<tenant>-Ingress and <tenant>-Egress Utilization Statistics in Bytes
The <tenant>-Ingress Utilization Statistics in Bytes and <tenant>-Egress Utilization Statistics in Bytes tables
display port and ingress and egress statistics for the selected tenant (Figure 77).
Figure 77. Splunk App Ingress and Egress Utilization Tables
Tenants: Microsegmentation
The Microsegmentation dashboard displays information about microsegmented endpoints by tenant (Figure 78).
Microsegmentation uses two primary filtering mechanisms: network-based and virtual machine–based attribute
filtering.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 42 of 57
Figure 78. Microsegmentation Dashboard
No. of EPGs Microsegmented per Tenant
No. of EPGs Microsegmented per Tenant is a column chart listing each tenant that contains one or more
microsegmented EPG and a count of them. Interacting with a column in the chart opens three additional tables to
the right and below the column chart.
Health Details of Microsegmented EPGs for Tenant: tenant
The Health Details table shows health details for microsegmented EPGs of the selected tenant (Figure 79).
Figure 79. Health Details of Microsegmented EPGs for Tenant: tenant Table
Microsegmented Domains (VMs and Bare-Metal)
The Microsegmented Domains table shows Cisco ACI domain and associated details for microsegmented EPGs of
the selected tenant (Figure 80).
Figure 80. Microsegmented Domains (VMs and Bare-Metal) Table
Client Endpoints
The Client Endpoints table shows endpoint details associated with the microsegmented EPGs of the selected
tenant (Figure 81).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 43 of 57
Figure 81. Client Endpoints Table
Network-Based Attributes
Network-Based Attributes is a table with specific information related to the value of a particular network attribute
and the specific filter used to microsegment an endpoint based on the particular network attribute (Figure 82).
Figure 82. Network-Based Attributes Table
VM-Based Attributes
VM-Based Attributes is a table with specific information related to the value of a particular virtual machine attribute
and the specific filter used to microsegment an endpoint based on the particular virtual machine attribute
(Figure 83).
Figure 83. VM-Based Attributes Table
VM Manager Dashboards
The VM Manager dashboards contain information related to virtualized endpoints (Figure 84). At this time, only
VMware is supported, but future versions of the application will support other virtualized tools.
Figure 84. Splunk App Virtualization Dashboards
VM Manager: VMware
The VMware dashboard contains important endpoint details related to your VMware virtualized environment
(Figure 85). Comprehensive filtering of this information is possible using the time picker drop-down menu or
filtering by tenant, application, EPG, ESX host, or virtual machine. This table contains no additional drill-down
capabilities.
Note: The VMware dashboard provides additional panels that become visible when the Splunk App for
VMware is installed and configured. The installation of the Splunk App for VMware is beyond the scope of this
document.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 44 of 57
Figure 85. Splunk App VMware Dashboard
Search
The Search window is similar to the main Splunk Search application, but it applies specifically to your Cisco ACI
fabric and machine data gathered from the Cisco ACI App for Splunk Enterprise (Figure 86).
Figure 86. Splunk App Search Tab
Setup Guide
Setup Guide is a guide to the setup and configuration contained in this document and is provided for easy future
reference (Figure 87).
Figure 87. Splunk App Setup Guide Tab
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 45 of 57
Creating Custom Dashboards
Splunk provides a native capability to create custom dashboards with visualizations based on searches of indexed
data. This section discusses the source types containing information about your Cisco ACI environment indexed
through the Cisco ACI App for Splunk Enterprise and describes the process for creating a custom dashboard.
Data Indexed by Cisco ACI App for Splunk Enterprise
One primary index is created when you use the Cisco ACI App for Splunk Enterprise. This index is referred to as
the apic index. This index contains five source types, which are discussed in detail here (Figure 88).
Figure 88. The Apic Index Source Types
cisco:apic:stats
The cisco:apic:stats source type contains information related to historical total and average aggregated statistics
for ingress and egress packets in a specified fabric.
cisco:apic:class
The cisco:apic:class source type contains the majority of configuration data (excluding health information) about
managed objects in the specified fabric.
cisco:apic:health
The cisco:apic:health source type contains historical health information for the managed objects of the specified
fabric.
cisco:apic:authentication
The cisco:apic:authentication source type contains user-authentication data.
apicsyslog
The apicsyslog source type contains syslog data.
Building a Custom Dashboard
Splunk offers many ways to visualize data searched from an index. This document discusses the setup for three
primary visualizations, explains the search used to build the visualizations, and describes how to create or add the
visualizations to your custom dashboard.
Custom Dashboard: Single-Value Visualization
You will get a distinct count of the number of microsegmented EPGs to use for this visualization.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 46 of 57
1. Click Search on the main navigation bar.
2. Search the apic index (index=apic) to find EPGs (component=fvEPG) that are attribute based
(isAttrBasedEPg=yes), which indicates that the EPG is microsegmented. Then pipe ( | ) the results to the
statistics command (stats) requesting a distinct count based on the name (dc(name)) of the EPG with the
following search string:
index=apic component=fvEPG isAttrBasedEPg=yes | stats dc(name)
3. Click the Visualization tab in the Search window and verify that the visualization type is set to Single Value
(Figure 89).
Figure 89. Single Value Visualization Setting
4. In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard
Panel.
5. Configure the Save As Dashboard Panel as shown in Figure 90. Then click Save.
Figure 90. Save As Dashboard Panel 1
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 47 of 57
6. In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your custom dashboard
should look similar to Figure 91.
Figure 91. My Custom Dashboard 1
Custom Dashboard: Column Chart Visualization
For this visualization, you will display errors by severity level categorized by tenant.
1. Click Search on the main navigation bar.
2. Perform the search as follows:
a. Search the apic index (index=apic).
b. Filter the source type by apic health (sourcetype=cisco:apic:health).
c. Filter by the specific apic cluster, referencing a node of that cluster by IP address
(apic_host=10.23.248.116).
d. Include all tenants (component=fvTenant) and all events that contain “warning,” “minor,” or “major”
((warning OR minor OR major)).
e. Pipe ( | ) the data to the chart command showing a count of each type of error for each tenant and
categorized by severity (chart count over name by severity).
Here is the complete search:
index=apic sourcetype=cisco:apic:health apic_host=10.23.248.116 component=fvTenant (warning OR minor OR
major) | chart count over name by severity
3. Click the Visualization tab in the Search window and verify that the visualization type is set to Column Chart
(Figure 92).
Figure 92. Column Chart Visualization Setting
4. In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard
Panel.
5. Configure the Save As Dashboard Panel as shown in Figure 93. Then click Save.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 48 of 57
Figure 93. Save As Dashboard Panel 2
6. In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your custom dashboard
should now look similar to Figure 94.
Figure 94. My Custom Dashboard 2
Custom Dashboard: Table Visualization
For the final visualization, you will represent the virtualization information for your VMware environment in a table.
1. Click Search on the main navigation bar.
2. This search is a little more complex:
a. Enter a pipe ( | ) character to indicate that what follows is a macro.
Note: Macros are predefined scripts that make complicated and repetitive searches easier to implement.
Macro creation is outside the scope of this document. You can find a list of predefined macros at Settings >
Advanced Search > Search Macros.
b. Enter the name of the macro enclosed in a single quotation mark (`) character: for example,
`end_point_detail`.
c. Pass the results of the macro to a pipe ( | ) followed by the search command and each of the limiters to
search (search apic_host=10.23.248.116 Tenant=* Application=* EPG=* VirtualMachine=* ESX-
Host=*).
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 49 of 57
d. Pass these search results further down the pipeline to the table command to list the table headers related
to the data you want displayed (| table Tenant, Application, EPG, EPG-Health, VirtualMachine, state,
Network-Adapter, ESX-Host, vCenter, Interface).
e. For the final pipeline connection, use the rename command to change some of the header names to make
them more user friendly (| rename VirtualMachine AS "VirtualMachine" ESX-Host AS "ESX host"
Network-Adapter AS "Network Adapter" EPG-Health AS "EPG Health" state AS "State").
Here is the complete search:
| `end_point_detail` | search apic_host=10.23.248.116 Tenant=* Application=* EPG=* VirtualMachine=* ESX-
Host=* | table Tenant, Application, EPG, EPG-Health, Virtual Machine, state, Network-Adapter, ESX-Host, vCenter,
Interface | rename VirtualMachine AS "Virtual Machine" ESX-Host AS "ESX host" Network-Adapter AS "Network
Adapter" EPG-Health AS "EPG Health" state AS "State"
3. On the Statistics tab, view the table resulting from the search (Figure 95).
Figure 95. Statistics Table
4. In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard
Panel.
5. Configure the Save As Dashboard Panel as shown in Figure 96. Then click Save.
Figure 96. Save As Dashboard Panel 3
6. In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your completed custom
dashboard should now look similar to Figure 97.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 50 of 57
Figure 97. My Custom Dashboard 3
Accessing Your Custom Dashboard
You can access your newly created custom dashboard by searching for it in the Find field or by assigning it as a
home dashboard.
Find Field Method
The Find field is accessible on the far right of the black ribbon in the Splunk web interface (Figure 98). Typing the
name of your custom dashboard and selecting it will display it.
Figure 98. Search for Custom Dashboard
Home Dashboard Assignment Method
To assign your newly created dashboard as a home dashboard, follow these steps:
1. Click the Splunk > link in the upper-left corner of the webpage.
2. On the Splunk start page (Figure 99), click anywhere in the box that says “Choose a home dashboard.”
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 51 of 57
Figure 99. Splunk Start Screen
3. In the Choose Default Dashboard dialog box, select your dashboard from the drop-down list and click Save
(Figure 100).
Figure 100. Choose Default Dashboard Dialog Box
Your custom dashboard is now accessible from the Splunk start page.
Tuning the Cisco ACI App
As installed, the Cisco ACI App for Splunk Enterprise requires no additional modifications. However, depending on
your Splunk license consumption, you may want to make modifications to better align your use with your Splunk
license.
The Splunk scripts used to enable the application specify data polling based at a predefined interval (represented
in seconds). Increasing this interval (to a higher number) will result in a longer polling cycle, less frequent indexing,
slightly less-current data, and lower Splunk license consumption. Decreasing the interval (to a lower number) will
do the opposite, resulting a shorter polling cycle, more frequent indexing, more-current data, and greater
consumption of your Splunk license.
You should adjust these timers only if you need to reconcile your Splunk license or to acquire a view of your data
that is closer to a real-time view.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 52 of 57
Conclusion
Cisco ACI allows you to automate provisioning of network and application services, provide a multitenant
environment with whitelist networking, and deploy a highly secure and policy-based microsegmented endpoint
environment, while integrating physical and virtual endpoints and achieving outstanding scalability.
Splunk, the world leader in making sense of your machine data, enhances Cisco ACI further by providing
organized dashboards on which you can easily view your entire system, troubleshoot, rapidly assess root causes,
and monitor system health, in real time or historically, for all your Cisco ACI physical, software, application,
virtualized, and connected components.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 53 of 57
Appendix
Solution Design and Specifications
Table 3 summarizes the specifications for the Cisco ACI and Splunk Enterprise reference design.
Table 3. Cisco ACI, Splunk Enterprise, and Cisco ACI App for Splunk Enterprise Reference Architecture
Cisco APIC Appliance Quantity: 3
Type APIC-M2
Cisco Integrated Management Controller C220M3.2.03i
Firmware version 2.0(3i)
CPU details
Number of CPUs 2
Clock speed (MHz) 2100
Number of cores per CPU 6
Type Intel® Xeon
® processor E5-2620 v2 CPU at 2.10 GHz
Memory configuration
Total memory 64 GB
Memory modules 4 x 16-GB DDR3 at 1866 MHz
Memory configuration Independent
Installation arrangement A1, B1, E1, and F1
Power supply details
Type 650 watts (W)
PCI adapters
Intel® I350 1-Gbps Network Controller
Firmware version 0x80000AA4-1.808.2
Slot L
Cisco UCS VIC 1225 10-Gbps 2-port converged network adapter SFP+
Firmware version 4.1(1d)
Slot 1
Cisco UCS C RAID SAS 2008M-8i
Firmware version 20.13.1-0249
Slot M
Physical drive 1
Size 113961 MB
RAID configuration 0
Virtual drive number 1
Physical drive 2 In RAID group with physical drive 3
Size 475883 MB
RAID configuration 1
Virtual drive number 0
Physical drive 3 In RAID group with physical drive 2
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 54 of 57
Cisco APIC Appliance Quantity: 3
Size 475883 MB
RAID configuration 1
Virtual drive number 0
Cisco ACI Leaf Switch Quantity: 2
Type Cisco Nexus 9396PX
BIOS version 07.41
Kickstart image 12.0(2f)
Software version 2.0(2f)
Hardware
CPU type Intel Core i3 CPU at 2.50 GHz
Memory 16 GB
Bootflash memory 64 GB
Cisco ACI Spine Switch Quantity: 2
Type Cisco Nexus 9336PQ
BIOS version 07.41
Kickstart image 12.0(2f)
Software version 2.0(2f)
Hardware
CPU type Intel Core i3 CPU at 2.50 GHz
Memory 16 GB
Bootflash memory 64 GB
Splunk Index Server Quantity: 1
Machine detail VMware virtual machine
CPU allocation 12 CPU cores
Server memory allocation 12 GB
Disk drive allocation 100 GB
Operating system Ubuntu Linux 64-bit 4.4.0-31-generic
Splunk Enterprise Software Quantity: 1
Software version 6.4.4
Splunk license 20 GB or more per day
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 55 of 57
Cabinet Configuration
Figure 101 shows the Cisco ACI physical infrastructure and connections.
Figure 101. Cisco ACI Fabric Physical Infrastructure and Connection Matrix
Note: Splunk can be installed either within a Cisco ACI fabric network or on a fabric network other than Cisco
ACI. Likewise, Splunk can run on a bare-metal server or in a host-based virtualized environment. The three servers
listed in Figure 101 are shown strictly to illustrate a sample physical environment and connection layout.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 56 of 57
Detailed Connection Diagram
Figure 102 shows Cisco ACI fabric connectivity.
Figure 102. Cisco ACI Fabric Connectivity Diagram
Bill of Materials
Tables 4 through 7 provide the ordering information for the single-pod Cisco ACI environment with Splunk
Enterprise.
Table 4. Cisco ACI APIC Appliance Bill of Materials
Part Number Description Quantity
APIC-M2 Medium configuration (up to 1000 edge ports) 3
CON-SSSNP-APICM2 SOLN SUPP 24X7X4 APIC appliance, medium configuration 3
APIC-PSU1-770W 770W power supply for Cisco UCS C-Series 3
APIC-PCIE-CSC-02 Cisco UCS VIC 1225 dual-port 10-Gbps SFP+ CNA 3
1000BASE-T 1-Gbps copper Ethernet cable (2m) 9
Table 5. Cisco ACI Spine Switch Bill of Materials
Part Number Description Quantity
N9K-C9336PQ Cisco Nexus 9000 Series ACI spine switch, 36 ports, 40-Gbps QSFP+ 2
CON-3SNTP-9336PQ 3YR SNTC 24X7X4, Cisco Nexus 9336 ACI Spine Switch with 36 ports 2
QSFP-H40G-AOC1M= 40GBASE active optical cable, 1m 4
1000BASE-T 1-Gbps copper Ethernet cable (2m) 2
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 57 of 57
Table 6. Cisco ACI Leaf Switch Bill of Materials
Part Number Description Quantity
N9K-C9396PX Cisco Nexus 9300 platform 48-port 1/10-Gbps SFP+ and additional uplink module required
2
CON-3SNTP-9396PX 3YR SNTC 24X7X4 Cisco Nexus 9300 platform with 48 ports 2
N9K-M12PQ Cisco ACI capable uplink module for Cisco Nexus 9300 platform 12-port 400Gbps QSFP
2
N93-LIC-BUN-P1 Cisco Nexus 9300 platform LAN and Cisco ACI Software License Bundle PAK 2
SFP-10G-AOC3M= 10GBASE active optical SFP+ cable, 3m 6
SFP-10G-AOC1M= 10GBASE active optical SFP+ cable, 1m 2
1000BASE-T 1-Gbps copper Ethernet cable (2m) 2
Table 7. Splunk Enterprise Software and Support
Part Number Description Quantity
Splunk Enterprise Splunk Enterprise Software 6.4.4 1
Service support 3 years 1
Printed in USA C11-738275-00 01/17