MULTIPLE SPLUNK AS A SERVICE (MSAAS)
-
Upload
nguyenmien -
Category
Documents
-
view
217 -
download
1
Transcript of MULTIPLE SPLUNK AS A SERVICE (MSAAS)
WHITE PAPER
MULTIPLE SPLUNK AS A SERVICE (MSAAS)An Architecture Framework for Multitenant Splunk Deployments
WHITE PAPER
2Multiple Splunk as a Service (MSaaS)
Table of ContentsIntroduction ................................................................................................................................................................................................................... 3
Architecture ................................................................................................................................................................................................................... 4
MSaaS Architecture Features and Benefits............................................................................................................................................... 4
Service-Oriented Approach to Deployment ............................................................................................................................................. 5
Architectural Concepts ..................................................................................................................................................................................... 6
Splunk Enterprise Architectural Concepts ......................................................................................................................................... 6
MSaaS Logical Architecture .................................................................................................................................................................................... 8
Search Tier ............................................................................................................................................................................................................ 9
Indexing Tier ......................................................................................................................................................................................................... 11
Data Collection Tier ........................................................................................................................................................................................... 12
Namespace Best Practices .............................................................................................................................................................................. 13
MSaaS Roles ......................................................................................................................................................................................................... 13
Summary: Architectural Components ......................................................................................................................................................... 14
MSaaS Physical Architecture .................................................................................................................................................................................. 15
Search Tier ............................................................................................................................................................................................................ 15
Indexing Tier ......................................................................................................................................................................................................... 16
Data Collection Tier ........................................................................................................................................................................................... 16
Administration .............................................................................................................................................................................................................. 17
Deployment .......................................................................................................................................................................................................... 17
Management and Monitoring .................................................................................................................................................................................. 19
Licensing and Chargeback .............................................................................................................................................................................. 20
Credentials ............................................................................................................................................................................................................ 20
Example Implementation ......................................................................................................................................................................................... 20
Architecture .......................................................................................................................................................................................................... 21
Authentication and Authorization ......................................................................................................................................................... 23
Licensing ......................................................................................................................................................................................................... 23
Monitoring and Management .................................................................................................................................................................. 23
Automated Provisioning ........................................................................................................................................................................................... 24
Post-implementation Conclusions ........................................................................................................................................................................ 24
Key Observations ................................................................................................................................................................................................ 25
Additional Resources ................................................................................................................................................................................................. 25
Summary ......................................................................................................................................................................................................................... 26
WHITE PAPER
3Multiple Splunk as a Service (MSaaS)
IntroductionToday data is growing at an ever-accelerating rate,
and it will continue to do so into the foreseeable
future. The use of Splunk Enterprise to monitor and
analyze large volumes of data continues to grow as
well, driving increased demand for Splunk Enterprise.
In this time of rapid growth, administrators are
increasingly called upon to provision new Splunk
Enterprise instances for their internal or external
customers. Service providers often provision separate
instances of Splunk Enterprise for each of their many
customers, and enterprises commonly set up separate
instances for each business unit that requires Splunk
Enterprise within their organization.
Organizations with large-scale, multitenant Splunk
Enterprise deployments need to provide data
segregation and access control for individual tenants
to meet regulatory requirements or internal security
policies. In addition, they need a scalable solution
that can successfully handle the volume of data and
the growing number of instances under management.
These organizations strive to speed deployment and
manage both deployment and upgrade risk, all while
controlling administrative costs. They need a cost-
efficient approach that reduces the marginal cost of
each additional Splunk Enterprise instance and helps
optimize their total cost of ownership of the platform.
Multiple Splunk as a Service (MSaaS) is an
architectural framework that proposes a multi-
instance approach to supporting multiple internal or
external customers. Although multiple customers can
be supported on a single Splunk Enterprise instance,
the multi-instance approach is inherently more
scalable and provides essential data segregation
capabilities in a multi-tenant environment. The Splunk
Enterprise licensing model, together with indexing
volume tracking for each individual instance, can
be used as a basis for flexible chargeback plans for
individual clients. Leveraging this multi-instance
approach can provide an economical and scalable
solution for managing multitenant deployments.
The MSaaS approach proposes an automated,
on-demand request process to provision Splunk
Enterprise instances as needed for individual clients,
or to tailor deployments for operational reasons
within an organization. Investing in the design and
implementation of an MSaaS architecture benefits
ROI when deploying new Splunk Enterprise instances.
The automated deployment process can be very time
efficient—taking minutes or hours to deploy, integrate,
and test a new Splunk Enterprise instance.
Deployment is packaged and modular, and it is
consistent across all instances. This consistency
reduces the risk of introducing errors and simplifies
managing the deployed instances. Integration with
a version control system (VCS) provides reliable
tracking and control of the deployed configuration
files, reduces the risks associated with periodic
configuration changes, and enables rollback to a
known stable state when carrying out changes.
Using a configuration management system (CMS)
provides ease of deployment and scalability for
large implementations and promotes consistency
and reliability. The MSaaS architecture is highly
flexible and supports custom configuration for each
deployed Splunk Enterprise instance.
This document describes a conceptual framework
for designing an MSaaS architecture that supports
multitenant Splunk Enterprise deployments as
a service. The sections that follow describe the
MSaaS architecture, outline typical administration
and deployment tasks, and describe an example
implementation of this architecture. This document,
intended as a starting point in designing and
implementing an MSaaS deployment, provides
a conceptual framework rather than a complete
implementation plan, followed by a description of
an example customer implementation.
WHITE PAPER
4Multiple Splunk as a Service (MSaaS)
ArchitectureThe MSaaS architectural framework defines a
conceptual approach to deploying Splunk Enterprise
instances as a service. When implemented, the
framework described in this document enables
the automated, centralized deployment of Splunk
Enterprise instances through a service-request
model. In the MSaaS architecture, multiple islands
are created, with each island containing a complete
traditional Splunk Enterprise deployment for an
individual tenant (a customer or business unit).
The underlying capabilities of each Splunk Enterprise
deployment are unchanged. By implementing MSaaS
with the architectural framework described in this
document, organizations can design a service-
oriented implementation that can quickly and easily
deploy new Splunk Enterprise instances. Each
instance retains the capabilities and distributed
deployment options of a normal Splunk Enterprise
instance, and each instance can be sized to meet
the performance requirements for the use cases
it supports.
MSaaS Architecture Features and Benefits
The MSaaS architectural framework is designed
to guide the implementation of a service-oriented
approach for automatically deploying Splunk
Enterprise instances for customers. Each customer—
for example, a distinct enterprise business unit—
requires a Splunk Enterprise deployment, and an
island is created for each deployment. A single island
can service multiple customers if their respective
segregation requirements permit; there is no need to
deploy them as multiple tenants.
Key features of the MSaaS architecture include
the following:
• Service-request model. When a new Splunk
instance is needed, customers submit a service
request that describes the use cases and attributes
of the new instance. For example, attributes might
include the replication factor, search factor,
recovery point objective (RPO) and recovery
time objective (RTO) for disaster recovery,
backup requirements, retention plan, and daily
indexing volume.
• Automated deployment of each Splunk Enterprise instance. Predefined scripts developed
using a configuration management system are
used to automatically deploy new Splunk
Enterprise instances.
• Highly scalable deployments. The MSaaS
architecture improves Splunk Enterprise’s inherent
scalability by partitioning large deployments—
those that support multiple organizations,
span multiple data centers or multiple geographic
locations, and index very large volumes—into
islands with low marginal overhead per-island.
The number of islands that can be created is not
limited, and each island can be scaled in the usual
manner by which Splunk Enterprise scales
through dynamically provisioning multiple
clustered or individual indexers and search heads
to handle the required workload.
• Multitenancy. The MSaaS architecture supports
multitenancy by deploying a set of Splunk
Enterprise components running on distinct
computing resources for each tenant, ensuring
complete functionality along with full segregation
between tenants. The architecture can also
support a unified view of multiple islands up
to the entire environment, and it delivers the
ability to track resource usage for each tenant so
that utilization-based chargeback models can
be implemented.
• Segregation of data, flexible jurisdiction. The
MSaaS architecture maintains separation of data
for Splunk Enterprise instances. Data routing
rules can be implemented to control the flow
of data to specific indexers, and searches can be
performed over specific data sets. Flexible
jurisdiction capabilities enable hierarchical or
WHITE PAPER
5Multiple Splunk as a Service (MSaaS)
overlapping groupings of islands and the data they
contain, and role-based access control (RBAC) can
be used to implement jurisdiction, restricting
access to data, as needed.
• Flexible and customizable instances. The MSaaS architecture imposes no fixed
requirements on the underlying Splunk Enterprise
instances that make up each island. Instances
can scale from very small to as large as necessary,
and they can be customized to fulfill a wide
range of use-case requirements.
Implementing automated, centralized deployment
of Splunk Enterprise instances provides significant
advantages for organizations that need to administer
a number of Splunk instances. This service-oriented
approach helps organizations:
• Improve time to value. Once the MSaaS approach
is implemented, administrators can simply submit
a service request specifying the characteristics of
the new instance, and the new Splunk Enterprise
instance can be automatically installed and running.
• Reduce the marginal effort to deploy each instance. Once MSaaS is implemented, all that is
required to deploy a new Splunk Enterprise
instance is its characteristics. The marginal effort
to deploy each new instance is minimal.
• Manage deployment and upgrade risk. The MSaaS
service-based approach reduces the risk of human
error and makes the deployment process
consistent for all Splunk Enterprise instances.
Common scripts are used for configuration, and
a version control system (VCS) is used to maintain
copies of these scripts, configuration files, Splunk
Apps, and any other required artifacts, providing
an audit trail and a consistent deployment process.
Upgrades can also be easily deployed to all
Splunk Enterprise instances from a centralized
server, ensuring the deployment remains
consistent across the data center.
Service-Oriented Approach to Deployment
With a service-oriented approach, deployment of new
Splunk Enterprise instances is streamlined: Once the
MSaaS framework is in place, administrators provision
and deploy a new island through a service request.
A first step in designing the MSaaS architecture is
analysis of the attributes that are required for each
Splunk Enterprise instance. For example, typical
attributes can include:
• Data indexing daily volume
• Licensing and chargeback requirements
• Storage volume
• Retention policies
• Resilience requirements
• App provisioning
• Disaster recovery requirements
• Data segregation and jurisdiction requirements
Data segregation and jurisdiction requirements drive
the data collection configuration for each island. A
service request for a new island includes a list of data
sources (also known as end points), an estimate of
the daily data volume they generate, and a method
by which data can be identified as required or not
required by the island in question. Each end point can
send data to any island, and each collected event can
be sent to multiple islands or ignored.
After the set of island attributes are defined, a
mechanism for specifying a request should be
provided. For example, an online form can be used
to help define each attribute. Predefined attribute
sets that define common instance types can facilitate
the users’ selection of these instance types.
As a final step in the MSaaS design, scripts are
created to map the specification of the new instance
to the automated deployment process. Configuration
management tools are used to automate the task of
creating the Splunk Enterprise components that are
required for the new instance. VCS tools are used to
maintain control of scripts, configuration files, add-
ons, and apps that are used for the instances.
WHITE PAPER
6
When the MSaaS architecture is implemented, it can
service requests. Each service request contains details
on the attributes for the new instance. Once the
request is submitted, the automated process creates
and deploys the Splunk Enterprise components
required for that service.
Architectural Concepts
The MSaaS framework uses normal Splunk Enterprise
architectural concepts: Each island in the MSaaS
architecture contains a complete, traditional
Splunk Enterprise deployment. Building on top of
the traditional architecture, the MSaaS framework
adds the new concepts of islands, bridges, routing,
jurisdictions, and the MSaaS deployment server.
The following sections first provide a brief overview
of the Splunk Enterprise architecture and then
introduce the MSaaS architectural concepts.
For a more thorough description of the Splunk
Enterprise architecture, refer to the Splunk Enterprise documentation.
Splunk Enterprise Architectural Concepts At a high level, each Splunk Enterprise deployment
includes a data collection tier, an indexing tier, and a
search tier. The data collection tier consumes data.
The indexing tier parses and indexes the incoming
data and services search requests from the search
tier. And the search tier requests interactive or
scheduled searches from the indexing tier on the
indexed data, and presents, consolidates, or stores
their results.
For small deployments, a single computer can handle
the resource requirements of all three tiers. However,
more typically the functionality is split across multiple
specialized Splunk Enterprise components running
on separate computing platforms (see Figure 1).
Each component handles one or more roles, such
as indexing or search. Additional components can be
added to each tier to scale it horizontally to meet
the performance requirements. For more information,
see the Splunk Enterprise Distributed
Deployment Manual.
Figure 1. Example Splunk Enterprise architecture with search tier, indexing tier and data collection tier.
Search Tier
Indexing Tier
Firewall Router
Data Collection Tier
Search Head Cluster
Indexer
Forwarder
Indexer Indexer Indexer
Forwarder Forwarder
WHITE PAPER
7Multiple Splunk as a Service (MSaaS)
The three tiers contain the following components:
• Data Collection Tier: Forwarder. The forwarder
component consumes data and forwards it
to the indexing tier either directly or through
intermediate forwarders. For configurations
with multiple indexers, the data streams from
the forwarders can be load balanced across
several indexers. Load balancing helps
performance, scalability and indexing-tier
resilience. If one or more of the indexers become
unavailable, the forwarder can send its data to
another indexer.
• Indexing Tier: Indexer. Each indexer component
or its aggregate (an indexer cluster) indexes
incoming data and performs searches across
its indexes.
Figure 2. Example Splunk Enterprise architecture using clustering to provide increased scalability and resilience.
• Search Tier: Search Head. Each search head
component or its aggregate (a search head
cluster) distributes the search queries to
its indexing tier, consolidates and finalizes the
search results, and presents them to the user.
Environments of all sizes can deploy clustering for
increased resilience and availability (see Figure
2). Larger environments commonly include a
global license master and a deployment server for
administrative convenience.
DeploymentServer
Search Head Cluster
ClusterMembers
Peer Nodes(Search Peers)
Indexer Cluster
Each cluster membercommunicates with the peer nodes.
Each forwarder communicateswith the peer nodes.
Forwarders
Master ClusterNode
Captain
LicenseServer
WHITE PAPER
8Multiple Splunk as a Service (MSaaS)
Figure 3. An island contains a single end-to-end Splunk Enterprise deployment.
Main components include the search head cluster,
indexer cluster, deployment server and license master:
• Search Head Cluster. A search head cluster is a
group of three or more search heads that share
the search-tier workload. Members of the search
head cluster share configurations, job scheduling
and search artifacts. One cluster member,
designated the captain, coordinates all cluster
wider activities. If the member serving as captain
goes down, another member automatically takes
its place.
• Indexer Cluster. Indexer clusters are groups of
Splunk Enterprise indexers that keep multiple
copies of replicated data. Index replication
prevents data loss and promotes data availability.
Splunk Enterprise indexer clusters feature
automatic failover. If an indexer fails, the surviving
indexers continue to perform indexing and
searches across all of the data. The work of
the indexer cluster is managed by its cluster
master node.
• Deployment Server. The deployment server
centrally manages configuration files and apps
across the Splunk Enterprise deployment.
• License Master. The license master tracks indexed
data volume and enforces Splunk Enterprise’s
licensing model.
MSaaS Logical Architecture
An island is a logical construct that defines a single
end-to-end Splunk Enterprise deployment (see
Figure 3). Each island contains one or more indexers
and zero or more search heads. Optionally, each
island can also contain a license master, deployment
server, a cluster master node, and zero or more job
servers (non-interactive search heads dedicated to
running scheduled jobs). Each island also controls the
configuration of a set of forwarders. Each forwarder
can send data to any number of islands. However,
there is a single island that controls each forwarder’s
configuration through its deployment server.
An island’s search heads and indexers can be
clustered. Islands can also include passive standby
instances of the license master, deployment server
and cluster master node for resilience.
Islands maintain data segregation. Each island
maintains its own indexed data, and islands do not
share information with each other. Each island also
leverages the enterprise authentication system to
manage permissions and roles, providing secure
access to its data.
Each island contains any number of apps or add-ons.
These can include standard or customized Splunk
Apps and Add-Ons, and also island-specific or site-
specific versions. The configuration management
and deployment automation systems maintain the
complete set of apps and add-ons.
Site-Aware Cluster
...
Island LicenseManager(optional)
DeploymentServer(optional)
Island SearchHead Cluster
Island JobServer Cluster(optional)
Indexer 1Cluster MasterNode
Indexer n
WHITE PAPER
9Multiple Splunk as a Service (MSaaS)
Figure 4. Islands provide segregation of data.
Search Tier The search tier in the MSaaS architectural framework consists of two distinct components: island search
heads (whether clustered or not) and bridges. The island search heads service only their island and cannot
access data indexed by other islands. This design enables the creation of separate jurisdictions of data,
providing data segregation that may be required to meet regulatory requirements or internal enterprise
security and privacy policies. In the configuration shown in Figure 4, for example, data indexed and stored
on Island 1 cannot be seen by Island 2.
Data Sources
SearchHead
Indexer IndexerIndexer
Island 1
Indexer IndexerIndexer
Data Jurisdiction 2
Island 2
Data Sources
SearchHead
Data Jurisdiction 1
WHITE PAPER
10Multiple Splunk as a Service (MSaaS)
A bridge is similar to an island but has no indexing or data collection capabilities or components and does not
require access to a license server. Bridges are used to provide search access to indexed data on one or more
islands and can be deployed to access data across multiple islands. In the example shown in Figure 5, Island 1 and
Island 2 use a bridge to create a new data jurisdiction that includes the data indexed on both islands.
Figure 5. Bridges are used to create data jurisdictions that span multiple islands.
The search heads in both the island and the bridge are shown as a single component in Figure 5, but they can be deployed as clusters or as multiple, distinct search heads, as required.
Search Head
Bridge
Data Sources
SearchHead
Indexer IndexerIndexer
Island 1
Indexer IndexerIndexer
Data Jurisdiction 2
Island 2
Data Sources
SearchHead
Data Jurisdiction 1
WHITE PAPER
11Multiple Splunk as a Service (MSaaS)
The MSaaS architectural framework also supports hierarchical and overlapping data jurisdictions (see Figure 6). The framework enables the creation of data jurisdictions that provide the data segregation required to support separate customer use cases while simultaneously allowing wider or global visibility needed for certain use cases, such as an enterprise security monitoring use case.
Cluster master node resilience is not implemented
in the core Splunk Enterprise software. However,
the island can be configured with either a warm
standby or cold standby cluster master node that
can be brought into service if the primary master
node fails. A warm standby is a cluster master node
that is running and ready to take over immediately if
the active cluster master node fails. A cold standby
is a similarly configured system that is normally
not running but can be started if the cluster master
node fails. The appropriate choice of the warm or
cold standby configuration depends on the disaster
recovery RTO for that island. The indexing cluster
continues to function as normal during the recovery
of the cluster master node, because it can sustain a
period of cluster master node downtime without loss
of functionality.
Indexing Tier The MSaaS architectural framework recommends a
clustered indexing tier. The indexing tier in an island
contains one or more indexers, a cluster master
node, and an optional redundant standby cluster
master node—identical to a standard distributed
Splunk Enterprise deployment. The specific indexing
components deployed and their configuration
are determined by the island’s service attributes,
including the required resiliency and disaster recovery
capabilities.
Even if multiple clustered indexers are not initially
required for resilience and disaster recovery, the
indexing tier should be deployed as an atrophied
site-aware cluster—consisting of only an indexer and
a cluster master node—to accommodate automatic
site-aware replication if resilience or disaster recovery
is required in the future.
Figure 6. Bridges can be used to create overlapping and hierarchical data jurisdictions.
Data Jurisdiction 1Data Jurisdiction 2
Data Jurisdiction 3
Island 1 Island 2 Island 3
Bridge 3
Bridge 1 Bridge 2
Search Head Cluster
Search Head Cluster
Search Head Cluster
Indexer IndexerIndexer Indexer IndexerIndexer Indexer IndexerIndexer
WHITE PAPER
12Multiple Splunk as a Service (MSaaS)
Data Collection Tier The data collection tier in the MSaaS architectural
framework consists of end points that generate data
and forwarders. A forwarder is a Splunk Enterprise
instance that forwards data to an indexer, either
directly or through any number of intermediate
forwarders. An instance that receives data from a
forwarder is called a receiver. Intermediate forwarders
are usually deployed to meet network-topology or
security requirements or to enhance data safety where
there is a risk of data loss due to network performance
issues and a shortage of in-flight data buffering.
Different types of forwarders can be deployed to meet
the performance and use case requirements for a
particular configuration.
• Universal Forwarder. A universal forwarder is a
dedicated, streamlined Splunk Enterprise instance
that contains only the essential components
required to send data. Universal forwarders have
no parsing, searching, indexing or alerting
capabilities. Their primary role is to send unparsed
data to indexers or intermediate forwarders. Of
particular relevance to the MSaaS framework,
universal forwarders can be used to implement
routing of data to different indexers. However, given
that the universal forwarder does not parse the data
in any way before forwarding it to a receiver,
content-based routing cannot be performed by
universal forwarders.
• Heavy Forwarder. A heavy forwarder is a full Splunk
Enterprise instance with some features disabled to
achieve a smaller computing-resources footprint.
Heavy forwarders parse data before forwarding it
to a receiver and can support content-based routing.
Unlike universal forwarders, heavy forwarders can
index data locally and can optionally be configured
to perform local searches and alerting.
Data routing is a key requirement of the MSaaS
architectural framework. A single end point in the
configuration can have multiple data sources and
generate multiple types of events. These events
may need to be routed to different islands or be
filtered out to eliminate the indexing of superfluous
events (see Figure 7). For example, certain security-
related events may be routed to indexers on an
MSaaS administrative island that provides security
monitoring for all managed islands, with other events
being routed to islands deployed for the customer-
specific use cases.
A more detailed, complete discussion of routing
implementation is beyond the scope of this document.
Although islands are separated from each other,
Figure 7. Forwarders are used to implement data routing.
their use cases can overlap and they may need to
share data or data sources. This can be achieved by
consolidating the islands that need to share data
into a larger island. However, this may not always be
possible if jurisdiction constraints or data segregation
requirements do not permit this sharing. In these
cases, data routing in the data collection tier can be
used to control the distribution of multiple copies of
the data in question to the appropriate islands. The
volume of data routed to each island is counted once
against each destination island’s license. Thus, events
sent to multiple indexers are counted multiple times,
once per destination indexer.
Data sources indexed by a given island are partitioned
into specific indexes based on security, ownership,
visibility constraints, and retention and resilience
requirements. Index design and routing is an
island-specific construct, and these attributes can
differ across islands depending on the use cases
implemented by each island.
Indexers
Customer-Specific Island Global AdministrativeIsland
Security Events
HeavyForwarder
Customer-SpecificEvents
WHITE PAPER
13Multiple Splunk as a Service (MSaaS)
Namespace Best Practices Splunk Enterprise implementers and users define
and maintain a large number of named entities—
knowledge objects and their consolidation into
apps or add-ons in addition to a wide range of
deployment-related constructs such as roles, users
and indexes.
As discussed in the chapter “Develop naming
conventions for knowledge objects” in the Splunk
Enterprise Knowledge Manager Manual, Splunk’s
recommended best practice is that Splunk
implementers and users develop and enforce
naming conventions for knowledge objects. These
conventions facilitate the reuse and sharing of
knowledge objects across a single deployment and,
by extension, across multiple Splunk Enterprise
deployments.
In an MSaaS deployment, the significance of
maintaining a common naming convention and
namespace across the entire deployment is increased
significantly due to the fact that the deployment’s
requirements are likely to evolve in a way that can
require the consolidation of multiple instances
into a single instance. If naming conventions and a
common namespace are not strictly adhered to across
all named entities, such a consolidation becomes
difficult, at best, or even impossible without significant
rework and a high risk of failure and data loss.
Thus, conventions for the naming of all Splunk
entities—knowledge objects, indexes, sources, source
types, roles, users and anything else—must be defined
and strictly enforced to help ensure that a single
namespace is maintained across the entire MSaaS
deployment.
MSaaS Roles Security for the MSaaS framework is implemented
through the combined concepts of islands,
jurisdictions and roles. Standard authentication and
authorization services used by the enterprise, such
as LDAP or Active Directory, are typically utilized to
manage user access throughout the deployment.
Splunk Enterprise users are typically assigned to
roles, each of which is associated with a set of
capabilities. Splunk’s best practices define a set of
common roles that can be used to partition and
authorize users in a systematic way across multiple
deployments. Table 1 lists these roles and their
corresponding responsibilities.
Role
Administrator
Search Expert
Email User
User
Project Manager
Developer
Responsibility
Design and execute the architecture and deployment; manage users and provide support; administer OS, network and infrastructure
Develop searches and analyze results; help users resolve search-related issues and understand data sources
Request searches from search expert and receive reports by email
Run saved searches and utilize dashboards to gain domain-related insights
Define project goals and set requirements; mange projects and coordinate across departments
Customize and configure the Splunk Apps and Add-Ons and develop the site- and island-specific apps and add-ons as required; implement configuration file changes; customize and develop dashboards, scripted inputs, hooks to data sources, and external inputs; use Splunk APIs and scripting tools
Table 1. Common roles and responsibilities.
WHITE PAPER
14Multiple Splunk as a Service (MSaaS)
ServiceAn IT service provided to one or more internal business units, including all applications that are managed separately in addition to wider infrastructure concepts that are provisioned to customers.
JurisdictionAn entity that defines a set of constraints on access to data contained in one or more islands.
MSaaS deployment server
A centralized system responsible for installing, configuring and updating the islands’ configurations.
In addition to the above roles, Splunk Enterprise
administrators can create custom roles to help assign
specific capabilities to users, as needed to meet
security requirements. For example, roles can be
defined with attributes such as the following:
• Default list of indexes to be searched
• Restricted list of indexes
• Restrictions on the data that can be searched
with a search filter
• Restrictions on the time window for which
searches are allowed
• Inherited roles or capabilities
Roles are defined for each island and are used to
enforce data access controls within a single island.
In the MSaaS architecture, a single global instance of
Splunk Enterprise’s configuration file is maintained.
This configuration file, which specifies the user roles
for each island, is centrally managed and propagated
to each island by the MSaaS Deployment Server. (For
a description of the MSaaS Deployment Server, see
the “Deployment” section on page 21, in the chapter
on Administration).
When a user logs into a specific island, that user’s
role within that island is determined from the local
copy of the configuration file and the user is assigned
the appropriate permissions and capabilities for that
island. A user’s role and corresponding permissions
can vary from island to island, enabling the
enforcement of flexible data segregation and security
policies. For more information on users and role-
based access control, see the section “Use access
control to secure Splunk data” in the Securing Splunk
Enterprise manual.
Component
Bridge
Definition
A type of island that consists of search tier components only and no indexing or data collection capabilities; used to provide search access to data on two or more islands.
IslandA single end-to-end Splunk Enterprise deployment with indexing cluster, search head cluster, and optional license, deployment server, and job servers.
Primary island Forwarder attribute: An island whose deployment server configures the forwarder.
Service attributesProperties such as replication factor, search factor, disaster recovery RPO and RTO, security, backup, storage tier, performance, retention plan, and daily volume.
TenantAn independent user of Splunk Enterprise. Normally an individual business unit or an external customer.
Summary: Architectural Components Table 2 contains a summary list of the primary concepts and components introduced in the MSaaS logical
architecture. For a complete list of Splunk terms, see the online Splexicon.
Table 2. Concepts introduced in the MSaaS architectural framework.
WHITE PAPER
15Multiple Splunk as a Service (MSaaS)
MSaaS Physical Architecture
The island is the basic deployment unit for the MSaaS
architecture. Each island is self-contained and is
deployed with the apps and add-ons it needs for its
use cases. It includes any number of search heads,
indexers, job servers and cluster master nodes,
together with an optional deployment server or
license master. Islands should also include redundant
components, if needed to meet their resilience
requirements.
The following sections describe the physical
architecture of the search, indexing, and data
collection tiers.
Search Tier In the MSaaS architectural framework, the search tier
of each island can contain both interactive search
heads and non-interactive jobs servers. Use of
search head clustering is recommended to provide
search tier resilience and scalability (see Figure 8).
Configuring a search head cluster with multiple
search heads enables these cluster members to run
searches across a common set of indexers, referred to
as the search head cluster’s search peers.
Figure 8. Search head clustering provides resilience for the search tier.
In a search head cluster, one cluster member is
designated the captain. The captain is a search
head, no different from the others, but it is
responsible for coordinating job scheduling and
replication activities between the cluster members.
Any search head cluster member can perform the
role of captain, but there is just one captain at any
time. When a captain fails, another search head
is elected as captain, providing resiliency for the
search cluster.
In addition to the search heads, each search
head cluster also contains a deployer (not to be
confused with the deployment server). This Splunk
Enterprise component distributes apps, add-ons
and other configuration updates to the cluster
members. In addition, a load balancer can be used
with the search head cluster for resilience and
simplicity of access, as described in the “Search
head clustering architecture” section of the Splunk
Enterprise Distributed Search documentation.
Load Balancer
Search Head Cluster
ClusterMembers
Search Peers
Deployer
Captain
WHITE PAPER
16Multiple Splunk as a Service (MSaaS)
Indexing Tier In the MSaaS architectural framework, the indexing
tier consists of one or more indexers with a nominally
optional cluster master node. Each indexer is assigned
to a single specific island to ensure data segregation
and to facilitate licensing.
Four basic indexing tier configurations provide
a range of data replication and disaster recovery
options, with each island able to select the
configuration model that best suits its needs.
• Non-replicated, single site
• Replicated, single site
• Non-replicated, multiple sites
• Replicated, multiple sites
Clustering and multisite clustering are supported in
the MSaaS architectural framework to provide data
replication and disaster recovery capabilities for an
island. Clustering provides replication of data, with
two or more indexers maintaining copies of the
indexed data. Multisite clustering provides disaster
recovery capabilities by deploying two or more
indexers at geographically separate sites. Configuring
each site with a full copy of data helps protect against
a site-wide failure.
As mentioned previously, if the island’s resilience
requirements do not dictate redundancy or disaster
recovery, the island’s indexing tier should still be
configured as an atrophied site-aware cluster
consisting of a single indexer and a cluster master
node. This configuration facilitates conversion of
the indexing tier to include redundancy or disaster
recovery if these are later required.
Each indexing cluster can include a standby cluster
master node to provide failover protection. Standby
cluster master nodes are configured with copies
of the primary cluster master node’s static state
prior to failover. Whenever changes are made to the
cluster master node’s configuration, the changes
must propagate to the standby cluster master node
to maintain the identical state. In addition, the island
must be configured to locate the replacement cluster
master node after a failure occurs. A load balancer,
DNS-based failover, or another technique can
be used to facilitate failover between the cluster
master node and the standby cluster master node,
enabling the cluster’s indexing peer nodes and the
search heads to locate the standby cluster master
node after failover and recognize it as the now-
current master. No loss of functionality is expected
during the failover process, because the indexing
cluster continues to function normally without a
functioning cluster master node until the standby
cluster master node is fully functional.
Data Collection Tier Each data collection dataflow in the MSaaS
architectural framework terminates with a Splunk
Enterprise forwarder that transfers data to an
indexer. Each forwarder is either a universal
forwarder or a heavy forwarder, depending on
whether the data it handles requires content-
based processing to reach its correct destination.
If content-based processing is required, a heavy
forwarder is used. By default, universal forwarders
are used.
Data sources in the MSaaS architecture can be
divided into two classes: devices and general-
purpose systems. Each class of data source
is accommodated differently in the MSaaS
architectural framework.
• Devices. Devices are those data sources that
generate log messages, have little or no local
storage capacity, and have no general-purpose
computing capability to run additional
software to support log collection. These
devices are normally configured to send their
data to forwarders that reside on general-
purpose systems that then forward the
information to the designated indexers, either
directly or through intermediate forwarders.
In addition to the Splunk Enterprise forwarder,
if required, these general-purpose systems can
run additional software to collect the data and
improve data safety. For example, in the case of
syslog data, a syslog server, such as syslog-ng,
can be used. Similarly, if the data is in the form
WHITE PAPER
17Multiple Splunk as a Service (MSaaS)
of files residing on the data source and the data
source supports a file transfer protocol, a compatible
software client can be used to pull the data file from
the data source.
• General-purpose systems. General-purpose
systems often have storage and computing
capacity to run additional software. If they do—IT
policies permitting—these systems run a Splunk
Enterprise forwarder to collect data. It forwards
this data to either an intermediate forwarder
running on another general-purpose system, which
is usually dedicated to this task, or directly to the
indexing tier.
A Splunk Enterprise configuration file is used to
define how forwarders send data on to receivers. A
single forwarder can have multiple configuration files,
and the settings from these multiple configuration
files are combined to determine event routing. In the
MSaaS architectural framework, custom configuration
files are created to handle the various data routing
and data segregation requirements for the islands.
These configuration files can be defined globally or
for each jurisdiction or any collection of islands. To
maintain consistency, these files are stored in the
MSaaS VCS. If changes are required, these edits can
be made centrally and then pushed to the forwarders.
Administration
Administrative tasks for an MSaaS deployment can
be classified into two primary categories: deploying
new Splunk Enterprise instances, and managing
and monitoring the deployed instances. VCS and
configuration management toolsets are needed to
develop, maintain and run an MSaaS implementation.
In addition, a dedicated administrative island is
recommended to oversee and manage all other
islands in the deployment and to maintain a site-wide
security posture.
The following sections discuss deployment and
ongoing management of an MSaaS implementation,
along with related licensing, chargeback and
credential management for the deployment.
Deployment
The MSaaS framework enables the fully automated
deployment of Splunk islands and defines a
centralized MSaaS deployment server consisting of
repositories of binaries and MSaaS-specific content,
configuration management tools, and a VCS to
maintain the configuration text files (see Figure 9).
WHITE PAPER
18Multiple Splunk as a Service (MSaaS)
Figure 9. The MSaaS deployment server maintains binaries in a package manager and configuration files in a VCS.
Configuration management tools are used to
automate the deployment of new MSaaS islands and
to roll out updates to existing islands. The MSaaS
framework allows a choice of any commercial,
enterprise-class configuration management system.
For example, a single configuration management
system that supports both Linux and Windows
systems can be used. Alternatively, two separate
configuration management systems—one for Linux
and one for Windows—can be used, at the cost of
additional management complexity.
A single, centralized MSaaS deployment server (MDS)
is used to manage the deployment of islands. This
system is responsible for distributing and installing
the Splunk Enterprise binaries, configuration files, and
apps, and it also maintains the static configuration
of each Splunk Enterprise component. The MDS also
updates binaries for each island when necessary.
Each island can contain a Splunk Enterprise
deployment server (DS) of its own to process
configuration updates for all Splunk Enterprise
components on that island. With this deployment
approach, the centralized MDS deploys configuration
files to the DS on each island, which then propagates
these files to the Splunk Enterprise components on
the island and the forwarders that island controls. If
the island does not include its own DS, the MDS must
maintain each individual component on that island.
...Standard
Island 1–Specific
Customized
Island n–Specific
Enterprise
Apps, Add-Ons and Configuration Files
Configuration ManagementSystem Files
ConfigurationManagement
SystemUniversalForwarderVersions
splunkdVersions
Binary Package ManagerVersion Control System
MSaaS Deployment Server
IntermediateForwarders
Forwarders
Customer IslandsAdministration andSecurity Island
Operational SourceCode, Scripts andConfiguration Files
Routing andTransformation
Table
Bridges
OperationalBinaries
WHITE PAPER
19Multiple Splunk as a Service (MSaaS)
The MDS server maintains two primary repositories:
one containing MSaaS-specific content and the other
containing binaries. The MSaaS-specific content
is stored in a VCS to maintain consistency across
deployments, enable version tracking and control,
and provide an audit trail.
The following entities needed for an MSaaS
deployment are typically maintained in the VCS:
• Splunk Apps and Add-Ons. Standard and
customized apps or add-ons, or those specific to a
subset of the islands.
• Operational source code, scripts andconfiguration files. Items that help maintain the
deployment, implement customized administration
tasks or perform other operational tasks.
• Routing and transformation tables. Configuration
files that control the routing and data segregation
requirements for the islands.
• Configuration management system files. Files
specific to the configuration management
system that is used, such as Chef recipes, Ansible
playbooks or Puppet language scripts.
• Island-specific configuration files. Includes
pointers to island-specific applications and other
applications that an island uses.
The MDS maintains the deployed binaries using one
or more package management tools, such as the YUM
package-management utility or Windows Installer. In
addition to the Splunk binary packages, this repository
should include any operational binaries that are needed
to implement the island’s functional requirements, such
as a Syslog server implementation, file transfer software
or file integrity monitoring software.
The MSaaS framework is platform-agnostic, and
it imposes no requirements on the use of specific
configuration management, package management
or VCS software. Any enterprise-class VCS can be
used to maintain the consistency of the configuration
files. Similarly, any configuration management system
or package management system with sufficient
capabilities can be used to automate the island
deployment. This flexibility enables enterprises to
capitalize on their existing in-house expertise and
reduces their learning curve when implementing an
MSaaS deployment.
Management and Monitoring
Deploying a separate MSaaS administrative island is
recommended in the MSaaS architectural framework.
This administrative island includes a Splunk Enterprise
instance and is used primarily to monitor and maintain
all the other MSaaS islands in the deployment, including
its global security posture. If indexing-volume-based
chargeback is required, the administrative island
can track the usage of each island and generate the
appropriate reports for this purpose.
In other respects, ongoing management and
monitoring of an MSaaS deployment is similar to that
of any Splunk Enterprise deployment. Administrators
can use the command line or Splunk Web interfaces,
and they continue to monitor the standard
configuration files as for a traditional distributed
Splunk Enterprise deployment. In addition, custom
apps can be built to perform any site-specific tasks
required for an individual deployment.
The monitoring console (MC) can play a part in
managing an MSaaS deployment. The MC provides
insights into various performance and usage statistics
for the Splunk Enterprise components deployed across
the multiple islands. The MC can be configured for
distributed mode, enabling administrators to log into
one instance and view performance information for
all instances in the deployment. It is also possible to
maintain distinct views for each island by using MC with
tagged groups.
In addition, the implementation of an MSaaS
deployment should include developing an MSaaS
administration app to consolidate information that is
not covered by the MC.
For example, this app could track the activities
of the MDS, report on per-island indexing for the
purpose of chargeback, and track the health of the
WHITE PAPER
20Multiple Splunk as a Service (MSaaS)
different islands.
Licensing and Chargeback Licensing for the MSaaS architectural framework
uses the standard Splunk Enterprise licensing
model with license pools (see Figure 10). Licenses
are aggregated into license stacks, managed by
a license master and each pool provides the total
licensing volume for a single island. Additional
licenses can be added to the license master’s
managed licenses dynamically, increasing capacity
as needed. Similarly, license pools can be increased
as required.
Figure 10. Each island is allocated a license pool.
In the MSaaS architectural framework, the license
is centrally purchased and controlled. While in
principle a local license master can be deployed
on each island, the MSaaS architectural framework
recommends that islands rely on a global license
master. Each island is allocated a specific license
pool from a license stack managed by the global
license master, based on its indexing volume
requirements. The license usage can form the basis
for a number of chargeback schemes that can
depend on the license reserved or utilized by each
island, or a combination of both.
Credentials The credentials for each island can be maintained
by the enterprise’s central identity management
system. The allocation of credentials for each role in
each island is carried out as part of the automated
deployment process. A detailed discussion of this
subject is beyond the scope of this paper.
Example Implementation
Schuberg Philis, a European business technology
company focused on supporting mission-critical
applications for its customers, implemented the
MSaaS framework described in this paper. It was
facing growing data volumes and a need for multiple
Splunk Enterprise instances implemented as a flexible
and adaptable solution that could handle its workload
and rapidly deploy new instances. The primary
objectives were a scalable and high-performance
system; high availability and resilience for the
instances; data security, with isolation between
customers and RBAC to control access to data; and a
flexible solution with automated deployment.
...
...
License
Indexer ClusterIndexer Cluster
LicensePool 1
LicensePool n
Island 1 Island n
License Master
WHITE PAPER
21Multiple Splunk as a Service (MSaaS)
Functional requirements of the MSaaS
implementation included:
• Automated deployment and update of multiple
Splunk Enterprise instances with packaged
binaries, scripts and configuration files.
• Flexible, scalable design, with each instance
customized as needed.
- Instances scalable from very small to large,
as necessaray.
- Archiving, backup, and resilience requirements
defined per customer; no single point of
failure dual-datacenter configurations, indexing
tier resilience and storage resilience.
- Indexing tier replication-based data protection.
• Strict data segregation, with no shared resources
nor shared information between individual
customer instances. Cross-jurisdiction access
possible only when explicitly enabled, for functions
such as security.
• Jurisdiction hierarchy supported, with the
ability for one data jurisdiction to include other
jurisdictions.
• Multiple licensing models, including central license
masters and a license pool per island or
independent licenses managed by customer
islands if needed, to meet individual customer
requirements.
• Credentials maintained in the enterprise
identity management system, using OpenAM
for authentication and single sign-on and Active
Directory for authorization of users.
Architecture
Figure 11 shows the architecture implemented by
Schuberg Philis. The Apache CloudStack cloud
computing platform provides virtualization capabilities
and creates the underlying cloud infrastructure.
Primary components include multiple islands, each
implementing a full Splunk Enterprise deployment for
a single customer; a Schuberg Philis internal island;
a bridge used for security information and event
management (SIEM); a global license master; and
an Active Directory server. Nagios IT infrastructure
monitoring solution monitors the IT infrastructure. The
OpenAM open-source authentication, authorization,
entitlement and federation software platform, acts as
an authentication proxy to provide secure, single sign-
on access to the systems for clients.
Supporting systems include Chef systems and cloud
infrastructure automation framework for configuration
management, the Yum automatic updater and
package installer and remover for RPM systems for
package management, the GitHub Enterprise version
control repository, an orchestration server running
internally-developed orchestration scripts, and a file
server serving up artifacts. All change management
and version control are automated using GitHub
Enterprise and configuration management is
automated using Chef. The artifact server and Yum
repository store and distribute binaries, and the
orchestration server runs scripts that facilitate the
deployment process.
Of note, the following components are not
implemented in this example: a Splunk Enterprise
deployment server on each island and dedicated job
servers. Schuberg Philis does not require dedicated
job servers for their deployment. And the deployment
servers are not needed, because deployment tasks
are handled by the Chef software, which deploys
each component.
WHITE PAPER
22Multiple Splunk as a Service (MSaaS)
Figure 11. MSaaS architecture implemented at Schuberg Philis.
Schuberg PhilisInternal Island
...
... ...
Support Systems
Deploy
Customer Islands
IdentifyInfrastructureAlerts
InfrastructureMonitor
IdentityManagement
Server
Self-LicensedInternal Island
SecurityBridge
Single Sign-OnAuthentication
Proxy
ConfigurationManagement
System
PackageRepository
VersionControlSystem
OrchestrationServer
ArtifactServer
GlobalLicenseMaster
WHITE PAPER
23Multiple Splunk as a Service (MSaaS)
This architecture was deployed in a dual data
center environment. Both the replication factor
and search factor are two (that is, two copies of
searchable data are maintained), and data backups
are not implemented. Search heads are configured
in an active-standby configuration for resilience
in the event of a search head failure. No routing is
implemented in the current deployment. Universal
forwarders are used, rather than heavy forwarders,
and each forwarder sends data to a single island. SSL
certificates are used to ensure the security of data
sent from forwarders to customer islands.
Authentication and Authorization Single sign-on is implemented through the OpenAM
authenticating proxy, which enforces certificate-
based, dual-factor authentication of users to securely
control external access to the islands. OpenAM
interfaces into the Schuberg Philis’ Active Directory
server. When a user logs in, OpenAM authenticates
the user and then sends the user name to the
Active Directory server. The Active Directory server
maintains a per-island mapping between the Active
Directory groups and the Splunk Enterprise roles.
All credentials for the islands are managed through
OpenAM and Active Directory; no user credentials are
stored on the islands.
This approach enhances scalability through ease of
administration while maintaining controlled access
and segregation. Administrators sign on once and
are authorized to access a specific island. They can
then transparently switch between islands—and
be authorized for each island they visited—without
requiring re-authentication. This design makes
administration of a multiple-island deployment very
easy, and the approach seamlessly scales as the
number of islands increases.
Licensing Two centralized license masters are deployed: one
to maintain the managed service provider (MSP)
licenses for Schuberg Philis, and one to maintain
their not-for-resale (NFR) license. The MSP licenses
are used for the customer islands, using a separate
license pool per island. The NFR license is used on
an island dedicated to the company’s infrastructure,
employee training and proofs of concepts.
Each individual customer-island is assigned its
own license pool to ensure license violations from
that customer-island do not affect the operation
of other islands. In addition, one customer is using
the full features of Splunk Enterprise. Because the
MSP license does not allow end-user customers to
build dashboards or have full access to the Splunk
Enterprise interface, it is necessary to deploy a
separate license master on that customer’s island.
Switching to a private-license model for this customer
was trivial; implementation consisted of simply
pointing that customer’s island to its own license
master rather than to the global license master.
Monitoring and Management Nagios software and the Schuberg Philis internal
island, which hosts a Splunk Enterprise dashboard,
tracks license utilization and indexing, and monitors
the Splunk Enterprise deployment. In addition to
managing the MSaaS infrastructure, the internal
island ingests logs from the global license manager.
The license pools for the customer islands are one
component that requires careful monitoring. When
a license violation occurs, the Nagios software raises
an alert. Initially, these alerts occurred often, leading
Schuberg Philis to proactively monitor the license
usage for each customer.
To help prevent indexes from growing out of bounds
and exhausting the available storage, a maximum
index-size is set to limit storage utilization. 250GB
of storage is assigned to each indexer for four
60GB indexes and a minimal 1MB for the default
index, which is normally not used. A volume-based
retention policy is used and if the indexing quota is
exceeded, older events are automatically dropped.
This approach scales well, as retention volume can be
controlled centrally for easy administration of multi-
island deployments. A default storage allocation per
indexer is centrally specified, and it customized to
accommodate non-standard customer requirements
when deploying a new island.
WHITE PAPER
24Multiple Splunk as a Service (MSaaS)
Automated Provisioning
When a new customer requests the provisioning of
an island, the service attributes are defined and the
automated provisioning process is triggered. The
attributes include the resilience and disaster recovery
requirements, licensing and storage volume, island
specific SSL certificates, retention requirements,
and data sources and data segregation requirements
for this customer. The automated provisioning
process then implements the specified attributes
and deploys the island.
An internally-developed orchestration script is used to
implement the automated provisioning of islands. This
script uses Apache CloudStack APIs, Chef cookbooks
and the Yum repository to deploy and configure the
software components for the infrastructure. Different
Splunk Apps are deployed on each island, based
on the service attributes of that island. Schuberg
Philis have evaluated the open-source Terraform
orchestrator to replace the internally developed
orchestration scripts and have found it suitable.
A hierarchy of Chef cookbooks is used for island
configuration. A global Splunk Enterprise cookbook
is used to set default parameters for all islands.
In addition, island-specific cookbooks are used
to override the default parameters, as needed, to
meet any specific configuration requirements for a
particular island.
Specifically, the orchestration script and cookbooks
automate the following tasks:
1. Create the island
2. Instantiate the server
3. Install and configure the Splunk
Enterprise software
4. Configure the cluster
5. Create and manage the data disk and indexes
6. Configure data replication between search heads
7. Configure security (firewall rules, SSL setup)
8. Configure monitoring (monitoring of processes,
connectivity, cluster health and Splunk
Enterprise alerts)
9. Configure single sign-on
10. Install Splunk Enterprise Apps
Change management and patching are also
automated in the Schuberg Philis deployment.
GitHub Enterprise version control software handles
all version control. When the Splunk Enterprise
software requires upgrading, new versions of the
appropriate binaries are deployed to the Yum
repository. The artifact server stages the files, and
the new binaries are deployed to the islands after
applying operating system patches and any Splunk
Enterprise software patches by the orchestration
server. This approach provides central management
for software upgrades and facilitates the automatic
upgrade of customer islands.
Post-implementation Conclusions
Implementing the MSaaS architecture at Schuberg
Philis was a success, with 15 deployed customer
islands (as of publication). Before embarking on
the MSaaS based design and implementation, the
company was confident of the Splunk Enterprise
software capabilities but required a strategy to help it
efficiently deploy and manage multiple instances. The
MSaaS-based Splunk Enterprise deployment enabled
Schuberg Philis to successfully provide support for
mission-critical applications for multiple customers
without having to allocate significant resources for
its ongoing maintenance, and the manageability,
performance and scalability of the solution met
their requirements. The dynamic schema-on-the-
fly functionality provided high performance for
search queries, and the superior reporting, graphing,
and search capabilities enabled Schuberg Philis to
efficiently monitor applications and support
its customers.
The biggest benefit of the MSaaS architecture
deployed at Schuberg Philis is the huge efficiency
achieved by the automatic deployment of new Splunk
Enterprise instances for customers. Deployment time
WHITE PAPER
25Multiple Splunk as a Service (MSaaS)
takes approximately one hour per island. In addition,
by having a scripted deployment, the potential for
human errors and misconfigurations are minimal.
The use of a VCS provides a valuable audit trail for
tracking new deployments, upgrades and changes to
configurations. This flexible solution enables Schuberg
Philis to efficiently provide a scalable solution for a
growing number of customers with very little Splunk-
related marginal overhead per customer.
The MSaaS architecture enables Schuberg Philis
to track and understand per-customer usage.
Although they are currently not implementing per-
customer utilization chargeback, they can easily
add this functionality at any time if their business
requirements change.
Schuberg Philis has deployed a bridge for security
information and event management (SIEM) and
developed an app to monitor its security posture. This
bridge provides visibility into events on all customer
islands, and it has a proven ability to detect security
breaches. The security app can also be deployed to
customer islands to help maintain the islands’ security.
Currently, the Schuberg Philis implementation does
not use complex routing. No heavy forwarders are
used and all the universal forwarders send data to a
single island. However, certain potential use cases are
leading them to consider adding heavy forwarders to
their infrastructure. In one case they are considering
using Splunk DB Connect to integrate a customer’s
structured data sources with the customer’s island.
In another case, a customer has sensitive information
that would need to be filtered out before sending
the data from the customer’s site to their island.
The MSaaS architecture inherently supports these
use cases.
Key Observations A key observation from Schuberg Philis is the value
of a central user directory and single sign-on
capabilities to provide management at scale. These
features are a prerequisite in enabling administrators
to securely access the islands and transparently
switch between customer islands while maintaining
authorization control.
In addition, use of a cloud virtualization framework
with API access is a prerequisite for scalability and
dynamic orchestration. Physical provisioning of
the infrastructure would be too difficult and time
consuming to enable the expeditious deployment
of new islands.
One final key observation from Schuberg Philis is
the importance of investing adequately in training
the system’s implementers and in designing the
implementation and its supporting environment
before embarking on an implementation project of
this scale. While Splunk Enterprise is relatively easy
to install and use, distributed architecture design and
implementation is non-trivial and any poor design
decisions would be multiplied and spread with each
new deployed instance. Training can help reduce the
risk, speed the deployment, and assist in getting the
administrative team up and running efficiently. In
addition to expertise with Splunk Enterprise software,
expertise with the software products used to automate
the deployment—including the VCS, configuration
management system, package management tools and
any cloud infrastructure software—is also essential.
Additional Resources
This document provides the conceptual framework
for implementing the MSaaS architecture. Once
implemented, this architecture provides significant
savings in Splunk Enterprise deployment and
administration overhead. However, designing,
deploying and administering an MSaaS
implementation is a significant undertaking. Splunk
offers a variety of support and training options to help
ensure an efficient and successful implementation.
The Splunk Professional Services team has experience
with Splunk Enterprise deployments of all sizes, and
this team of experts can provide a range of services
to meet the specific needs of each organization.
For example, they can provide architectural design
guidance and a technical assessment of the proposed
Splunk environment, working along with local Splunk
administrators. If preferred, the Splunk Professional
Services team can take the lead on project
management, monitoring and managing the project.
WHITE PAPER
Download Splunk for Free or explore the online sandbox. Whether cloud, on-premises, or for large or small teams,
Splunk has a deployment model that will fit your needs. Learn more.
© 2017 Splunk Inc. All rights reserved. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. WP-Splunk-MSaaS-Arch-104
www.splunk.comLearn more: www.splunk.com/asksales
To help ensure a successful deployment and the
continued effective administration of the MSaaS
architecture, local Splunk administrators require a
thorough understanding of the technical concepts
along with practical skills. Splunk Education Services
offers direct, focused training programs that set the
stage for long-term success and efficiency. Investing
in education builds productivity and competence, and
it reduces the risk of problems caused by user error.
Splunk Education Services offers flexible training
options including instructor-led virtual classes, in-
person classes and self-paced eLearning. In addition,
Splunk Education Services offers dedicated on-site
classes and can create a custom curriculum to match
the specific training requirements of an organization.
Summary
The MSaaS conceptual architecture described in
this paper supports multitenant Splunk Enterprise
deployment as a service. This scalable multi-instance
approach features individual islands—complete, end-
to-end Splunk Enterprise deployments—that provide
data segregation for tenants while allowing global
visibility as needed for a range of functions including
security. An automated, service-request model
quickly provisions new Splunk Enterprise instances as
needed. Integration with configuration management
and VCS tools provides scalability and consistent
implementation for large installations, reducing
deployment and upgrade risks and simplifying
ongoing administration. This cost-efficient approach
can reduce the marginal cost of each additional
deployed instance and reduce the total cost of
ownership for large multitenant deployments.