An enterprise-grade approach to monitoring

and troubleshooting with Prometheus for

containers and microservices.

Prometheus metrics add a valuable dimension to cloud-native monitoring giving developers the ability to gain visibility into applications in highly dynamic service-oriented architectures. Sysdig provides enterprise-grade Prometheus monitoring with automatic collection of Prometheus metrics. Our highly scalable platform provides powerful way to aggregate, query, and visualize deep metric and events data, including Prometheus.

Contents.

Executive summary. 2

Prometheus for monitoring. 3

The Sysdig platform. 4

Sysdig and Prometheus. 6 1. Scalability, reliability and long-term data retention 6 2. Multi-cluster, multi-cloud observability 7 3. Service-oriented workflow and topology for problem isolation 8 4. Deeper troubleshooting with NO instrumentation 11 5. Enterprise capabilities 13

What metrics should I use when? 13

How it works 15

Conclusion. 16

1

Executive summary.

As enterprises transition their software architectures toward containerized cloud-native applications for speed, scale and multi-cloud mobility, DevOps teams find themselves facing a significant gap – the lack of operational intelligence to monitor, troubleshoot and secure a highly distributed and dynamically orchestrated environment.

The challenge of managing modern microservices comprising dozens to hundreds of services across thousands of containers is immense. Key to addressing this “cloud-native visibility gap” is a comprehensive data platform that assembles context-rich records of all activity and provides insights that enable DevOps teams to efficiently operate reliable and secure cloud-native applications.

Prometheus has become the default open-source tool for monitoring in conjunction with containers and Kubernetes environments. At the same time, building an enterprise monitoring platform requires several additional capabilities such as scalability of the metric store, mapping service-to-service dependencies, incorporating troubleshooting workflows, providing RBAC functionality, etc.

Sysdig’s Cloud-Native Visibility and Security Platform is a container-native data platform that marries the most comprehensive, granular and context-rich data with deep insights for monitoring, detection, troubleshooting and forensics.

This white paper discusses how Sysdig incorporates Prometheus into its rich platform to deliver an unmatched “Enterprise Prometheus” monitoring and troubleshooting solution. Sysdig’s security functionality complements the monitoring and troubleshooting functionality of the Sysdig platform but is beyond the scope of this document.

2

Prometheus for monitoring. The architecture for software development is going through a fundamental transformation - from large monoliths into highly distributed microservices, driven by the rapid adoption of containers as underlying building blocks and Kubernetes as the default orchestrator. This move towards microservices architectures accompanies another significant organizational shift in IT - the emergence of DevOps - to drive faster and more agile software design and delivery.

As DevOps teams deploy containerized, cloud-native applications in production, they quickly start to face a major barrier to operating a reliable container platform - the lack of operational intelligence to answer seemingly straightforward and frequently faced questions such as the following:

● “What applications are running within containers? How are my services mapped and what are the dependencies on each other and to underlying infrastructure?”

● “Are all my services performing normally? If not, what is the anomalous behavior? Which team should I involve in troubleshooting the anomalous behavior?”

● “How do I project my infrastructure capacity needs?”

Without the ability to answer these kinds of questions, the journey to modern, cloud-native applications can get stalled by reliability concerns. One of the key pillars of being able to answer questions such as these is what has often been referred to as “Observability” - the availability of data that provides visibility into operations.

Microservices infrastructure and application observability

3

Prometheus, an open source monitoring tool with over 20 thousand stars on GitHub has gained tremendous traction as a key monitoring tool relevant to containers and microservices environments. Key drivers for the rapid adoption of Prometheus include the following:

● Community adoption. Prometheus is a graduated CNCF project, with a rich community of contributors and users.

● Growing ecosystem. Mirroring the strong community adoption, many application & service developers are starting to build native Prometheus exporters. Key container platforms such as Red Hat OpenShift bundle Prometheus to provide users with basic cluster monitoring. Further, popular open source dashboarding tools such as Grafana have built-in support for Prometheus dashboards.

● Rich data model and query language. Prometheus incorporates a multi-dimensional data model, with time series data identified by metric names and key/value pairs. Further, it provides a flexible query language to leverage this dimensionality.

At the same time, enterprises looking to use Prometheus as their monitoring solution face some challenges as well.

● Scalability and long-term data retention: The current storage model that Prometheus uses causes it to be constrained to short-term monitoring with a few weeks of data retention and not being suitable for long-term data retention and querying.

● High availability is complex. In order to operate at scale in full HA across clusters, clouds and data centers, users need to federate across several Prometheus servers, which is an extremely complex process for an average user. Further, cross-cloud querying is complex, error-prone, and does not perform well.

● Lack of deep troubleshooting: While Prometheus provides good dashboarding and alerting functionality, it lacks capabilities needed for advanced troubleshooting and root cause analysis.

The Sysdig platform. Sysdig designed its Cloud-Native Visibility and Security Platform as a container-native platform, focused on extracting extremely granular data that is enriched and automatically tagged to provide rich context to facilitate monitoring, detection, troubleshooting and forensics at scale.

Sysdig started by leveraging system calls as a deep and untapped source of data and insights and has evolved its platform continually to embrace custom metrics such as JMX, StatsD and Prometheus. It automatically detects applications running within containers and extracts

4

relevant metrics, maps service topologies with service golden signals, and goes beyond monitoring to incorporate deep troubleshooting workflows.

Some of the core strengths of the Sysdig approach include the following:

● Minimal instrumentation. Once a host agent is deployed, Sysdig requires very little to no manual configuration to collect metrics and events from any service or application deployed on that host.

● Rich data model with extensive, automatic metadata and labels. Sysdig’s unique instrumentation not only reports metrics at all layers of the application stack (service to infrastructure), it auto-labels these metrics to provide context. Sysdig then builds a unique data model that associates metrics with a hierarchical organization of all associated entities. The data model is optimized for high-churn, high-cardinality, high-frequency, multi-dimensional data analysis.

● Deeper data. Sysdig’s approach of tracking every system call to derive metrics and events, further complemented by custom metrics and metadata from API integrations, creates an extremely comprehensive source of intelligence.

● Service orientation. The Sysdig platform maps the topology of all services as well as the physical infrastructure, including the ability to map flows and interdepencies between services.

● Troubleshooting. Sysdig not only collects metrics and surfaces problems through dashboards and alerts, it goes beyond metric storage and monitoring, enabling users to troubleshoot gnarly problems at a very granular level.

Sysdig Cloud-Native Visibility and Security Platform

5

Sysdig and Prometheus. Sysdig has been integrating with Prometheus since 2017.

Alongside its own natively-collected metrics, the Sysdig backend ingests Prometheus metrics as well as metrics from other sources such as JMX and StatsD, while tagging these metrics with metadata from Kubernetes. Users can use the Sysdig UI and APIs to create dashboards, alerts and more using Prometheus data. Additionally, Sysdig also offers a Grafana plugin for dashboarding with Prometheus data as well as any other metrics collected.

The combination of Sysdig and Prometheus allows users to get the best of both worlds: monitoring based on data from many diverse sources combined with sophisticated dashboards, alerting and deep troubleshooting. In other words, Sysdig and Prometheus are better together for the enterprise.

The principal benefits of Prometheus with Sysdig fall into the following main buckets:

1. Scalability, reliability and long-term data retention 2. Multi-cloud, multi-cluster observability 3. Service-oriented workflow and topology for problem isolation 4. Deeper troubleshooting without requiring manual instrumentation 5. Enterprise management capabilities

Let’s look at each of these areas in more detail.

1. Scalability, reliability and long-term data retention An out-of-the-box Prometheus setup scales an instance vertically by adding more resources such as disk and memory to a single instance. This is a great starting point but can hit scaling issues beyond a certain cluster size. A single Prometheus server can scale up to a thousand metrics per second on a host, limited by its ability to handle no more than a few million time-series per minute. To scale beyond the ability of a single Prometheus server, users need to address certain issues:

● Federation of data across multiple Prometheus setups ● Configuring each Prometheus server for high availability (HA) and dealing with

deduplication ● Long-term data retention for historical trend analysis ● Global querying across distributed prometheus servers with a single API call

6

A set of new open source technologies are emerging in an attempt to resolve this problem. This includes Thanos developed by Improbable, Cortex which is a sandbox CNCF project (and used by Grafana labs) and M3 open sourced by Uber. These technologies are designed to fill the gaps of isolated Prometheus servers. They look promising and confirm the need for a scalable, supported backend for Prometheus. However, these tools alone do not solve the problem of manageability and scalability.

Sysdig’s high-performance metric store provides a single, centralized, horizontally-scaled data store that can grow up to 100M+ metrics per second managed as a single stand-alone instance. This implies Sysdig can ingest at the rate of 15K unique time series per host per second as compared to Prometheus’ 700 time series per host per 10 seconds from 10K hosts. This allows data (metrics) to be ingested from many sources and many clusters.

Within monitored clusters, Sysdig agents deploy as a daemonset through Kubernetes and auto-discovery ensures that application and cluster metrics, including any existing Prometheus metrics, start flowing to the metric store within seconds of initial deployment, with no additional configuration or sidecars required. The Sysdig backend is optimized to not just ingest at scale, but also to efficiently handle reads and respond to queries at scale leveraging some key architectural choices:

● A horizontally scalable time series database running on a fully distributed highly available data pipeline.

● Distributed query processing to ensure efficient global reads across large distributed islands of Prometheus servers, across data centers and clouds.

● A microservice-aware hierarchical data structure and automatic context enrichment of the data set, which provides an advantage to read operations over a generic metric store. Sysdig provides custom intelligence to convert customer-preferred hierarchies into efficient packing algorithms and indices.

● Data compaction and rollups to ensure long-term data can be retained and queried effectively.

● Live stream, near-real time and historical data are handled differently by decoupled highly available components.

● Pre-aggregated queries for commonly used dashboards and alerts to ensure the most frequently used requests are enabled for faster reads.

● Ease of deployment using Helm charts without the need for sidecars or distributed data pools.

2. Multi-cluster, multi-cloud observability Enterprises are adopting a multi-cloud strategy to avoid vendor lock in and benefit from the strengths of different cloud environments for different use cases. Containers are fundamental

7

to enabling this shift. Prometheus does not natively support global querying across different Prometheus servers with a single API call. Similarly, OpenShift ships Prometheus out of the box with one Prometheus server per cluster.

“I have two clusters, one in AWS and one in GKE. We have two distinct Prometheus servers for each. I would like the ability to correlate and compare performance across the two clusters.”

Multi-cloud monitoring & troubleshooting

Sysdig has provided cross-cluster, cross-cloud visibility from its very beginning. You can build dashboards, compare and contrast metrics, and create PromQL expressions to analyze data from the rich data collected.

3. Service-oriented workflow and topology for problem isolation Sysdig is a critical tool for allowing development teams to deliver more reliable, secure code more efficiently.While Grafana allows for visually appealing and useful dashboards with Prometheus metrics, the workflow associated with investigating a problem necessitates that users navigate back and forth across multiple dashboard widgets making it hard to isolate problems.

In talking to Prometheus users, a few themes that emerge regarding their workflow are as follows:

8

● Service-to-service problem isolation. Prometheus helps users surface an issue via an alert or dashboard but does not take them beyond that to find the needle in the haystack.

“I know latency is high. However, I have no idea beyond that about which dependent service is causing the high latency.”

● Problem isolation across the stack. Isolating issues up and down the stack requires spanning applications and services, networks, hosts and processes, which is often not possible with Prometheus and requires manually navigating multiple different dashboards to manually solve problems.

In this context, Sysdig complements the Prometheus workflow with 3 powerful tools:

1. Detect bottlenecks visually using the service topology. Sysdig integrates natively with all orchestrators and container platforms, extracting service labels and constructing an optimized data model to aggregate data for services. Sysdig’s service topology shows inter-service communication without any instrumentation or configuration. The topology map can be overlaid with Prometheus metrics that are instrumented by users such as latency and error counts.

“I was having an issue with latency and needed to isolate what was wrong. The application-level dashboard provided me the information that the URL related to one of the services was slow. I opened the service topology for that service and boom there it was - an outbound connection to an external IP that was the culprit. Literally in a few clicks”.

Microservice topology map

9

2. Sysdig’s popular Explore workflow provides a top down view of your services letting you drill down from one level of the hierarchy to another to view relevant data at each level.

“The first thing I do every day is look at the Sysdig Explore workflow and I get a bird’s eye inventory view of my services -- my namespaces, services, and pods. From here, I can look at heap usage at any level and drill down to see which process running in which pod is the root cause of an issue. And I’m able to do it without configuring anything at all. I can then switch to the infrastructure view and find out what nodes my services are running on.”

Explore drill down view

3. Service golden signals such as response time or latency, request count and error

count are key signals used for monitoring the performance of your applications. Traditionally these metrics are gathered by code instrumentation. Sysdig provides these signals in several different ways today such as via custom metrics, application component metrics such as for NGINX, HAProxy or Istio - or by decomposing protocols such as HTTP/S on the fly. Users can observe trends on which URLs have high error count or which SQL queries are slow, to isolate where to look next.

10

“Our developers don’t have to spend weeks instrumenting every HTTP call to be able to look for things like regressions that impact performance. Without any code changes we can look at things like who is making SQL queries, how slow are these queries, and who talks to who. It’s this foundational layer that lets us move faster and worry less about breaking stuff with very little effort from our side.”

Golden signals dashboard

4. Deeper troubleshooting with NO instrumentation Sysdig was designed with troubleshooting as a primary use case. Sysdig collects a ton of rich data for hosts, containers, orchestrators and platforms, network, process and file system activity that cannot be obtained easily by other tools. Sysdig then enables correlation of metrics across metric types. Let us look at some workflows for troubleshooting that Sysdig excels in:

● Process level granularity: Through Prometheus you can get application metrics such as JVM heap usage for a deployment and identify the pod it is running on. To further isolate the specific process on the pod that was using the max heap, you would next need to SSH into the specific node on which that pod is running and look through process names.

With the Sysdig platform, with a few clicks in the user interface, you can see all the processes running on the pod that lead to high heap usage, getting you to root cause significantly faster.

11

● Network connections and file system activity: Networking data is another area where Sysdig shines. Sysdig reports all network connections including ingress and egress for any process, container, and service / deployment / namespace. This is a valuable data source that is critical to troubleshooting and forensics workflows.

A large investment bank needed to record all network connections established or attempted by a service to use for troubleshooting and audit. Specifically, they wanted to retain the details of every network connection including details on source IP, destination IP, port, traffic, connection count and more. They tested a wide range of monitoring tools for this use case and only Sysdig was able to successfully meet their requirement. Sysdig is now deployed across their environment to keep track of all network activity for all containers in their microservices infrastructure.

● Sysdig captures are incident recorders which record every single system call that was executed on a host when a problem occurs. These capture files are invaluable to deep troubleshooting to find the root causes of infrastructure issues. Captures can be triggered on alerts based on Prometheus metrics to capture every possible event on the host during before and after the alert was triggered. This allows users to actually find the needle in the haystack.

Capture file analysis with Sysdig Inspect

12

5. Enterprise capabilities As organizations look to establish an enterprise-wide monitoring solution, Sysdig meets security and manageability needs that ease adoption at scale.

● Prometheus users have come to rely on community contributions for exporters, dashboards and alerts but at the same time run into challenges of choice (Which out of the 7 dashboards should I use?) and backward compatibility. Sysdig today offers a highly curated repository of dashboards and alerts based on application best practices and is actively working on a similar repository for the most popular Prometheus integrations.

“I don’t have time to understand what Kafka metrics I need. I love the out of the box dashboards you provide.”

● Teams and Role Based Access Control helps isolate data and restrict permissions per user per team.

"This gives us multi-tenancy that scopes what each team can see, but with a single central place to host Prometheus."

● Single sign on integrates with an organization’s preferred LDAP, SAML or other authentication and authorization methods leading to better adoption.

● Audit and compliance to ensure a fully secure and compliant Prometheus environment, Sysdig provides access to audit logs that show users who did exactly what on the system. Additionally, Sysdig is embarking on the journey of becoming compliant with several standards starting with SOC2, HIPAA and so forth.

● Support: Sysdig provides enterprise class phone and online technical support.

What metrics should I use when? When monitoring the performance of your application and services, metrics are collected at every layer across the stack. Not surprisingly, there is some inevitable overlap between Prometheus metrics and what Sysdig collects natively. When evaluating which metrics to use when, let us look at these recommendations:

● If you are monitoring Java based applications and middleware (such as Cassandra, Kafka, Tomcat etc) Sysdig’s data collection has an edge over anyone in the industry. Sysdig automatically discovers JVM processes and collects all JMX metrics with no additional configuration.

13

Sysdig identifies which deployments are running Java, automatically out of the box, without needing any further configuration. This means that you can detect issues with heap usage, thread count or garbage collection for any deployment - even those that are not present yet and sleep peacefully knowing that Java monitoring is always on.

“Sysdig unifies all the metrics we need to monitor our environment. We use lots of JMX metrics from existing applications in addition to Prometheus for newer apps. We can then correlate all of it to our underlying infrastructure – that allows teams to really understand the performance of their systems.”

● For non-Java based applications, those components in your application stack that natively export Prometheus metrics (that are officially supported by the vendor e.g., MySQL, Akamai) we recommend using Prometheus metrics as they will be most current. For others, Sysdig auto-collects a wide range of application metrics with zero-instrumentation and provides out-of-the-box dashboards based on best practices.

"I don’t want to tell my monitoring system what my infrastructure looks like. Old-fashioned monitoring requires declaration of resources – usually at both ends. You have to configure not just checks on the client and the server. You need to configure the clients on the server and the servers on the client, which means you need to know where the server is when you provision the client, which is something completely against the design paradigm we have in this new platform. With Sysdig, you only need to run a container on the host and it's done. We get monitoring and security at the container level, at the host level, and at the application level.”

● For custom developer applications Sysdig is capable of collecting other custom metrics such as StatsD. If your organization has heterogeneous or legacy applications, Sysdig will provide a single pane of glass to view and correlate metrics of any time, across infrastructure and services.

“From the infrastructure team’s perspective, it’s nice that we can tell our developers to emit metrics with either StatsD or Prometheus. They don’t need to think about it or re-factor metrics. They’ll just show up in Sysdig. There are a ton of Prometheus exporters we use. It all shows up in one dashboard.”

● Hosts, containers and networks: Prometheus exposes host metrics via node exporter and container metrics via cAdvisor, which are their most popular exporters. While this is ‘good enough’ for many scenarios, to get deeper data for troubleshooting down to process level, Sysdig’s native system and network metrics are advantageous.

● Network connections: Prometheus does not provide network connection data. Sysdig does this natively using packet captures to automatically derive process-to-process connections. This network data is then aggregated up to container, pods, deployments, services and namespaces. This lets Sysdig users solve several use cases that Prometheus alone will not such as determining which services are connected to what other services and how each hop is performing to isolate bottlenecks.

14

● Kubernetes monitoring kube-state metrics: Sysdig natively integrates with the Kubernetes API and ingests Kube-state metrics to provide holistic Kubernetes monitoring. Based on best practices, Sysdig has curated alerts and dashboards that provide meaningful Kubernetes insights.

“Kube-state can tell me requests, limits and quotas but cannot tell me the actual CPU consumption against that”.

Additionally, Kube-state alone can generate 20K metrics per node and lead to scale challenges. Sysdig’s native ingestion of Kube-state metrics ensures the most relevant data is collected to avoid metric overload.

Kubernetes components: Kubernetes components such as the master api-server or controller--manager all natively export Prometheus metrics. Sysdig recommends using these Prometheus exporters by default along with curated dashboards provided by Sysdig to ensure the health and performance of Kubernetes clusters.

● Multi-cloud monitoring across cloud platforms: Sysdig natively integrates with

CloudWatch to bring AWS metrics and provides out-of-the-box dashboards and alerts. Alternatively, users can export AWS CloudWatch and Google Stackdriver metrics as Prometheus. With Sysdig you can blend metrics from cloud provider services such as S3 and EBS with other metrics and events to provide comprehensive multi-cloud insights.

How it works The Sysdig agent acts like a push gateway and automatically pulls the Prometheus metrics exposed on a host from instrumented applications and exporters. It sends metrics from all monitored hosts via the Sysdig agent to the Sysdig backend. All Prometheus metrics are available for viewing in context - that is, users can see the data organized by Kubernetes objects (by namespace, pod, etc.) or by container, cloud, etc.

Using the full-featured Sysdig UI, leveraging topology, dashboarding, exploration and more, users can observe and analyze metrics along with infrastructure events. In addition, UI driven alerting and anomaly detection help users make best use of their Prometheus metrics. Alternatively, users can query the Sysdig backend using PromQL and utilize the data using Grafana dashboards and alerts.

15

Prometheus metric collection and visibility with Sysdig

Conclusion. The Sysdig Cloud-Native Visibility and Security Platform embraces and extends open source Prometheus to deliver an “Enterprise Prometheus” monitoring and troubleshooting solution. By providing the operational intelligence needed for critical insight across dynamic, distributed services, Sysdig enables DevOps teams to deliver reliable and secure cloud-native applications faster.

With the Sysdig platform the world’s largest organizations are able to accelerate their transition to Kubernetes and container-based architectures. Sysdig’s holistic approach eliminates the visibility challenge of containers and provides the scale, security, and cross-cloud capabilities required by today’s enterprises. As a result, these companies are better equipped to keep pace with business demands and better compete in their respective markets.

Please check out the resources section on sysdig.com to find additional information and customer case studies. If you’d like to see more of how Sysdig products can be applied to specific use cases, please visit the use cases section of the site.

16