Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required...
Transcript of Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required...
![Page 1: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/1.jpg)
Monitoring Memory Usage Based on Kernel Trace Events
Progress Report Meeting
December 12, 2016
École Polytechnique de Montréal
Laboratoire DORSAL
Houssem Daoud Michel Dagenais
![Page 2: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/2.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Agenda
Introduction and objectives
Literature review
Memory usage monitoring
Memory leak detection
Conclusion
![Page 3: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/3.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Introduction
Running RunningWaiting
Running RunningWaiting
● Using faster processors doesn't always improve the performance of the applications
1 GHz
2 GHz
A 2x faster processor gives an acceleration rate of 1.3
![Page 4: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/4.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Introduction
Reasons of blocking:
● The operating system uses many mechanisms to improve the performance of those peripherals: Disk scheduler, memory allocator, caching.
● I/O operations (disk, network, etc.)● Memory Access
Our goal is to provide tools and algorithms able to evaluate the efficiency of those mechanisms and to detect
performance problems.
![Page 5: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/5.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Introduction
In a previous work, we proposed a tool that provides a lowlevel analysis of the storage subsystem based on Kernel traces.Research Paper: "Recovering Disk Storage Metrics from Lowlevel Trace Events" (Submitted)
![Page 6: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/6.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Introduction
The high frequency of memory operations makes the trace file huge
It is not possible to target a single process using basic filtering techniques since physical memory releasing is usually done out of context
Analyzing memory management mechanisms using tracing present the following challenges:
Tracing can contribute in polluting the memory of the system
![Page 7: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/7.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Introduction
Objectives● Study the different memory management mechanisms● Instrument the Kernel to get the required information● Provide filtering and aggregation mechanisms to reduce the
frequency of events.● Generate metrics and visualization from the trace file
RQ : Is it possible to analyze memory management mechanisms using tracing without perturbing the normal behavior of the system ?
![Page 8: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/8.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Agenda
Introduction and objectives
Literature review
Memory usage monitoring
Memory leak detection
Conclusion
![Page 9: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/9.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Background
● Virtual memory is an abstraction used by the operating system to compensate the shortage of the physical memory
Virtual memory RAM
![Page 10: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/10.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Background
● The mapping between virtual and physical addresses is hardwareassisted (MMU)
task_struct
mm
mm_struct
mmap
pgd
vm_area_struct
vm_area_struct
vm_area_struct
MMU
page
page
page
Virtual address
![Page 11: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/11.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Memory usage monitoring
Printezis and al. : "GCspy, an adaptable heap visualization framework. Generic clientserver architecture Uses JVM instrumentation Requires the instrumentation of the allocator Uses animated images Not appropriate for high → frequency events.
Jurenz and al. : "Memory allocation tracing with VampirTrace” Require library preload instrumentation Uses 2D charts, memory=f(time)
Cheadle and al. : “Visualizing dynamic memory allocators” Extended GCspy to support dlmalloc, a Glibc memory allocator
![Page 12: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/12.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Memory Fragmentation Detection
Dynamic allocation allows the applications to allocate and release memory at runtime.
Allocation strategies:
Sequential Fit: 256K 8K 512K
Segregated Free Lists: 4K 4K 4K 4K
8K 8K 8K 8K
16K 16K
(First Fit, Best Fit, ...)
(Buddy System)
![Page 13: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/13.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Memory Fragmentation Detection
"On the external fragmentation produced by FirstFit and Bestfit", Shore The authors used a simulator to compare the external fragmentation of FirstFit and BestFit.
"The memory fragmentation problem : Solved ?", Johnastone A tracer have been used to study the fragmentation of the heap The fragmentation is estimated by comparing the memory usage to the total size of the heap
![Page 14: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/14.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Literature review
Memory Leak Detection
M. Hauswirth and al. : “Lowoverhead memory leak detection using adaptive statistical profiling” The instrumentation of memory operations is done at compile time.
Memcheck Provided as a Valgrind module Uses the dynamic instrumentation of the application Rubanov et al. : “KEDR, Runtime verification of linux kernel modules based on call interception” Detects memory leaks in kernel modules by instrumenting the functions kmalloc and kfree.
![Page 15: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/15.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Agenda
Introduction and objectives
Literature review
Memory usage monitoring
Memory leak detection
Conclusion
![Page 16: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/16.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
● Tracecompass provides a view that shows the virtual memory usage of processes using Lttngust
● The instrumentation is based on liblttngustlibcwrapper, which defines wrappers to the glibc functions.
Monitoring virtual memory
![Page 17: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/17.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Only programs started with the LD_PRELOAD option can be analyzed.
Monitoring virtual memory
To have a systemwide analysis, we created a similar view that uses the system calls mmap(), munmap(), and sbrk() as a source of information.
size <= MMAP_THRESHOLD size > MMAP_THRESHOLD
Malloc, calloc, realloc sbrk(size) mmap
free None, or sbrk(negative) depending on M_TRIM_THRESHOLD
munmap
![Page 18: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/18.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring virtual memory
![Page 19: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/19.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring Physical memory
● Allocating a physical page is usually done in the context of the process that requires them (kmem_mm_page_alloc)
● Releasing physical pages can be done at any time by a kernel thread (kmem_mm_page_free)
Monitoring the physical memory usage of a specific process requires tracing all memory allocation/deallocation events, which are very frequent lost events + very slow analysis. →
![Page 20: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/20.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring Physical memory
Sampling
● The Kernel keeps information about physical memory usage in the mm_struct data structure (RSS: Resident set size)
● This counter is adjusted each time a physical is inserted or removed from the page table of the process.
Time512 720 900 850 600
It is difficult to select good sampling rate
![Page 21: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/21.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring Physical memory
(Time, Space) Sampling
An event is triggered if memory variability exceeds a certain threshold
![Page 22: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/22.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring Physical memory
(Time, Space) Sampling
● The sampling mechanism is implemented as a Kernel module.
● Lockfree data structures are used to provide a good scalability RCU Hashmap is used to hold process information Memory variability is defined as atomic_long
● The module is configurable through the proc file system. (sampling rate, variability threashold.
![Page 23: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/23.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Monitoring memory usage
Monitoring Physical memory
● Unified virtual/physical memory view
![Page 24: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/24.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Agenda
Introduction and objectives
Literature review
Memory usage monitoring
Memory leak detection
Conclusion
![Page 25: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/25.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Memory leak detection
● A memory leak occurs when a computer program does not free memory objects that are no longer needed.
● Tracing can be used to track the potentially leaked objects, but only the programmer is able to confirm if an object is leaked or not.
t1 t2 t3
A
BC
J
AK
A
A is potentially a leaked object
![Page 26: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/26.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Memory leak detection
Use case (Kernel 3.4)#!/bin/bash
while true;dodd if=/dev/zero of=/output.dat bs=10M count=1done
![Page 27: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/27.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Memory leak detection
Use case (Kernel 3.4)
● We used Lttng to find the pages that are allocated and never released.
● The events used are kmem_mm_page_alloc and kmem_mm_page_free with the call stack context.
'__alloc_pages_nodemask','alloc_pages_current','__page_cache_alloc','find_or_create_page','__getblk','jbd2_journal_get_descriptor_buffer','jbd2_journal_commit_transaction','kjournald2','kthread',
We found out that the "leaked" pages are allocated in the following code path
![Page 28: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/28.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Memory leak detection
Use case (Kernel 3.4)
● We contacted the developers of the file system to confirm the problem. By looking at the call stack, they quickly recognized the problem and gave us the required patch.
● The problem is in the journal mode of the ext4 file system. It is solved in Kernel v4.4rc5
![Page 29: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/29.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Conclusion
● The preliminary results show that tracing can be used to detect interesting problems related to memory management mechanisms.
● The overhead of tracing should be carefully measured on different workloads to make sure that the tracer is not altering the normal behavior of the system.
● As a future work, we started to instrument the KSM mechanism to evaluate its efficiency in different contexts.
![Page 30: Monitoring Memory Usage Based on Kernel Trace Events · Instrument the Kernel to get the required information Provide filtering and aggregation mechanisms to reduce the frequency](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec55ec0419eb03a822196dc/html5/thumbnails/30.jpg)
POLYTECHNIQUE MONTREAL – Houssem Daoud
Thank You !