Knowing your Garbage Collector / Python Madrid

Knowing your garbage collector

Francisco Fernandez Castano

Rushmore.fm

[email protected] @fcofdezc

October 21, 2014

Francisco Fernandez Castano (@fcofdezc) Python GC October 21, 2014 1 / 37

Overview

1 IntroductionMotivationConcepts

2 AlgorithmsCPython RCPyPy


Motivation

Managing memory manually is hard.

Who owns the memory?

Should I free these resources?

What happens with double frees?


Dangling pointers

int *func(void)

{

int num = 1234;

/* ... */

return &num;

}


Ownership

int *func(void)

{

int *num = malloc (10 * sizeof(int ));;

/* ... */

return num;

}


John Maccarthy


Basic concepts

Heap

A data structure in which objects may be allocated or deallocated in anyorder.


Basic concepts

Heap


Mutator

The part of a running program which executes application code.


Basic concepts

Heap


Mutator

The part of a running program which executes application code.

Collector

The part of a running program responsible of garbage collection.


Garbage collection

Definition

Garbage collection is automatic memory management. While themutator runs , it routinely allocates memory from the heap. If morememory than available is needed, the collector reclaims unused memoryand returns it to the heap.


CPython GC

CPython implementation has garbage collection.

CPython GC algorithm is Reference counting with cycle detector

It also has a generational GC.


Young objects

[elem * 2 for elem in elements]

balance = (a / b / c) * 4

’asdadsasd -xxx’.replace(’x’, ’y’). replace(’a’, ’b’)

foo.bar()


PyObject

typedef struct _object {

_PyObject_HEAD_EXTRA

Py_ssize_t ob_refcnt;

struct _typeobject *ob_type;

} PyObject;


PyTypeObject

typedef struct _typeobject {

PyObject_VAR_HEAD

const char *tp_name;

Py_ssize_t tp_basicsize , tp_itemsize;

destructor tp_dealloc;

printfunc tp_print;

getattrfunc tp_getattr;

setattrfunc tp_setattr;

void *tp_reserved;

.

.

} PyTypeObject;


Reference Counting Algorithm


Cycles

l = []

l.append(l)

del l


Cycles


PyObject

typedef struct _object {

_PyObject_HEAD_EXTRA

Py_ssize_t ob_refcnt;

struct _typeobject *ob_type;

} PyObject;


PyTypeObject

typedef struct _typeobject {

PyObject_VAR_HEAD

const char *tp_name;

Py_ssize_t tp_basicsize , tp_itemsize;

destructor tp_dealloc;

printfunc tp_print;

getattrfunc tp_getattr;

setattrfunc tp_setattr;

void *tp_reserved;

.

.

} PyTypeObject;


PyGC Head

typedef union _gc_head {

struct {

union _gc_head *gc_next;

union _gc_head *gc_prev;

Py_ssize_t gc_refs;

} gc;

double dummy; /* force worst -case alignment */

} PyGC_Head;


CPython Memory Allocator


Demo


Reference counting

Pros: Is incremental, as it works, it frees memory.

Cons: Detecting Cycles could be hard.

Cons: Size overhead on objects.


PyPy


Mark and Sweep Algorithm


Mark and sweep

Pros: Can collect cycles.

Cons: Basic implementation stops the world


Questions?


The End


Knowing your Garbage Collector / Python Madrid

Technology

Transcript of Knowing your Garbage Collector / Python Madrid